If you are looking at it in Windows Explorer, you can hit alt+v to bring up the view menu, and then choose folder options. Under the view tab, scroll to the checkbox that says 'hide extensions for known file types.' Make sure that is unchecked. That should be all you need.
I'd just try the obvious extensions before looking for any tools to do this for you.
.avi .wmv .mp4 .mpeg
90% chance it's one of those. There are not a lot of extensions that do h.264 natively, and surprised the .mp4 extension didn't work first thing. My next suspicion would be .wmv. The file COULD BE .mkv, but I doubt it. Generally a high-def format for video. Might be worth a try if you're out of options.
@MAGZine: Chances are the video player won't care which of those extensions are used. If it's a format the program recognizes it will probably play it regardless of whether the extension is correct.
How about opening it in a text editor or hex editor? I've never tried that with video formats, but there should be some indication of the format in in the file header. Whether or not you'll be able to decipher it, I don't know.
@LordAndrew: We already know the extension doesn't matter - it's just a matter of finding what extension works.
You could always just create your own extension as .h264 or something, and get VLC to handle it.
And Pazy, just re-reading now. There is a good chance that Quicktime won't play the video, and you could be missing a package/piece of software that allows DirectShow to play the h264 encoded file. Just because a file has a certain extension doesn't mean that it'll be compatible with player X or player Y - it's all in the codec support.
The extension that works isn't always the right extension. It may work in players that supports both formats, but certainly not in players that don't. He already knows that he can play the file, so technically it doesn't need any extension.