Correction: Quicktime doesn't like the MKV. Solution: Don't use Quicktime. VLC is available for OS X as well.
the MKV doesn't like quicktime player because it'll glitch out like crazy
Right, Quicktime doesn't like MKV. I've been able to convert some MKVs using Quicktime before so I figured it was worth a shot.I managed to find a solution that gets me part of the way there. iFFMpeg gives me an error saying that for audio it " can't resample input channels greater than 2" but it does manage to convert the mkv to h264. For the audio I found iMKVExtract which can demux the different streams and then I used MPEG Streamclip to convert the AC3 to AIFF so that I could sync them up in Final Cut Pro. When I put the video into a 24 fps sequence the audio drifted, but when making the sequence 29.97 they seemed to sync up, but I have to render the video out. Not a perfect solution but it'll do fine for now.
EDIT: I've since got an internal Blu-ray drive for my Mac Pro (LG WH10LS30K), and using MakeMKV on Mac has worked great and the MKVs can be opened in Quicktime Player just fine so I can select sections and convert to a format to edit with. I guess the PC to Mac transferring from before was messing something up.