Transcoder Feedback

Created by will on Feb. 20, 2015, 7:48 p.m.
  • Sticky
  • Hey guys, here's the place to put feedback on the Tested transcoder project. We're specifically looking for bugs (things that don't behave as you'd expect), problems with the install or setup process, or anything else that isn't behaving as you'd expect.

    The transcoder works best with MKV files and can handle multiple audio and subtitle tracks. By default, we're leaving the audio streams intact and only adding forced subtitles to the compressed file. You can tweak the command line that gets passed to Handbrake-cli if you want to burn subtitles in or change the audio formats too, but I haven't written docs for that yet.

    The VM should run on any system that can run Virtualbox. You'll also want to turn on all the virtualization instructions for your CPU in the motherboard, or your performance will be really bad.

    The instructions for installation are available in the readme file on Github, but they're pretty rough. Right now, we assume you know a fair amount about running virtual machines and ripping discs. If you don't know about at least one of those things, it might be tough to get this working with the info we've provided. We'll have a more complete how-to next week.

    Lots of the heavy lifting here comes courtesy of Don Melton's great video transcoding scripts. Much of what we're doing is just automating the setup and fiddlier bits of his scripts.

    Please post your comments and feedback below or on the Github page.

  • Test

  • I went through the install process and attempted to make it a bit more friendly to new users.

    It definitely works pretty well so far, it at least gets the video and audio correct.

    Having some issues with the subtitles though, which I remember you mentioning. It seems to only want to keep the forced ones. Can it just be set to pass through all subtitles, or ones that match a list of languages even? Then, as long as they were ripped, it should be good.

  • I actually just realized that it is doing something a bit odd with the audio. It is actually adding an additional track which did not exist before. It seems to be taking the AC3 surround audio and making an AAC stereo track as well. Is this intentional, or is it something the Don Melton scripts are doing on their own.

    I know I am not interested in having anything other than the original audio tracks, if I need something else Plex can handle it.

    I am going to play around with it to see if I can get it to keep only original audio and add all subtitles for now.

  • @Bedon292: All of that stuff should be addressable in the command line. You can adjust that in, but make sure your text editor uses unix style newlines.

  • @will: Right, I dug through and found those settings. I think I have that all in a working state.

    What are your thoughts on allowing all subtitles which appear in the MKV through to the final one? That is completed, but if you would like more logic on it (language filter, etc), I can play with it some more.

    Personally, I do not like to AutoBurn the subtitles, so I just left all of the original subs in without it on. I did have to go into Plex language settings to make sure subtitles are on for foreign language segments. It seems to work, but I only have one movie ripped with forced subs so it is a limited sample size. Do you prefer to AutoBurn them? I can probably make that some personal setting is people want to do it both ways.

    I also attempted to make a separate simple config file for people to use, instead of needing to mess with environment variables or the Vagrantfile. This seems to work well for me, and I have attempted to edit it in multiple different text editors to break it, but have been unable to on Windows at least. It could probably use some cross platform testing. This seems like a logical solution to me, but would be really interested in if that is a direction you like?

    Both of those have been issues as pull requests, and should be ready for testing either independent or combined. I run them merged on my machine. I have no OS X to test on at the moment, and have not moved any files to test on Ubuntu with yet, but that is on my todo list.

  • Okay i have this installed on OS X Maverick, i will be testing on a mid 2011 mini with a 2.7 quad core i7 and 4gb RAM.

    First thing i ran into was maverick's requesting the Developer Pack to be installed which is a simple yes click and it went away and did what was needed.

    Now its been a long time since i have had to use command line so I'm a little rusty but i found that i needed to issue the 'vagrant init' command before anything else like copying the git link would function.

    Once done the script worked pretty well, i dod struggle to find the tested transcoder file but that could be my finder settings, it was hidden in the user directory which apple likes to hide it seems, I'm sure it used to be easier to locate in previous os versions like tiger.

    One change i did need to make to the VM spec was that its default RAM was 4gb which was all the machine has and thus had nothing left for the OS X system so i have dropped this down to 2gb for now.

    I will start testing this week but i need to configure the output to mp4/m4v which I'm guessing is in the python script? is it as simple as changing mov to mp4?

  • @GR$EME: That is very strange that you needed to do anything with vagrant before git worked. They should be completely independent of each other. It could be something on OS X, which I have not used in quite some time, though.

    I am hoping to move more towards having a config file which will make it easier to change the ram, and other settings, but still waiting for feedback from @will: on that part. Did you have any issues changing it?

    Just curious, why change the format to MP4? You do lose some features going that route. MP4 cannot have multiple sound tracks included in it, so if you are interested in the commentary or multiple languages, then you will need to stick to MKV. MP4 also misses out on AC3 audio, and chapter information, which are not as big a deal, but can be annoying. If you are not interested in that, then it should be a very simple change.

    Thew following should work, but comes with no guarantee, as I have not tested it. If you want MP4 then remove '--mkv' from line 40 after 'TRANSCODE_OPTIONS.' If you want it M4V then change it to '--m4v'. You also would need to change line 259 after 'output_name' there is '.mkv' which will need to be the same as your decided format.

    If there is more interest in changing the output type, it should be possible to make a config file for output settings as well, but don't want to over-complicate things if not necessary. Just need to know what people are interested in.

  • @Bedon292 Im not sure what the issue was but trying to use vagrant up returned an error stating it needed to be initiated first, as i said I'm a little rusty on my command line.

    It also appears i don't know my OS X version as its yosemite not mavericks lol

    As for the mp4 over mkv its more to do with my whole eco system, unfortunately my main viewing of movies is through apple TV's and mkv isn't supported, i did consider transcoding on the fly from my synology but its to intensive and still has issues with the apple tv's

    I did make a couple of short cuts to start and stop the transcode VM using automator though.

  • @GR$EME: I figured that might be the case. Plex is great for transcoding on the fly, but you do need to have a machine which can spare the power to do it all the time. Not sure what is required for plex and apple tv, but I think it works fine. It may be worth looking in to at least, I know I love plex. Other awesome part is being able to watch it anywhere. I am out of town this weekend, but I can still watch all of my movie and TV through plex. Didn't even require any extra configuration or firewall issues or anything.

  • Im struggling to get this to run? The VM is running but the script doesn't seem to be. I have edited the vagrant script to change the memory as it was forcing 4gb when i only have 2gb usage, i have also tried changing from 4 to 2 cores

    Any ideas? it doesn't even generate a log file?

  • @GR$EME: When you run `vagrant up` does it give you any sort of warning or error messages? Not sure why it would not be running otherwise. Need a bit more information to track down where the error is happening. A few things to check in VirtualBox Manager:

    Under Tested Transcoder is there a green arrow and 'Running'?

    In Tested Transcoder, Shared Folders, is your folder set correctly?

    If those are both good, check to see if input/output/work/completed-originals folders are created within that folder? If they are, there should also be a log there.

  • @Bedon292

    Yes the VM is running and the folders were generated by the initial script. Im just not seeing any logs in the output folders.

  • If i launch from VM directly its looking like the VM machine is going to a login and waiting

  • okay i deleted and have re installed and its now working. I had used automator to make start and stop scripts which looked to be functioning but to be sure i tried using the vagrant up cmd from terminal and it was trying to reinstall the whole package WTF

    What is the correct command to start? i had assumed it was to cd to the correct directory then run vagrant up?

  • Is it supposed to just sit at a login screen? I got it to work the first time, but ever since it will not go past a login. I can ssh into it without issue and without being prompted for a login, so I think the RSA key is fine. I have tried starting over a few times and even uninstalled and reinstalled Vagrant and Virtualbox. Any ideas?

    I'm on Windows 8.1. Also, I wouldn't recommend installing Virtualbox on Win 10. I couldn't boot into Windows after installing it and had to roll back to 8.1. (Obviously no fault of this project.) Thanks!

  • @nickernzen: @GR$EME: I always run it through `vagrant up` any other way will just sit at the login screen as described. Vagrant machines are not meant to be turned on and off through VBox directly. I imagine this also means that it is not logged in, and will not be running any of the scripts, but I am not sure if that is true.

    If you run the `vagrant up` command and it starts to install again it sounds like the previous install did not complete correctly. It will go through a handful of messages, checking if things are the latest version or not, but should not redownload everything.

    `vagrant halt` is the command to run if you are trying to turn the machine off. If you don't run this, I am not sure what the consequences may be, but that could cause issues.

  • I'm having trouble getting the 'input', 'output', 'work', and 'completed-originals' folders to show up after the instalation instructions. When I run vagrant up, I get this from the command line. The virtual machine in VirtualBox always says Powered off, not sure if that has anything to do with my problem.

  • I've ripped six new Blu-Rays with MakeMKV and used the VM to transcode the MKVs. This has been fantastic so far. I've only noticed two issues and one potential problem for the future. Both of the issues were unrelated to the Tested transcoder project, but I'll share them in case someone notices the same problems.

    The first was, I think, a Virtual Box problem. It could have been related to Vagrant, but probably not. I took down the VM with 'vagrant halt' to modify the processor settings and then everything stopped working. It took me some time to realize Virtual Box had dropped the shared folder. The 'Make Permanent' setting didn't stick the first time. Then again, maybe I double-clicked and never noticed. Watch for this in case you're ever wondering why the 'work' folder remains empty after you drop-in a video.

    The second issue was a MakeMKV/Blu-Ray/user issue. I ripped and processed Mockingjay Part I and it passed my spot check. However, we later learned many of the scenes were out of order. It played for several minutes and then would abruptly jump ahead (or back) to another scene.

    This is new one for me. This Blu-Ray is the first movie I've purchased that displays dozens of duplicate entries in MakeMKV. All them are 16 chapters, 24.1 GBs, and display the same audio track options, but each one has a unique segment map. Not only that, it seems each disc is different based on the merchant (e.g. Target, Amazon, etc). Apparently this sort of playlist obfuscation is normal for some of the newer discs.

    This Blu-Ray is also notable for another reason: TrueHD 7.1 Atmos audio. It's one of the first to have it. I don't get any sound when I play the video when this audio track is selected. Plex doesn't support Atmos yet. Neither does VLC or a lot of receivers. I want to include it for the day it is supported, but I have no way of testing the audio track in the video produced by the transcoder. I won't hear any audio even if it's OK. The good news is Plex seems to detect it can't play the audio and switches to audio track 2, the DD 5.1 track, so maybe it is passing through the transcoder without any trouble.

    If I run into any problems, I'll be sure to mention them. It's been a great tool so far. I'll be using it to shrink my other MKVs.

  • One additional comment/question: The logs can take a long time to load, especially after the transcoder has been running for a while and with the CPU usage (probably) at or close to 100%. I use PowerShell to get real-time updates. If you'd like to check-up on the progress as you walk by the PC, do this:

    1. Open PowerShell and cd into the 'work' folder.

    2. run Get-Content log.log -Wait

    You can do the same thing with tail on Linux or OS X.

    This works great, except when I have an MKV with lots of these:

    [NULL @ 0x7fced80057c0] major sync info header checksum error

    That line will repeat several dozens times between % completion updates. I can't figure out the problem. The input MKV is fine. The final output looks fine. Why the errors? I saw this with three of the six MKVs I have processed.

  • @Mendelson9: The log you pasted looks exactly like mine, with one exception. Mine is mounting the share folder to `/media/transcoder`, and yours is not. I do not think that would cause the whole VM to crash, but have not tried it without it. Can you try and boot the image directly in Virtual Box? Not sure why it is not staying on, when you do `vagrant up` but that might give you some insight into it.

  • @Saucy:

    (Post 1) Thanks for mentioning the track obfuscation. I did not realize that it might cause issues. Now I just have to remember what disks those were and check if they actually ripped right...

    (Post 2) Not sure why there is the checksum error you are getting. Those logs are piped straight from Handbreak though, so you may just want to check out their help section to see what it means. It may actually be related to the Atmos audio. It looks similar to errors people are having with it. Is it happening on the MKVs with it?

    (Post 3) I have put probably 60 or so movies through the transcoder, and have not run into that error myself. I have been thinking about buying Pacific Rim amyways, rather than just watching it on HBO. I may try and grab it and see if I can't duplicate the error myself. Not sure it will be possible though. Agian that is going to be from handbreak, and looks like actually an FFmpeg error. Can't seem to find anyone else with the same error though.

  • @dark_james: There should be a Make Permanent box at the bottom of the Shared Folder. You may need to add it, uncheck everything, hit OK, and then edit it again?

    If that doesn't work, let me know and I can try to dig in more. I submitted a pull request that should make it easier to configure, but still waiting on more direction from @will for it.

  • @dark_james: If you figure that out, please post. I haven't been able to figure out why the Make Permanent option is sometimes there and sometimes not.

  • @bpmckim: Not sure what Will did, but this is what I have done. Also not sure what your experience is, so will try to go step by step:

    • In freenas, go to the jails tab and click Add Jail.
    • Click Advanced Mode.
    • Template Dropdown choose VirtualBox, change any other settings you want
    • At the bottom is a black folder icon for storage. Add some shared storage to it. Make sure the destination is `/media/transcoder`
    • Once it is up and running there is a shell icon at the bottom, click it
    • Once in the shell run `pkg update` and `pkg upgrade`
    • Then `pkg install git vagrant` I think git might already be installed, but do it to be safe
    • From there you should be able to `git clone` as the normal instructions say and be good to go.
  • @bpmckim: I have actually not checked in phpVirtualBox, so I do not know. Is it creating the folders and working though? If so I wouldn't worry about it too much. I will try and play around with it this weekend though, see what I come up with.

  • Tried this again but ran into the same problem as @dark_james, with the Make Permanent folder option not showing. Giving up and going back to MakeMKV + Handbrake.

  • @wyterabbit: You haven't mounted the folders in VirtualBox properly, I think,

  • @wyterabbit: Does that video have different permissions than the other ones? That specific file may not be accessible to VirtualBox. Bit of a long shot, but worth a look. Otherwise I would consider going to Don Melton's github and submitting a issue for it, since I cannot find a reason for that.

  • Can some one help me with the processor limits? It's currently only using 4 threads, and I want to use them all. I usually make a bunch of MKVs and then set it all in motion when I go to bed, so no worries about it slowing me down. Thanks