Hey guys, here's the place to put feedback on the Tested transcoder project. We're specifically looking for bugs (things that don't behave as you'd expect), problems with the install or setup process, or anything else that isn't behaving as you'd expect.
The transcoder works best with MKV files and can handle multiple audio and subtitle tracks. By default, we're leaving the audio streams intact and only adding forced subtitles to the compressed file. You can tweak the command line that gets passed to Handbrake-cli if you want to burn subtitles in or change the audio formats too, but I haven't written docs for that yet.
The VM should run on any system that can run Virtualbox. You'll also want to turn on all the virtualization instructions for your CPU in the motherboard, or your performance will be really bad.
The instructions for installation are available in the readme file on Github, but they're pretty rough. Right now, we assume you know a fair amount about running virtual machines and ripping discs. If you don't know about at least one of those things, it might be tough to get this working with the info we've provided. We'll have a more complete how-to next week.
Lots of the heavy lifting here comes courtesy of Don Melton's great video transcoding scripts. Much of what we're doing is just automating the setup and fiddlier bits of his scripts.
Please post your comments and feedback below or on the Github page.
I went through the install process and attempted to make it a bit more friendly to new users.
It definitely works pretty well so far, it at least gets the video and audio correct.
Having some issues with the subtitles though, which I remember you mentioning. It seems to only want to keep the forced ones. Can it just be set to pass through all subtitles, or ones that match a list of languages even? Then, as long as they were ripped, it should be good.
I actually just realized that it is doing something a bit odd with the audio. It is actually adding an additional track which did not exist before. It seems to be taking the AC3 surround audio and making an AAC stereo track as well. Is this intentional, or is it something the Don Melton scripts are doing on their own.
I know I am not interested in having anything other than the original audio tracks, if I need something else Plex can handle it.
I am going to play around with it to see if I can get it to keep only original audio and add all subtitles for now.
@Bedon292: All of that stuff should be addressable in the command line. You can adjust that in transcoder.py, but make sure your text editor uses unix style newlines.
@will: Right, I dug through and found those settings. I think I have that all in a working state.
What are your thoughts on allowing all subtitles which appear in the MKV through to the final one? That is completed, but if you would like more logic on it (language filter, etc), I can play with it some more.
Personally, I do not like to AutoBurn the subtitles, so I just left all of the original subs in without it on. I did have to go into Plex language settings to make sure subtitles are on for foreign language segments. It seems to work, but I only have one movie ripped with forced subs so it is a limited sample size. Do you prefer to AutoBurn them? I can probably make that some personal setting is people want to do it both ways.
I also attempted to make a separate simple config file for people to use, instead of needing to mess with environment variables or the Vagrantfile. This seems to work well for me, and I have attempted to edit it in multiple different text editors to break it, but have been unable to on Windows at least. It could probably use some cross platform testing. This seems like a logical solution to me, but would be really interested in if that is a direction you like?
Both of those have been issues as pull requests, and should be ready for testing either independent or combined. I run them merged on my machine. I have no OS X to test on at the moment, and have not moved any files to test on Ubuntu with yet, but that is on my todo list.
Okay i have this installed on OS X Maverick, i will be testing on a mid 2011 mini with a 2.7 quad core i7 and 4gb RAM.
First thing i ran into was maverick's requesting the Developer Pack to be installed which is a simple yes click and it went away and did what was needed.
Now its been a long time since i have had to use command line so I'm a little rusty but i found that i needed to issue the 'vagrant init' command before anything else like copying the git link would function.
Once done the script worked pretty well, i dod struggle to find the tested transcoder file but that could be my finder settings, it was hidden in the user directory which apple likes to hide it seems, I'm sure it used to be easier to locate in previous os versions like tiger.
One change i did need to make to the VM spec was that its default RAM was 4gb which was all the machine has and thus had nothing left for the OS X system so i have dropped this down to 2gb for now.
I will start testing this week but i need to configure the output to mp4/m4v which I'm guessing is in the python script? is it as simple as changing mov to mp4?
@GR$EME: That is very strange that you needed to do anything with vagrant before git worked. They should be completely independent of each other. It could be something on OS X, which I have not used in quite some time, though.
I am hoping to move more towards having a config file which will make it easier to change the ram, and other settings, but still waiting for feedback from @will: on that part. Did you have any issues changing it?
Just curious, why change the format to MP4? You do lose some features going that route. MP4 cannot have multiple sound tracks included in it, so if you are interested in the commentary or multiple languages, then you will need to stick to MKV. MP4 also misses out on AC3 audio, and chapter information, which are not as big a deal, but can be annoying. If you are not interested in that, then it should be a very simple change.
Thew following should work, but comes with no guarantee, as I have not tested it. If you want MP4 then remove '--mkv' from line 40 after 'TRANSCODE_OPTIONS.' If you want it M4V then change it to '--m4v'. You also would need to change line 259 after 'output_name' there is '.mkv' which will need to be the same as your decided format.
If there is more interest in changing the output type, it should be possible to make a config file for output settings as well, but don't want to over-complicate things if not necessary. Just need to know what people are interested in.
@Bedon292 Im not sure what the issue was but trying to use vagrant up returned an error stating it needed to be initiated first, as i said I'm a little rusty on my command line.
It also appears i don't know my OS X version as its yosemite not mavericks lol
As for the mp4 over mkv its more to do with my whole eco system, unfortunately my main viewing of movies is through apple TV's and mkv isn't supported, i did consider transcoding on the fly from my synology but its to intensive and still has issues with the apple tv's
I did make a couple of short cuts to start and stop the transcode VM using automator though.
@GR$EME: I figured that might be the case. Plex is great for transcoding on the fly, but you do need to have a machine which can spare the power to do it all the time. Not sure what is required for plex and apple tv, but I think it works fine. It may be worth looking in to at least, I know I love plex. Other awesome part is being able to watch it anywhere. I am out of town this weekend, but I can still watch all of my movie and TV through plex. Didn't even require any extra configuration or firewall issues or anything.
Im struggling to get this to run? The VM is running but the script doesn't seem to be. I have edited the vagrant script to change the memory as it was forcing 4gb when i only have 2gb usage, i have also tried changing from 4 to 2 cores
Any ideas? it doesn't even generate a log file?
@GR$EME: When you run `vagrant up` does it give you any sort of warning or error messages? Not sure why it would not be running otherwise. Need a bit more information to track down where the error is happening. A few things to check in VirtualBox Manager:
Under Tested Transcoder is there a green arrow and 'Running'?
In Tested Transcoder, Shared Folders, is your folder set correctly?
If those are both good, check to see if input/output/work/completed-originals folders are created within that folder? If they are, there should also be a log there.
Yes the VM is running and the folders were generated by the initial script. Im just not seeing any logs in the output folders.
If i launch from VM directly its looking like the VM machine is going to a login and waiting
okay i deleted and have re installed and its now working. I had used automator to make start and stop scripts which looked to be functioning but to be sure i tried using the vagrant up cmd from terminal and it was trying to reinstall the whole package WTF
What is the correct command to start? i had assumed it was to cd to the correct directory then run vagrant up?
Is it supposed to just sit at a login screen? I got it to work the first time, but ever since it will not go past a login. I can ssh into it without issue and without being prompted for a login, so I think the RSA key is fine. I have tried starting over a few times and even uninstalled and reinstalled Vagrant and Virtualbox. Any ideas?
I'm on Windows 8.1. Also, I wouldn't recommend installing Virtualbox on Win 10. I couldn't boot into Windows after installing it and had to roll back to 8.1. (Obviously no fault of this project.) Thanks!
@nickernzen: @GR$EME: I always run it through `vagrant up` any other way will just sit at the login screen as described. Vagrant machines are not meant to be turned on and off through VBox directly. I imagine this also means that it is not logged in, and will not be running any of the scripts, but I am not sure if that is true.
If you run the `vagrant up` command and it starts to install again it sounds like the previous install did not complete correctly. It will go through a handful of messages, checking if things are the latest version or not, but should not redownload everything.
`vagrant halt` is the command to run if you are trying to turn the machine off. If you don't run this, I am not sure what the consequences may be, but that could cause issues.
I'm having trouble getting the 'input', 'output', 'work', and 'completed-originals' folders to show up after the instalation instructions. When I run
vagrant up, I get this from the command line. http://cl.ly/aQc3/image The virtual machine in VirtualBox always says Powered off, not sure if that has anything to do with my problem.
I've ripped six new Blu-Rays with MakeMKV and used the VM to transcode the MKVs. This has been fantastic so far. I've only noticed two issues and one potential problem for the future. Both of the issues were unrelated to the Tested transcoder project, but I'll share them in case someone notices the same problems.
The first was, I think, a Virtual Box problem. It could have been related to Vagrant, but probably not. I took down the VM with 'vagrant halt' to modify the processor settings and then everything stopped working. It took me some time to realize Virtual Box had dropped the shared folder. The 'Make Permanent' setting didn't stick the first time. Then again, maybe I double-clicked and never noticed. Watch for this in case you're ever wondering why the 'work' folder remains empty after you drop-in a video.
The second issue was a MakeMKV/Blu-Ray/user issue. I ripped and processed Mockingjay Part I and it passed my spot check. However, we later learned many of the scenes were out of order. It played for several minutes and then would abruptly jump ahead (or back) to another scene.
This is new one for me. This Blu-Ray is the first movie I've purchased that displays dozens of duplicate entries in MakeMKV. All them are 16 chapters, 24.1 GBs, and display the same audio track options, but each one has a unique segment map. Not only that, it seems each disc is different based on the merchant (e.g. Target, Amazon, etc). Apparently this sort of playlist obfuscation is normal for some of the newer discs.
This Blu-Ray is also notable for another reason: TrueHD 7.1 Atmos audio. It's one of the first to have it. I don't get any sound when I play the video when this audio track is selected. Plex doesn't support Atmos yet. Neither does VLC or a lot of receivers. I want to include it for the day it is supported, but I have no way of testing the audio track in the video produced by the transcoder. I won't hear any audio even if it's OK. The good news is Plex seems to detect it can't play the audio and switches to audio track 2, the DD 5.1 track, so maybe it is passing through the transcoder without any trouble.
If I run into any problems, I'll be sure to mention them. It's been a great tool so far. I'll be using it to shrink my other MKVs.
One additional comment/question: The logs can take a long time to load, especially after the transcoder has been running for a while and with the CPU usage (probably) at or close to 100%. I use PowerShell to get real-time updates. If you'd like to check-up on the progress as you walk by the PC, do this:
1. Open PowerShell and cd into the 'work' folder.
2. run Get-Content log.log -Wait
You can do the same thing with tail on Linux or OS X.
This works great, except when I have an MKV with lots of these:
[NULL @ 0x7fced80057c0] major sync info header checksum error
That line will repeat several dozens times between % completion updates. I can't figure out the problem. The input MKV is fine. The final output looks fine. Why the errors? I saw this with three of the six MKVs I have processed.
I've run into an issue with several of my Blu-Rays: Pulp Fiction, 9, and Pacific Rim.
The transcoding ends with an error. Sometimes it's in the 90-95% range, but it's happened at 47% (the earliest failure so far). This error occurs several times, once for each different track number. I saved the log files, but here's a small sample.
Encoding: task 2 of 2, 47.40 % (11.98 fps, avg 12.15 fps, ETA 01h22m27s)ERROR: avformatMux: track 0,
av_interleaved_write_frame failed with error 'Operation not permitted'
ERROR: avformatMux: track #
@will Is this something you have encountered? I assume it's a problem related to the scripts. Maybe something I need to alter for these files.
To rule-out an issue with the input file, I've re-ripped the Blu-Rays. I still run into the same errors with these movies.
@Mendelson9: The log you pasted looks exactly like mine, with one exception. Mine is mounting the share folder to `/media/transcoder`, and yours is not. I do not think that would cause the whole VM to crash, but have not tried it without it. Can you try and boot the image directly in Virtual Box? Not sure why it is not staying on, when you do `vagrant up` but that might give you some insight into it.
(Post 1) Thanks for mentioning the track obfuscation. I did not realize that it might cause issues. Now I just have to remember what disks those were and check if they actually ripped right...
(Post 2) Not sure why there is the checksum error you are getting. Those logs are piped straight from Handbreak though, so you may just want to check out their help section to see what it means. It may actually be related to the Atmos audio. It looks similar to errors people are having with it. Is it happening on the MKVs with it?
(Post 3) I have put probably 60 or so movies through the transcoder, and have not run into that error myself. I have been thinking about buying Pacific Rim amyways, rather than just watching it on HBO. I may try and grab it and see if I can't duplicate the error myself. Not sure it will be possible though. Agian that is going to be from handbreak, and looks like actually an FFmpeg error. Can't seem to find anyone else with the same error though.
Hey guys I'm also having issues. I set everything as per the instructions, though the "Make Permanent" shared folder option was not visible to me.
When I boot up the VM via 'vagrant up' it doesn't seem to mount the shared folder, is that intended?
I then waited 5 minutes and no folders were created within the shared folder. Any ideas? Posting my log below:
PS A:\tested-transcoder> vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Checking if box 'ubuntu/trusty64' is up to date...
==> default: Clearing any previously set forwarded ports...
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
default: Adapter 1: nat
==> default: Forwarding ports...
default: 22 => 2222 (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
default: SSH address: 127.0.0.1:2222
default: SSH username: vagrant
default: SSH auth method: private key
default: Warning: Connection timeout. Retrying...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Mounting shared folders...
default: /vagrant => A:/tested-transcoder
==> default: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> default: to force provisioning. Provisioners marked to run always will still run.
==> default: Running provisioner: shell...
default: Running: inline script
==> default: stdin: is not a tty
==> default: Restarted supervisord
@dark_james: There should be a Make Permanent box at the bottom of the Shared Folder. You may need to add it, uncheck everything, hit OK, and then edit it again?
If that doesn't work, let me know and I can try to dig in more. I submitted a pull request that should make it easier to configure, but still waiting on more direction from @will for it.
@Bedon292 Hmm... I don't recall seeing that when I tried it on Windows 7.
Oh! I'm on Linux right now and I tried running vagrant halt and closing Virtualbox. Ran vagrant up and re-opened Virtualbox (VB on Linux was actually showing as 'running' it wasn't doing that on Windows). Now I magically have the Make Permanent checkbox and the folders appeared.
I'll see if I can reproduce this on Windows 7 as that's where I'd primarily like to do the rip/transcode process.
Thanks for the help!
@dark_james: If you figure that out, please post. I haven't been able to figure out why the Make Permanent option is sometimes there and sometimes not.
Found the video about making FreeNAS box which led me here. Great article and great video, it got me up and running. But I haven't seen a writeup on getting the transcoder script to run in virtualbox in a jail on FreeNAS. I got it work on my macbook no problem but would like to offload the heavy lifting to a higher end CPU. I don't need a step by step just a general idea of how to make it happen.
1. Export appliance from macbook into freeNAS?
2. Can I do this without vagrant?
Thanks for the help.
@bpmckim: Not sure what Will did, but this is what I have done. Also not sure what your experience is, so will try to go step by step:
@Bedon292 Got the VM up and running on FreeNAS but it's not showing in the phpVirtualBox GUI. Think is has to do with two different users, vbuser and root. Thoughts?
@bpmckim: I have actually not checked in phpVirtualBox, so I do not know. Is it creating the folders and working though? If so I wouldn't worry about it too much. I will try and play around with it this weekend though, see what I come up with.