Transcoder Feedback

Created by will on Feb. 20, 2015, 7:48 p.m.
  • Sticky
  • @wvc: This piece just automates the Handbrake part of the process. Which I find to be very valuable. You still use MakeMKV to rip, and Plex to play. What OS are you running on the machine you are trying to use this on, and what are you having trouble with?

    @MrCoe1989 running vb and vagrant inside docker, seems to be a very strange way of doing things. Its virtualization within virtualization and who knows if there will be a performance hit for that. You can definitely run the tool right inside docker. I have most of a Dockerfile done, but have not tested it yet. I am also planning on writing a quick .sh which will get everything done as well.

  • Bedon 292 It is a bit odd but since my server has it's own hypervisor built in theres no way for me to run the vagrant command to start the vm once i import it into the server :-(, dockerfile sounds exciting though :-)

  • I'm on Windows 8.1 (haven't quite switched to 10 yet, too many things going on at home to focus on upgrading the computer yet). I understand MakeMV and Plex and have a moderate handle on Handbrake -it's the Virtual Machine part where I don't even know where to start. I understand, conceptually, what a virtual machine is but have never set one up (never used Virtual Box). That's why I was looking for a sort of 101 course - I feel like I don't know enough to even start on this process yet. With all the cartons I've collected for the kids (Looney Tunes, Disney, and the highly recommended Danger Mouse), I'd love a "set it and forget it" solution for mass ripping and processing.

  • @wvc: Virtual machines are actually a very simple concept, which I am sure you already grasp. They are just a virtualized version of an operating system running on your machine. In the case of the transcoder, it is a version of ubuntu running inside your session of Windows.

    VirtualBox is a version of VM software, it is a simple cross platform option. VMWare and HyperV are other similar software, which can have a lot more options and control. I would suggest installing VBox and starting up a machine to play with, just to understand more what it can do. https://www.virtualbox.org/manual/ch01.html

    Vagrant just simply wraps the creation of the VM up into a script. It allows you to have a easy, and repeatable provisioning. It is meant for developers to be able to distribute simple small scripts instead of full images, and still have the exact same setup on multiple machines. It also works great for this case where we all want to be running the same VM.

    In this case, it sets up an ubuntu image, installs the required packages for handbreak, and sets up a python script to constantly be looking at the folder for new files and passes them to handbreak.

  • I can import the vm into my hypervisor, is there a way to log into the black box ?

  • Tested-Transcoder in a FreeNAS Jail Guide:

    https://goo.gl/lnLTH5

    While the document is locked from edits please feel free to make a copy, fix any bugs, make it prettier, or whatever and send/post it back here.

    So, I have done about 12 hour's worth of thrashing, screaming and cursing in general. All so that I could get the transcoder up and running on FreeNAS properly. While it is now running perfectly there is a lot of refinement that needs to go into the install. For instance, the only way I could get phpvirtualbox to show the VM was to run vboxwebsrv and phpvirtualbox as root.

    .. and wait for it..

    Now that all you security conscious people out there are done screaming, it was simply because vagrant requires root (through sudo or root login).

    As I am no Linux/FreeNAS savant, I could use some help especially on securing the install (not having to run everything as root)

    Although I admit, there is a possibility it could be months before I look at this again, I will try and refine this guide ( https://goo.gl/lnLTH5 ) to make it easier to read and in general easier on the eyes.

    Also feel free to make a copy, fix any bugs, make it prettier, or whatever. All I ask is that you either send me a copy, or post it back here on the forum.

  • @Bedon292 I got your docker version of the transcoder to install it creates the files input work etc the only problem i've found is permissions, if i try and write an mkv to the input folder i get a permission denied error ?

  • @Bedon292 if i specify /media/transcoder as the container volume then map a host path to my share the folders are created but i don't have permission to write to the input file, thought a screenshot may help

  • @Valherun: I briefly mentioned it in an earlier post, but what I did was use the phpVirtualBox template which is already configured to work. Unfortunately vagrant does not work with it, so I had to set up the scripts manually. Though I will make a shell script for it to simplify the process.

    Under the Jails tab at the top there is a tab for Templates, and if it does not already exist, you can add http://download.freenas.org/jails/9.2/x64/freenas-virtualbox-4.3.12.tgz Then when you add a jail hit advanced and choose that template. As soon as it boots phpVirtualBox is working, and you can go from there.

    Some day I am going to get it right, but I have fought with it for two weekends so, its an eventually thing. I have it transcoding right now, just not as nice a setup.

  • @MrCoe1989: Thats interesting, did not expect a permission issue there. Can probably modify the script to set the permissions to totally open on the directories for further use. For now though, you could probably try to get rid of those folders, and create them yourself with open permissions.

    What GUI is that by the way? I do all my docker stuff from the CLI and do not recognize it.

  • The Gui is for an unraid server, docker is part of it :-)

  • If you could set it to totally open id be happy to test it for you, in the screenshot ive posted above i've specified the container volume as /media/transcoder and then mapped it to my host drive, i got around the permissions error by specifying the container volume as something different "/media" for example. it created the input out files etc but it didn't start converting so i assumed the path needed to be /media/transcoder

  • @Bedon292: I understand, I've been fighting on and off with it for about a month now.

    It boils down to FreeNAS 9.3's virtualbox template being outdated and therefore not allowing the install of vagrant without updating a bunch of other packages, which in turn breaks virtualbox. While, I tried a number of times to get things running using the template and vagrant to no avail. However, Saturday I checked here on the forums to see if anyone else had made progress and came across your manual installation method.

    So I set to work doing a bit of research (again I am a Linux/BSD Noob in my opinion) when I ran across two things, the virtualbox-ose-kmod package (the current required kernel modules for FreeNAS and Vbox) as well as cyberjocks post for the pre-vbox-template days. So I gave it a shot to get things up and running albeit cyberjock and 99% of the rest of the FreeNAS forums said.. "nope not possible to update the template."

    I can confirm that updating the virtualbox template is possible, but there is more to doing it this way instead of just starting with a fresh standard jail install. Although it could just be my system that broke about a dozen dependencies that had to be fixed before everything would work again.

    That all being said I still may end up switching to a manual method such as yours if I cannot find workarounds for having to run everything as root.

  • @Bedon292 You're a legend mate, once i deleted then replaced the folders the permissions error went away and the dock app works beautifully, encoding speed is like lightning and the quality is great :-)

  • @MrCoe1989: Awesome, yeah I will mod the script to set folder permissions so further builds with not have that issue. Need to work on further integration with the vagrant version, because the script has a few modifications. But once that works I can make a shell script to do a local install as well. Hopefully all from the same git, and have 3 ways for it to run as a black box.

  • @Valherun: I have been contemplating the idea of digging up an older version of the vagrant package which does not need to upgrade the other packages. Not sure the feasibility of it, but it could be a solution as well. Great job figuring out how to get it up and running either way. Its a real pain.

    I wouldn't be too worried about running it as root in VB in the jail, its still two levels of protection from the real OS. First whatever would have to escape VB, and then it would have to escape the jail as well. It I were running it directly in VB on an Ubuntu box or something I would be much more concerned. Also, now that the script is running you can remove the network connectivity to the jail as well since it is no longer needed just for that little bit more of safety.

  • So having an oddity/problem -

    Even after adding the shared folder and clicking "make permanent", if I stop/restart the vagrant box, I have to re-add the folder into virtual box. Is there something else that I need to adjust?

    Originally ran this on OS X Yosemite, but the issue still happens in OS X El Capitan.

  • @saintnicster: You are doing everything just fine, its vagrant resetting the shared folders that is causing the "make permanent" fail to work.

    @Bedon292: You do bring up a good point with disconnecting network activity. Only downside for me is my personal dislike of the WebUI's shell.

  • @saintnicster: I never ran into the problem on Windows at least. May be a Mac thing? Although I did end up making a modified version that has settings in Vagrant for it all, which should solve that.

    @Valherun Yeah the web UI definitely sucks, but never have to actually see it once its up and running either.

  • @Valherun: So it's only just now hitting me, but since the scripts really don't seem to update that often, is there much of a benefit to using Vagrant to boot it every time? Or would I be fine with just directly booting the Virtual Box?

    @Bedon292 - Yeah, was thinking that maybe it was some weird permissions thing, but no clue otherwise.

  • @saintnicster: I did a quick test (on windows) of starting the Tested-Transcoder VM without vagrant and it did start just fine and start transcoding as normal.

    Doesn't seem to have any problem although it may become slightly out of date if you always do it this way, as vagrant I believe is pulling all the most-current scripts and packages when it starts.

  • That's cool, I actually thought vagrant was needed every time due to user / start issues. The scripts basically never change though, so there is no real need to update them

    In fact I would actually intentionally avoid updating them once they are working the way you want. While an update should not break anything previous, there is always the possibility and no reason to change if its already working.

  • I've just got my new FreeNAS machine up and running, which I built with a Xeon E3-1231 in the hope that I would have enough CPU power for some encoding tasks. I've been following this thread and it looks like getting the Transcoder Project up and running on a FreeNAS system is not straightforward, but there has been some progress over the past week, especially with the potential to do without Vagrant.

    Is there any chance of someone putting together a FreeNAS install guide for the Transcoder Project, based on the most recent information in this thread? I would really appreciate it.

    Also, I can confirm that this works 'out of the box' under Windows 10. It was a breeze to set up! Just make sure you don't have 'Hyper-V' turned on in Windows.

  • Hey guys, is there a way to encode Blu-ray videos at 720p? I've got the new box set of The Wire and this show looks fine in 720p.

    I tried to add the '--720p' flag to the 'transcode-video.sh' command in transcoder.py but that had no effect.

    Any help would be much appreciated.

  • @nhall: Multiple install guides and some more tweaks are definitely something I want to get to, just have not had time. Valehrun does have a guide he made further up the page though.

    Not sure about 720p bit, the purpose of this was highest quality possible so I never looked. Is that what it says to do on Don Melton's page?

  • Has anyone run into trouble recently with MakeMKV? It is telling me that the evaluation period has expired and that I need to purchase a copy. In the past, I've been able to re-install and reset the evaluation period....

  • I just paid for it. As soon as I started using it I knew it was something I wanted to keep around. It is a reasonable price, and I wanted to support him to make sure he keeps it up to date. Over a library of movies, the price is barely noticeable for something so useful.

  • In fact, I realized that this is something a little strange audio. "In fact, adding more track that was not there before.

  • I just started using this whole setup a couple nights ago and I have a couple questions. I'm pretty new to this stuff, but I've read the whole thread and aside from a mention about 9 months ago I never found a resolution to my problem. I'm ripping my collection for use within my Apple ecosystem, but I can't seem to get the audio to work on my iPad. My process has been as follows...

    Rip the DVD with MakeMKV

    Throw the file over to the "input" folder

    After that finishes, I open it with MP4Tools to convert it into an MP4

    Run it through iDentify to get all the metadata

    Copy into iTunes

    ... I am able to play the movies on my MacBook Pro and Apple TV 4 with no problems (other than the optional subtitles are gone), but I get no audio on my iPad Air or iPhone 6s. I'm guessing that the video playback is defaulting to the 5.1 which the iPad and iPhone cannot decode. Is there another way to get the files into iTunes that includes the subtitles and both 5.1 and stereo audio tracks that will work for both iPad and Apple TV without using MP4Tools... something for m4v maybe? I'd love to be able to use the full 5.1 with my Apple TV, but also be able to use my iPad as well. I am also using Plex with the iTunes library (so the m4v files iTunes makes) but I guess I'm wanting to have it all haha. I've also tried to edit the transcode.py file to switch mkv to m4v, but I still got an mkv file in the output folder. Any help is appreciated! Thanks to everybody's hard work to make this possible, even with my limited knowledge of command line, I was able to get this setup the first time and running the way it's intended to be run.


  • @bdrob16: I can't answer anything about the MP4 side of things, but if you rip the stereo audio in MakeMK along with the 5.1 you will have both audio tracks available. You might be able to tweak the script into going straight to m4v, but I have not tried that yet. I need to re-rip a few movies that I pulled the wrong video from, so I will play with it when I do that.

    Another option would be just keeping them all as mkv in plex, and having plex do the transcoding on the fly to apple devices. I don't have any, but I do transcode on the fly to mobile devices at least and it works well.