HandBrakeCLI Automation Script (Transcoding Server)

I just finished my first unRaid build to replace an external drive for an HTPC.  The HTPC runs XBMC and unRaid will be serving content to it.  In addition to storing media, I built out the box to be a mid-range transcoding box.  Transcoding video files converts them from one codec or quality setting to another codec or quality setting.  For example, a main title from a DVD in its native format (mpeg2 for DVDs) often takes 6-7 GB of storage.  Using HandBrake, you can transcode that file into approximately 1.5-2 GB with very little change in picture quality.  This is due to the efficiencies in H.264 compression versus mpeg2.  As a bonus, many video cards found in HTPCs support hardware acceleration of H.264 video content.  In other words, the smaller, more compressed file will actually result in lower CPU usage than the original file.  Blu-Ray material is most often encoded in VC-1 or H.264 codecs natively, but some use mpeg2.  Blu-Ray rips are often well above 30 GB for one movie.  While some quality will be lost, a Blu-Ray rip can be transcoded into a file of 8-9 GB that is of the same quality or higher than cable or satellite HD programming.  The ability to reduce file sizes results in significantly less storage cost.

 

Transcoding, though, is often cumbersome and time consuming.  On a moderate processor (like the X4 605e below), transcoding one DVD rip might consume 1-1.5 hours.  Then you are left with a source file and a new compressed file.  If you are using media software, you need to replace the old file with the new file and tell the software to index the new file.  This is a process ripe for automation but not particularly simple.  Ideally, it should accomplish these steps:

 

• Search the array for movies and tv shows that can be transcoded
  
• Queue them up for transcoding, one at a time
  
• Replace the old file with the new
  
• Store the old content in a safe place, just in case
  
• Tell the media software to look for the new content
  
• Repeat
  

 

With such a system in place, you would rip the file to a known directory on the server.  The server would repackage each movie or episode automatically and without intervention.  You would only clean out the "replaced" directory periodically.

 

I thought my workflow might be of some use to those out there with similar needs.  I will apologize in advance for the overabundance of lists.  I find they are terribly useful, even when overused.

 

General Build Info

• AMD Athlon II X4 CPU
  
• 4 GB DDR3
  
• One drive outside the array (mounted with SNAP)
  
• Directories within the array:  mkv, mkvhd, movies, movieshd
  
• MKVs created using makemkvcon on a client machine saved to mkv or mkvhd within the array
  

 

Extra tools needed

• HandBrakeCLI 0.9.5 (an unMenu .conf file is available on the forum)
  
• Task Spooler (ts) (binary and source included in the zip)
  
• Build Environment (to compile Task Spooler once)
  

 

For those of you unfamiliar with Task Spooler, it provides the same functionality as batch with some advantages.  You can add jobs to be run, determine how many jobs run simultaneously, capture the output (and optionally email it), conditionally run jobs based on the outcome of other jobs.  If you want your computer to do one thing over and over sequentially, it is wonderful.  Yes, you could achieve something similar with batch, but batch waits until system load is low and lacks the reporting and control of ts.  And no, I didn't write ts and have never talked to the guy that did.  But it is possibly the most useful tool I have encountered.  If it were not for the fact that I have to type Yes, I would have used ts to run all my preclears.

 

UPDATE:  Task Spooler (ts) is now included in the zip file for the tc utilities.

 

Raw DVD MKV files are deposited into the mkv directory on the array.  Each MKV is in a directory with the movie title (year) as its name.  The MKV itself is the title with spaces repl

 

I name MKV files that have been transcoded deliberately.  This allows me to easily identify likely files that need to be transcoded.  This identification is performed in a script called tcrepack which searches for "unmarked files" on the array and queues files up for processing.  tcrepack identifies files on the array that are candidates for transcoding and run tcremkv on them somewhat like the following:

 

find /mnt/disk[0-9]+/movies | grep "\.mkv$" |grep -v H264|grep -v "trailer\."|sed "s/^/ts tcremkv \"/;s/$/\"/"|sh

 

I skipped some of the checking, but the line above gives you an idea of the core of the script.  This looks for files that are MKVs, but not trailers and not previously transcoded (H264 in the filename).  For each one of them it runs ts tcremkv filename.  Note that I looked for the files under the disk shares for a reason.  When I move the file to the replaced folder, I want it to be a quick move (same disk).  Knowing which disk it is on allows me to choose the replaced directory on the same disk.  I also check to see if you have a cache drive on the system and make some adjustments for efficiency.  Running tcremkv in ts allows me to transcode one file after another with all the horsepower on the box (using nice to make sure core unraid functions still get their share).  

 

This will result in a work queue in ts like the following:

• tcremkv /mnt/disk1/movies/Some Movie (1999)/SOME_MOVIE_00.mkv
  
• tcremkv /mnt/disk4/movies/This Movie (2004)/THIS_MOVIE_00.mkv
  
• tcremkv /mnt/disk3/movies/That Movie (2008)/THAT_MOVIE_00.mkv
  
• tcrepack
  

 

Jobs in ts run sequentially unless you tell them not to do so.  So these will run, one at a time.  When they are all completed, tcrepack will once again check for new candidates.  tcremkv, in turn, does a number of things:

 

• Run HandBrakeCLI with the output going to a disk outside the array and the input the array file
  
• Move the original file to an array directory called replaced, just in case there was some unusual error
  
• Move the completed file from the outside drive back to the same directory as the original file, but with _H264 added before the extension
  
• Uses wget to invoke a library update on my XBMC server, so it will pick up the new filename and codec
  

 

 

A very similar process is used for TV shows.  Once it is a properly named MKV, the same tcremkv script works on it.  The only difference in the process for BR rips is in the default HandBrakeCLI options.

 

I have added a line to the go file to copy all of my scripts + ts from /boot/tc/bin to the /usr/local/bin directory upon each boot.  I have added a cron job (under cron.daily) to run tcrepack and tcrepackblu (for HD files).  tcrepack calls itself (running ts tcrepack as its last step) in case more content has been added while jobs were running.  If tcrepack finds no candidates it will not call itself at the end.  This alleviates the possibility of a continuous loop.  In that case tcrepack would simply run when cron next initiated it.

 

I designed the scripts to make best use of resources and reduce drive contention.  The file (in movies) is read from an array disk by HandBrake with writes going to a disk outside the array.  The file is moved to the replaced directory on the same disk (just changing the filesystem entry, not rewriting the whole file).  The only full write to the array is that of the new (much smaller) file back to the same disk (which is obviously already spun up).

 

Any improvements or suggestions are welcome.  I would be glad to post the scripts if there is enough interest.  These were all modified from the original scripts I created on Ubuntu, so I am still in the process of making sure they are optimized for the unRaid environment.

 

 

BETA RELEASE ATTACHED BELOW

 

Please make sure to make a backup of your /boot/config/go and use at your own peril.  I welcome any comments

 

Version 0961

 

I added checks to tcrepack and tcrepackblu to avoid the infinite loop when a file fails to transcode properly.  A new variable is in tcOPTIONS to control how long it forces you to wait between runs of tcrepack and tcrepackblu (MINBTWN).

 

Version 096

 

Found the last issue (hopefully) with cron and /usr/local/bin.  This should solve the issue where it could not find HandBrakeCLI due to PATH issues when run from cron.

 

Version 095

 

Cleaned up some bad assumptions regarding the cache drive.  Now final files are written directly to the cache drive.  I especially need testers without a cache drive.

 

Version 094

 

Reached beta status now.  Accommodated the fact that unraid seems to leave /usr/local/bin off of the path from cron.

 

Version 093

 

Found an error in search strings, now corrected.

 

Version 092

 

Fixes a typo in tcreblu that led to the transcoded file being left on the cache drive instead of moving back.

tc-0961.zip

Wow, that is very impressive work.  Although I'm sure it helped having everything in Ubuntu first, nevertheless getting it all ported over to Slackware isn't trivial.  Perhaps it might help the less informed if you indicate at the beginning of your post what the purpose of transcoding is, and why automating it is so damn cool.  Again, very high kudos (and welcome to unRAID)!

Wow, that is very impressive work.  Although I'm sure it helped having everything in Ubuntu first, nevertheless getting it all ported over to Slackware isn't trivial.  Perhaps it might help the less informed if you indicate at the beginning of your post what the purpose of transcoding is, and why automating it is so damn cool.  Again, very high kudos (and welcome to unRAID)!

 

Thanks.  I took your advice and added more basic information.  When I built these tools, I knew they would eventually run on unRaid.  This lead me to make them all bash scripts.  The challenge in porting them was in taking care to use the correct reference (/mnt/disk3/somedir instead of /mnt/user/somedir) in order to limit the number of drives I spun up and avoid moving files across disks.  Changing a filesystem reference is obviously preferable than a copy-and-delete across disks.

Thanks.  I took your advice and added more basic information. 

Yeah, it reads very well now for the uninitiated.

 

The challenge in porting them was in taking care to use the correct reference (/mnt/disk3/somedir instead of /mnt/user/somedir) in order to limit the number of drives I spun up and avoid moving files across disks. 

That's still difficult if you have a lot of references to address.  When I'm writing code, I find the algorithms and heavy objects get written easily, but the simple things like infinite loops and non-initialized variables always bite me somehow. 

but the simple things like infinite loops and non-initialized variables always bite me somehow. 

Yeah, and those infinite loops still seem to run forever, even on the modern faster CPUs.  ;)

 

Ask me how I know...

Wow, that is very impressive work.(...)  Again, very high kudos (and welcome to unRAID)!

+1 for that! Really impressive and well described. I will definitely take this for a spin soon. A lot of the cartoons recorded could do with some shrinking!

 

Related question: Would Handbrake support cutting commercials as well, given that there is a cut file already available next to the original file in a suitable format (i.e. commercials have already been identified)?

I'd love to see these scripts. While I don't often transcode my existing media, I'm often adding new media and a big part of that process is manual.

 

nia, HandBrake has no commercial detection/removal capabilities.

nia, HandBrake has no commercial detection/removal capabilities.

I already have the commercial detection software that can report/save cutlists in a number of formats. I was simply hoping that Handbrake could use such cutlist to avoid the marked sections in the timeline when re-encoding... Apparently no such luck.

 

I still like this functionality even without the cutting 

GJones, any updates? Care to share some of your scripts?

 

I am documenting them now to make them a bit more user-friendly. 

I have attached an alpha release to the original post.  This version does not automatically add the cron job (so you can get accustomed to it) but has all the other running parts.  If there are questions, I will try to answer them in this thread.

Found the first bug, a typo in tcreblu, and corrected it.  Version 092 is up for download now.

Now that this is a true add-on, perhaps this thread should be moved to "User Customizations", and maybe it could even be hooked into unMENU as a package?

 

I'm very experienced in Linux but still very new (a few weeks) to unraid.  With a bit of help in packaging, I would be glad to provide it as an unmenu package.

 

I'm very experienced in Linux but still very new (a few weeks) to unraid.  With a bit of help in packaging, I would be glad to provide it as an unmenu package.

 

There are people willing to help with the unMenu package (at least testing it).  The best way to learn about and build the packages is to look through the ones that are already created.

 

The Airvideo and Transmission packages are two that I have created and are full of good stuff to learn from.  There are many others, ranging from simple to complex, that you can use as a starting point.  If you have any questions I will try to help as much as possible and Joe L would probably be willing to answer questions also (he is the one that created unMenu and the package manager).

 

Oh, and I am going to move this to the Customization Forum.

 

I'm very experienced in Linux but still very new (a few weeks) to unraid.  With a bit of help in packaging, I would be glad to provide it as an unmenu package.

 

If that could be done, it would simplly rock!

Thanks for moving it over.  I will take a look at some of the conf files and packages.  Fortunately, installing it only requires file copies.  The last hurdle is in working out the editable options using unmenu's configuration screen.

At this point, testers are needed most.  Here are the steps to test:

• Install Handbrake from the unmenu conf file
  
• Copy the contents of the zip file to /boot/tc
  
• Run:  ln -s /boot/tc/bin/* /usr/local/bin/
  
• Edit /usr/local/bin/tcOPTIONS to match your configuration (comments are throughout)
  
• Run one job by:  ts tcremkv /mnt/user/full/path/to/mkvfile.mkv if you have a cache drive
  
• OR Run one job by:  ts tcremkv /mnt/diskX/full/path/to/mkvfile.mkv if you do not have a cache drive
  

 

You can check on the progress with ts -t.  Just remember to CTRL-C when you want to quit watching.  The job will still be running, just not with you watching it.  The jobs will survive a logoff, as well.  If you just want to check on the status of the job (running or finished), type just ts.  When you are satisfied with the low-level utility, try the following:

• ts sleep 5m - Gives you five minutes to review before it starts
  
• tcrepack - finds and queues up jobs
  
• ts - shows you what it queued
  

 

If you see something you don't like, type:  ts -K, which will kill the job queue.  The HD utility, tcrepackblu, can be tested just like tcrepack.  When you are supremely confident, copy the cron job over:  cp /boot/tc/cron/* /etc/cron.daily/

Found an error in search strings, now corrected in version 093, attached to the first message in the thread.  This should be a fairly stable version at this point.  I am running it unaltered on my box now.

Awesome concept - I love having routine tasks automated so you can "fire and forget".

 

I just wanted to send you the link to this.  Someone on the Handbrake forum spent a huge amount of time in building a similar Mac-based product.  I love using it, and might give you some ideas as to features for the future and process steps.

 

I haven't installed it, so perhaps this base is already covered.  My only comment about this script would be to have some option to ignore data that has already been encoded, but doesn't follow the file format required by the script - I would rather not have to go and rename all of my already-encoded files (adding ".H264" to the filename) to ensure they don't get re-encoded!

 

Thank you for the great work!

I had considered using a .skip file or something similar. If you want a file skipped, just make a file with the video filename with .skip added to the end.

That would certainly be good, but we need a way to implement this across data that already exists... some people in this forum have 20+TB of data... that could be tens of thousands of files to go through to append ".skip" to each one.

 

Perhaps instead it might be better to have an option to only encode files with a ".process" in them, so that the tool can be configured to work the other way?

 

Sorry if I'm missing/ over-simplifying something here...

Or use the search directories to search only directories that have original files in them. It would be easier to make a new director to search (currently supported and more flexible).

At this point, testers are needed most.

I am about ready to do this, but since I have never used Handbrake, I have to ask an ignorant question.  All my files are BluRay ISO with lossless audio tracks, so do I need to convert them to MKV first to go through your procedure, or what?  And what CLI options do you guys use?  The Handbrake forums are all over the place with an infinite number of configurations.  I just want a high quality setting that reduces file size by 50 - 66%.

aiden, yes you'll need to convert ripped BluRays to MKV first. HandBrake can't parse BluRay ISOs yet (it's happy to do DVD ISOs though).

 

As for "best" settings, it's hard to define exactly. I'd try using the GUI version of HandBrake (Mac, Windows and some Linux distros) to experiment on a few short segments of your BluRays to see what settings you like. I haven't done any BluRays yet, so I don't have much info, but I'm happy to discuss DVDs via private message if you'd like.