Tips for Noobs - Duplicates, Versioning, and General Organization

[RussellinSacto]

Hi Everyone,

 

I've had my UnRaid a few years now and I thought I'd try to offer a few tips for noobs - as a contribution to the community and as a thanks for the help I've received from others on these forums.

 

Having an UnRaid with lots of TBs can create significant organization issues -  in particular with duplicate files.  As an avid photographer, I have thousands upon thousands of image files, but these tips apply to pretty much any collection.

 

Here are my suggestions (using a Windows box to connect to my UnRaid):

 

Step 1  - Create a "Dump folder" on Disk1 that is alphabetically before all your other folders (or utilize all of Disk1) and place any new files that you load onto your UnRaid into this folder.  (When I get back from a photo shoot, I simply copy all of my files to this folder.  Do the same thing with your movies, MP3s, etc.)

 

Step 2 - Scan your UnRaid for duplicate files utilizing DoubleKiller (or DoubleKiller Pro) which can be downloaded from:  http://www.bigbangenterprises.de/en/doublekillerpro/comp.htm  For many users, the FREE version will be more than adequate.

 

On the Options Tab, you can set the tool to check for duplicate files by Name, Size, Date, and CRC checksum (a math calculation that is file unique).  I suggest the CRC checksum for maximum reliability, but it's the slowest.  Also note that when you "Include Subdirectories" (to choose the regions you want scanned) instead of browsing you can always type the location directly into the "Folder" box - for example, enter: \\tower\disk1 and it will be added to the scan list.  I simply add all of my UnRaid drives in this fashion and then click "RUN" to find the duplicates throughout my UnRaid.

 

Step 3 - When the Scan is complete - I push the button for "Check the First Dupes" which will check the duplicate files in the earliest alphabetically found location (this is the reason to use a folder like "aaaaa" or all of Disk1 as your "Dump Folder").  In a moment, all of the duplicates will be checked, and simply clicking "Delete Checked Files" will get rid of them.  NOTE:  If you purposefully have duplicate files elsewhere in your UnRaid, you can page through the DoubleKiller list and uncheck files you don't want deleted (or if there's a system to this duplication you have, simply take those regions out of your scan in Step 2).

 

By using this alphabetical system, if you have duplicate files somewhere else on your UnRaid, they will be left in their location (probably an organized/sorted, specific location) - the duplicates will be removed from the earliest alphabetical locations first. 

 

Step 4 - DoubleKiller doesn't remove folders, even if they are empty because of its operation - so to clean things up, you can use "Remove Empty Directories" - a FREEWARE utility you can download from: http://www.jonasjohn.de/lab/red.htm  The software will delete all of your empty folders (allowing you to preview them before it does so).

 

This "system" works with just about any collection - movies, television shows, MP3s, photographs, etc.  The key benefit is that you can use this "Dump Folder" (as I call it) to dump all of your files to when you get them, download them, or otherwise offload them to UnRaid and you can quickly ensure you don't have duplicate files all over your UnRaid system.  I often load old CDs of files right into this folder then scan and dump anything I already have.  I can then review the remaining files and determine if I want them or not.

 

Other tips:

 

Photographers and Videographers who shoot a lot of material - I highly recommend having a hard drive based field backup.  My current device is the HyperDrive Colorspace UDMA  - I can backup 500GB in the field, without power.  Back in the car I can offload that 500GB to one of several USB drives (no laptop required) and repeat the process.  It's not unusual to come home with several TB of images (I often backup files more than once (to multiple USB Drives in the field - wanting to be "safe, not sorry")) - so removing duplicates is critical.

 

Photographers should also take a hard look at Photool's IMatch software - for organizing your images.  It's simply incredible.  I can't do it a worthy job in this forum, but I assure you that you won't be disappointed - it's amazing (and simply a bargain for $65):  http://www.photools.com/imatch-3-overview/

 

Programmers, Photographers, Authors, and others who create multiple versions of their work - if you don't have a versioning system - there's an excellent (very simple) versioning software that you can run on your Windows computer called AutoVer which is FREEWARE (donations accepted) downloadable from: http://beanland.net.au/AutoVer/  You simply tell it what folders you want monitored and where you want versions stored, and it does the work.  Very easy to use (works with FTP, network drives, etc.)

 

Hopefully this will come up in searches for people looking for a few tips - and hopefully this is helpful.  Thanks again to everyone on here who's helped me with UnRaid.

 

Russell Schutte

 

[RussellinSacto]

I was hoping others might contribute some great ideas for noobs/newbies - or those who use their UnRaid as an appliance (like I do).

 

I've got a couple more tools to suggest for Windows users:

 

TeraCopy - which allows you to move files around on your UnRaid or to your UnRaid (http://www.codesector.com/teracopy.php).  What I like most about it is the CRC check of file copying.  I suggest choosing "Options" from the "Menu" button in the bottom right corner of the software, and then checking the box for "Always test after copy."  I also always click the "More box" to see the details when I copy files - but TeraCopy has been great for copying files from unreliable sources (DVDs and CDs in particular).  When the CRC shows an issue, I simply give the disc a little extra cleaning and try it again, until I get a proper match.

 

(Another way to do this - which is probably faster for Unraid to Unraid moving/copying involves dipping your toes in the Linux pool.  You can download Putty.exe (from http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html), set it to connect to Host "Tower" and click the "TelNet bubble", then click "Open" and it should connect to your UnRaid system.  Login with your username and password (default user is "root" and no password).  At the next prompt, after your login, you can access a file manager called Midnight Commander by simply typing "mc" and then Enter key.  It's very much like the old DOS utilties, but fairly easy to figure out - and you get to use those cool "Function Keys" (like F6) at the top of your keyboard).  (NOTE:  You can open up to FOUR telnet/Putty connections, so you can have a few of these going simultaneously if you're doing a majore resorting of files - four seems to be a limit).

 

IndexYourFiles - from indexyourfiles.com - keeps a database of your files, unlike Windows Search which scours your hard drives when you excute a search.  In doing so, searches take portions of a second instead of 5+ minutes.  And you can schedule the software to reindex every once in a while if you want (so it stays accurate - mine is set to the dfault 12 hours and I manually reindex if I move a lot of files - which only takes a few minutes for 15TB).  If you're constantly looking for files (by name or contents) that you can't find, and you're really tired of waiting for Windows Search, this is a great tool.  Oh - and it won't spin up your drives when you run searches - that only happens if you decide to access the files you locate.

 

(another frequently suggested, similar solution, is Locate32 from http://locate32.net/content/view/18/31/ ) but I simply didn't like it as much as the IndexYourFiles solution.  (It may be fractions of a second faster, but doesn't have "Preview" panes for viewing contents of files).)

 

Anyone have any other great ideas/tools for new users / appliance users?  Any good ideas what should be your first (SAFE) steps into Linux if you're ready to make that step?

 

Thanks,

 

Russell Schutte

[krackato]

Anyone have any suggestion for a Duplicate File Finder on the Mac?