Program to verify multiple file integrity via md5


Recommended Posts

I needed to verify that my files were coping over to my unRAID server okay, and that everything was set up.  So, I used hkSFV in Windows to calculate the md5 checksum of the first batch of a few hundred GB or so of files, and transferred them and the hash file (with names and file locations and md5 hashes) over to the server.

(By the way, hksfv is a wonderful and easy to use Windows program with a good GUI for calculating file hashes - just make sure and read all the options before you run it)

 

At this point, I needed a commandline tool that would run in unRAID and process the hash file to make sure my files had been transferred over correctly.

 

Surprisingly, finding one was rather difficult...

 

Finally, I managed to find this wonderful program called md5deep, here:

http://md5deep.sourceforge.net/

 

Even better, it comes as a precompiled slackware package that will run in unRAID (running BubbaRAID and unMENU, if it makes any difference) here:

http://www.linuxpackages.net/pkg_details.php?id=9470

 

(There is a newer version available, but this is the one I used, so I know it works)

 

It's able to process a list of md5 hashes and file locations, and either tell you what files are present that are not in the list, or the files that matched teh list - quite useful.

 

Anyway, I'm sure most of the regulars on the forums know all about this, but just in case someone else needs something similar, I thought I'd share and hopefully save someone an hour or three of searching (feel free to message me if you need help setting it up)

 

Hope this helps someone,

Zithras

 

(Edit: I think someone else was looking for something similar earlier, but I don't think they posted if they actually found a solution - off to dinner now)

Link to comment

Thanks,

 

I posted about two weeks ago for something like this.  I ended up using the checksum option in Rsync. (not sure if it was md4 or md5).  I never got a respose to my thread, so I was unsure if using rsync was recommended. Everything turned out good though, and the vebose output didn't show anything was copied incorrectly.

Link to comment
  • 3 weeks later...

AWESOME guys. I was using a couple of different packages, but I think this one works the best, and a bonus having it precompiled in Slackware.

How often are you running it and what is your script for running it?

Can you save all the md5 files on a separate disk by themselves?

Link to comment
  • 4 weeks later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.