bunker - yet another utility for file integrity checks

Did you find any corruption in the extended attributes?

 

When I moved from RFS to XFS I lost the extended attributes, the way I verified the correctness of the files afterwards was by using the -C command - this recalculates the hash and compares it to the value stored in the export file, and restoring it in the extended attributes.

You did it different, but the results should have been the same.  Once you restored the extended attributes and compared to the copied file, you should have seen if there was any corruption in the RFS extended attributes.  They would have been copied  to your export file.

I just finished (bunker -v -n) on 2 more RFS disks.  This 6TB of data didn't have a single error.  RFS isn't always bad??? 

I may be being stupid here but I've read through the entire thread and the help command for bunker but couldn't find the answer. Can I simply just calculate the checksums of the file and export them to a text file WITHOUT writing them to the file extended attributes?

I may be being stupid here but I've read through the entire thread and the help command for bunker but couldn't find the answer. Can I simply just calculate the checksums of the file and export them to a text file WITHOUT writing them to the file extended attributes?

No that is not possible. The export function reads the values stored in the extended attributes and writes them to a file.

I may be being stupid here but I've read through the entire thread and the help command for bunker but couldn't find the answer. Can I simply just calculate the checksums of the file and export them to a text file WITHOUT writing them to the file extended attributes?

 

No that is not possible. The export function reads the values stored in the extended attributes and writes them to a file.

I see, thanks for the answer!

Ok, had a look at your changes, thanks for the update.

 

I have updated the OT with the new version v1.13.

Here are yet more improvements to the syslog output for you to check out.  I noticed the output was not being tagged as from 'bunker' if they were warning or error messages so have rectified that.  I also move some tests into the writer() and Logger() functions that simplified the rest of the code a bit.

I have also included copies of my current bunker-verify.sh and bunker-update.sh scripts that I currently run as cron jobs.  Not sure if these will be of interest to others?

bunker_v1.14.zip

Ok, had a look at your changes, thanks for the update.

 

I have updated the OT with the new version v1.13.

Here are yet more improvements to the syslog output for you to check out.  I noticed the output was not being tagged as from 'bunker' if they were warning or error messages so have rectified that.  I also move some tests into the writer() and Logger() functions that simplified the rest of the code a bit.

 

I have also included copies of my current bunker-verify.sh and bunker-update.sh scripts that I currently run as cron jobs.  Not sure if these will be of interest to others?

Thanks for your input, I'll have a look and update the OT accordingly.

Here are yet more improvements to the syslog output for you to check out.  I noticed the output was not being tagged as from 'bunker' if they were warning or error messages so have rectified that.  I also move some tests into the writer() and Logger() functions that simplified the rest of the code a bit.

 

I have also included copies of my current bunker-verify.sh and bunker-update.sh scripts that I currently run as cron jobs.  Not sure if these will be of interest to others?

New version 1.14 is made available in the OT together with your additional scripts (these are all located in the zip file).

Thanks for your contributions, much appreciated.

Here are yet more improvements to the syslog output for you to check out.  I noticed the output was not being tagged as from 'bunker' if they were warning or error messages so have rectified that.  I also move some tests into the writer() and Logger() functions that simplified the rest of the code a bit.

 

I have also included copies of my current bunker-verify.sh and bunker-update.sh scripts that I currently run as cron jobs.  Not sure if these will be of interest to others?

 

New version 1.14 is made available in the OT together with your additional scripts (these are all located in the zip file).

 

Thanks for your contributions, much appreciated.

I would have preferred to PM something like this, but it appears that the forum does not allow attachments with PM's.

Here is a RFS drive that is the destination of rsnapshot.  12 times per day, the contents of my documents drive are snapshotted into this drive.  This allows me to pull previous versions of a file should I have made an oopsy and need to recover a prior version.  See the number of files below, and the ETA times.  Is bunker going to actually do a full calculation on all 2.6 million files? 

rsnapshot link http://lime-technology.com/forum/index.php?topic=26888.0

Here is a RFS drive that is the destination of rsnapshot.  12 times per day, the contents of my documents drive are snapshotted into this drive.  This allows me to pull previous versions of a file should I have made an oopsy and need to recover a prior version.  See the number of files below

http://i.imgur.com/Q5jGrlU.png

That is a very large number of files, perhaps you are hitting a limit for one of the commands used in bunker or you run out of space for the temp files ?

The latter one you can check by telnetting into your system and do:

df -h /tmp

The latter one you can check by telnetting into your system and do:

df -h /tmp

root@Server1:/boot/config# df -h /tmp

Filesystem            Size  Used Avail Use% Mounted on

-                        0    0    0  -  /

root@Server1:/boot/config#

Its not stopped.  It has 4 other bunker instances running, and this one is proceeding too. 

It appears that the RFS errors are a plague though.  When we have the RFS errors, is it possible that the extended attributes just don't get written??  Later, a "bunker -a /mnt/disk6" seems to find  a few more files to add on what I thought was a static drive.

The latter one you can check by telnetting into your system and do:

df -h /tmp

 

root@Server1:/boot/config# df -h /tmp

Filesystem            Size  Used Avail Use% Mounted on

-                        0    0    0  -  /

root@Server1:/boot/config#

 

Its not stopped.  It has 4 other bunker instances running, and this one is proceeding too. 

 

It appears that the RFS errors are a plague though.  When we have the RFS errors, is it possible that the extended attributes just don't get written??  Later, a "bunker -a /mnt/disk6" seems to find  a few more files to add on what I thought was a static drive.

I moved away from RFS...

That is a very large number of files, perhaps you are hitting a limit for one of the commands used in bunker or you run out of space for the temp files ?

I actually should have a lot more files than that actually.  95,000 files rsnapshotted 10 times per day for 2 years.  We should have billions....

But I am only keeping 7 hourly, 7 daily, 4 weekly, and 12 monthly generations.  I guess the math is close...

I moved away from RFS...

Me too, but its a process.  Legacy systems don't die easy.  I have way too many RFS disks to convert.  About 30 - 3tb drives to go.

I moved away from RFS...

 

Me too, but its a process.  Legacy systems don't die easy.  I have way too many RFS disks to convert.  About 30 - 3tb drives to go.

Yeah, it's a lengthy process, weeks or months is not unlikely.

Found a potential issue with bunker in that if it is running it can stop the array from being stopped.  I assume that this is probably fixable by adding a periodic check somewhere to see if the array is being stopped and if so aborting bunker from checking any more files.

If anyone knows what can be checked to see if unRAID is trying to stop the array I can probably work out the patch to bunker to make use of the information.

Found something wrong with v1.14 of  bunker

I was trying to update the files that had be modified after the  bunker -a had already hashed them.  No matter what options I used, it seemed to ignore them.  It ran for 2 hours on a 3tb drive looking for files that had been modified in the last 2 hours.  (there would have been no files in that status)

unRaid v5.05 on Xeon processor Supermicro XCM9 with drives attached to IBM m1015

root@Server1:/boot/config# bunker -u -D 2d /mnt/disk2
./bunker: line 212: -1: substring expression < 0
date: invalid date `@'
Program terminated.
root@Server1:/boot/config# bunker -u -D 2w /mnt/disk2
./bunker: line 212: -1: substring expression < 0
date: invalid date `@'
Program terminated.
root@Server1:/boot/config# bunker -u -D 2h /mnt/disk2
./bunker: line 212: -1: substring expression < 0
date: invalid date `@'

bunker-update.sh goes crazy with these errors.  Something must be different with my system

/boot/bunker/bunker: line 212: -1: substring expression < 0
date: invalid date `@'
Program terminated.
/boot/bunker/bunker: line 212: -1: substring expression < 0
date: invalid date `@'
Program terminated.
Program terminated.
/boot/bunker/bunker: line 212: -1: substring expression < 0
date: invalid date `@'
Program terminated.
/boot/bunker/bunker: line 212: -1: substring expression < 0
date: invalid date `@'
Program terminated.
Program terminated.
/boot/bunker/bunker: line 212: -1: substring expression < 0
date: invalid date `@'
Program terminated.
/boot/bunker/bunker: line 212: -1: substring expression < 0
date: invalid date `@'
Program terminated.
Program terminated.
root@Server1:/boot/bunker# ^C
root@Server1:/boot/bunker#

Appears to be a ver5 issue.  UnRaid 6.01 is working much better with it.

Appears to be a ver5 issue.  UnRaid 6.01 is working much better with it.

My guess, it is a bash thing, unRAID 6 has a newer version.

Assuming the above scripts work out OK, then I was thinking of trying my hand at a plugin to put a GUI front-end on the installation, configuring and scheduling execution of bunker. If I get that far (which is not at all guaranteed ) do you have any objections about me including the bunker binary as a download in such a plugin or should users just be prompted to install it manually? 

Any news on the front-end plugin?

Assuming the above scripts work out OK, then I was thinking of trying my hand at a plugin to put a GUI front-end on the installation, configuring and scheduling execution of bunker. If I get that far (which is not at all guaranteed ) do you have any objections about me including the bunker binary as a download in such a plugin or should users just be prompted to install it manually? 

 

Any news on the front-end plugin?

I got distracted and so far have done nothing towards getting this done. 

I need to get bunker set up in cron, but before I do, shouldn't a

bunker -U /mnt/disk1

catch a file that I have just edited and newly saved under the same name with say MS Word?  Bunker, goes ripping through the 4tb disk in about 18 minutes, but doesn't find any files to update.

(My understanding is that "bunker -U" compares the file date with the hash date and if they are different, it creates a newly updated hash. )

Shouldn't a

bunker -A
bunker -U

be all you need to do each day to make sure all your hashes are up to date?

Shouldn't a

 

bunker -A
bunker -U

 

be all you need to do each day to make sure all your hashes are up to date?

Correct this will first scan for new files and update those which have been modified (file date is newer than scan date).

The above commands will be performed on the entire disk, to improve performance you can restrict the commands to only look for files added/changed in the last day (assuming you are going to run this on daily bases). Use:

bunker -A -D 1
bunker -U -D 1

Have also a look at the supplementary scripts in the zip file for automation purposes.

New version 1.15 of bunker is available, see OT for details and download.