SilverRubicon Posted September 10, 2014 Share Posted September 10, 2014 If anyone has audio files that they would like to check, Foobar2000 has a File Integrity Validator. Seems to work well. http://www.foobar2000.org/components/view/foo_verifier Link to comment
neilt0 Posted September 11, 2014 Share Posted September 11, 2014 If anyone has audio files that they would like to check, Foobar2000 has a File Integrity Validator. Seems to work well. http://www.foobar2000.org/components/view/foo_verifier Thanks for the tip. I ran it on the archive of original audio recorded for my podcast (905 FLAC files, 97.7GB) and it only found one corrupt file. However, that file was already known to be corrupt as it was a failed upload from one of my co-hosts. That's also a good thing, as the verifier correctly identified a known truncated file, suggesting none of those files have been truncated by the bug. This suggest that none of my important source files have been corrupted. However, whether or not they have been jumbled up and have the wrong filename, as reported before is yet to be determined. At some point, I'll pull down all my backup files from OneDrive (some recent files have not been backed up, but most have) and do a byte compare. The files were on a volume (drive) that had about 1 million writes to it since booting with Beta 6, so that's encouraging, I think. That drive does have a lot of other files I care about, so it'll take a while to check the rest. Link to comment
neilt0 Posted September 11, 2014 Share Posted September 11, 2014 On the topic of error checking, given that a lot of the files we have on our servers are audio and video (audio was covered earlier with the foobar plugin), would this be an easy way to check video file integrity: http://superuser.com/questions/100288/how-can-i-check-the-integrity-of-a-video-file-avi-mpeg-mp4 You can use a feature in ffmpeg video converter: if you will specify it to recode video to nothing it will just read input file and report any errors that will appear. This is very fast process because video frames are just being read, checked and silently dropped. Example command line: (for Linux) ffmpeg -v error -i file.avi -f null - 2>error.log -v error means a certain level of verbosity (to show some errors that are normally hidden because they don't affect playability a much). You will get a full error log with some generic information about file ffmpeg will output, so this will probably require your attention, through filters can be written to perform batch check of similar files. We'd just need a script to traverse through all directories, check every file, then parse any error messages out, right? Link to comment
neilt0 Posted September 11, 2014 Share Posted September 11, 2014 Perhaps a mod can split out the off-topic discussions in the other thread. On with more data integrity checking -- I mentioned audio and video in other posts. Some of my data is RAR'd. Here's a way to check all your RAR files for errors in one fell swoop: http://superuser.com/questions/359245/how-do-i-check-many-rar-files-for-corruption-at-once ETA: That doesn't appear to work with RAR5 archives. Checking for another solution now... Link to comment
neilt0 Posted September 11, 2014 Share Posted September 11, 2014 From the command line, this works to test the integrity of any RAR file: unrar t filename.rar Again, a script could traverse directories to check all RAR files on a drive (or more than one even). Link to comment
neilt0 Posted September 11, 2014 Share Posted September 11, 2014 I found a Windows app that will check a batch of RAR files for integrity: http://www.extractnow.com/usage.php#rclick Slower than doing it natively on the server, but it means I don't have to learn how to write code! Link to comment
neilt0 Posted September 11, 2014 Share Posted September 11, 2014 OK, so all I've been testing so far is my own data, a small subset of the data on my server, but obviously it's the stuff I care about most. I tested about 100GB of FLACs, plus some MP3s and WAVs. All are uncorrupted. There's no quick way of checking if filenames are corrupted/transposed, though. I've also tested 82 RARs (28GB). No corruption: If someone wants to put a little shell script that can run ffmepg recursively on /mnt/user/Movies, I'm up for trying that on the 24TB of Movies on my server! http://lime-technology.com/forum/index.php?topic=35161.msg327534#msg327534 Cheers, Neil. Link to comment
ogi Posted September 11, 2014 Share Posted September 11, 2014 I'm not on the beta but one thing all these checks have me worried is can the act of checking a file cause corruption, and would it be better to either hold off on checking for corruption until Beta 9 is released or downgrade to Beta 6 when checking? Link to comment
neilt0 Posted September 11, 2014 Share Posted September 11, 2014 I'm not on the beta but one thing all these checks have me worried is can the act of checking a file cause corruption, and would it be better to either hold off on checking for corruption until Beta 9 is released or downgrade to Beta 6 when checking? No, because the checking is read only. I'm only checking shares that are made read-only (e.g. /mnt/user/Movies, not /mnt/disk6/Movies). Link to comment
WeeboTech Posted September 11, 2014 Share Posted September 11, 2014 I'm not on the beta but one thing all these checks have me worried is can the act of checking a file cause corruption, and would it be better to either hold off on checking for corruption until Beta 9 is released or downgrade to Beta 6 when checking? It depends on the root cause. In this case, if you mount the drive in read only mode, you cannot cause further corruption. Since the drives are mounted with noatime, generally, there are no writes when reading/accessing the directories and files so we are pretty safe to do reads. To be 100% sure, remount in readonly mode makes it safer. Link to comment
WeeboTech Posted September 11, 2014 Share Posted September 11, 2014 I'll quote Tom here. Tom or Jon, is mount -o remount,ro /dev/md<number> a safe way o avoiding this bug? Yeah that should work. Link to comment
NAS Posted September 11, 2014 Share Posted September 11, 2014 i suggest bash + the file command + a bunch of file type specific tools could be used here. 7z alone could cover zip + rar + all the comic formats etc Link to comment
ogi Posted September 11, 2014 Share Posted September 11, 2014 I'm not on the beta but one thing all these checks have me worried is can the act of checking a file cause corruption, and would it be better to either hold off on checking for corruption until Beta 9 is released or downgrade to Beta 6 when checking? No, because the checking is read only. I'm only checking shares that are made read-only (e.g. /mnt/user/Movies, not /mnt/disk6/Movies). That would be the way to do it. Figured in case some of the less technical people (like me) wanted to check for corruption, the worst thing that could happen is introduce new corruption while checking for it. Thanks for the reply. Link to comment
neilt0 Posted September 11, 2014 Share Posted September 11, 2014 i suggest bash + the file command + a bunch of file type specific tools could be used here. 7z alone could cover zip + rar + all the comic formats etc Does linux 7z support RAR5? The release version for Windows doesn't. I didn't try the beta, though. Link to comment
NAS Posted September 11, 2014 Share Posted September 11, 2014 i suggest bash + the file command + a bunch of file type specific tools could be used here. 7z alone could cover zip + rar + all the comic formats etc Does linux 7z support RAR5? The release version for Windows doesn't. I didn't try the beta, though. TBH i dont know. I never liked rars due to their license and only know them indirectly. What I am suggesting isnt ideal but its a file level tool that could help if no FS level tool is available. And as i mentioned previously its not a bad tool regradless I for one would like to know what mp3s are corrupt etc regardless of how they got that way. Link to comment
jonp Posted October 2, 2014 Share Posted October 2, 2014 Guys, this isn't really a "defect report" so I'm going to move this to general support. Thanks! Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.