The Parity check Nocorrect option

[RobJ]

With apologies, this will be lengthy.  This is in response to comments made in the "SimpleFeatures" Plugin thread, beginning here, with continuing discussion the rest of that day (May 7).  The quote that started the discussion:

Second Item:  On 'Main', the box for "Write Corrections to parity disk" is checked.  I would prefer that (1) it either be unchecked  or (2) have an option on the settings page to determine the status of the check in the box.  Perhaps, a good place for this second option would be on the 'Settings' page under the 'Simple Features' in the "Confirmation' area.

 

On the 'Settings' page again on the 'Simple Features' tab, there is the 'Scheduler' for the automatic parity check.  The default choice for 'Write corrections to parity disk' is 'Yes'.  I would prefer that it be 'No'.

 

The reason for wanting these box(s) unchecked is that should NOT be any parity errors found during a parity check.  If there is an error, the user should be given an opportunity to check for the cause and correct it.  Correcting parity before the cause has been determined will cause data loss if the error is in the data array (most likely one of the data disks) and not in the parity array (most likely the parity disk).  Until the user has made that determination, an automatic parity check will simply hide a data array error from the user and give a false sense of security that all is well.

I strongly recommend reading the subsequent discussion before continuing here.

 

<...waiting for interested readers to return...>

 

There have been related discussions before, of which I was a part of at least one, found here: Revamping the Parity Check.  I got sidetracked and never returned to that discussion, but it starts with great ideas, and then includes some of my long-winded thoughts.  I remember part of what sidetracked me was I spent about 2 full days tracking down info concerning one affected users motherboard (I think), and that it was known for bazaar behavior, and therefore its data integrity capabilities were highly suspect.  I also remember that Joe and Bubba came up with 2 brilliant ideas, that could be valid sources of undetected changed data.  Wish I had followed up ...

 

Frank, you are obviously an experienced data professional, and what you state is based on solid principles, with which I fully agree.  However, I believe there are one or two things you may not have thought of, and I don't think you have had time to think through the related scenarios.  I can't claim to have a comprehensive knowledge of all possible scenarios either, but I do have some experience, and have tried to analyze all the possibilities I could conceive of.  I do respect your ideas, and those of a number of others who feel similarly to you, but here is why I remain unconvinced of the usefulness of the Nocorrect option.

 

Number one and foremost, if you detect a parity error and do not correct it, then your data is in danger until you do correct it!  It does not matter whatsoever what the cause of the parity error is, until you correct parity, any drive that has to be rebuilt will be rebuilt wrong at that block.  The only possible exception is if the bad parity reverses the bad data associated with the parity error, making it correct, but that is hardly an argument *against* correcting the parity info.  Correcting parity does not damage data, does not even touch it.  It only corrects the parity info, by making it reflect the state of the current reality, whether that reality includes corrupted data or not.  If by some means you were actually able to determine what disk contained the faulty block (a huge task!) and then determined what file was involved (if any, and that in itself is a huge task I'll try to touch on later), you would then correct the data, which in turn would change the parity to reflect the new and correct reality.  If you did NOT correct the parity first (by using the Nocorrect option), and then corrected the data, the corresponding parity update process will assume that parity is correct, perform the parity calculations based on your data changes, and write this parity info to the parity disk, but this parity will STILL BE WRONG!  It will NOT spin up all the other disks to perform a complete parity calculation.  Parity will still be wrong until the next correcting parity check, and your data will still be in danger, including the data you just corrected!  I really cannot think of a good reason why parity errors should not ALWAYS be fixed, ASAP.  Parity should ALWAYS be perfect, and as soon as an imperfection is detected, then it should immediately be corrected.  I know that others I respect disagree with me, so I'm willing to admit that I may not fully comprehend their arguments.  I'm happy to be corrected.  For now, if pressed, I would vote for the complete removal of the Nocorrect option.

 

One thing we can certainly agree on is that the parity check process should inform us of the precise blocks with parity errors, whether or not they are corrected.  This provides a starting point for file data corruption detection, but this is a huge subject, and a tough one.  You really need an existing par2-like system in place, that can inform you of any specific files that are corrupted.

 

Now we get to another point where I disagree with others.  For various other reasons, some found in the Revamping the Parity Check thread I mentioned above, I personally believe that almost all parity errors result from errors on the parity disk, NOT on the data disks.  I have a very hard time accepting the possibility of corrupted data on a data disk.  Yes, data in transfer may be corrupted before being saved to disk, perhaps by bad memory, program bug, or network or bus transmission error, but those are outside the scope of what we are discussing here, the parity protection of data AS WRITTEN to disk.  I won't repeat what I've stated in that thread, but the gist is that once data is written to disk, then it CANNOT be read back in corrupted form.  It is either all read back correctly, or a disk error is returned.  (Joe and Bubba did come up with very unusual circumstances that would qualify as exceptions, but they could not at all be considered even close to 'normal'.  It is a related but different discussion.)

 

As to why I believe the parity drive contains most if not all of the parity corruption?  Parity-protected data writing is a 2-stage process, first you write the data to its data disk, then you write the updated parity info to the parity disk.  It may be too simplistic, but the only cases relevant to this discussion are those where data is successfully written to a data disk.  If the data is not successfully written, then that's a real problem, but it's not one that is related to parity issues.  If the data is written successfully and the parity info is written successfully, then that is a successful case, but not one that's interesting to us here, because there is no error involved.  So almost by definition, problem cases relevant to our discussion are those where data is written successfully, but for whatever reason the parity info is not updated correctly, which results in a parity error.  But there is only one conclusion we can make from that, that the data disks are ALL correct, and that the parity disk is incorrect.  So we should correct it.  I suspect my use of 'simplistic' and 'relevant cases' above may cause some discomfort, and my argument may sound like a logic trick.  If I'm wrong, I'm open to correction.

 

I understand why Tom acquiesced with requests to add the Nocorrect option, because it has always sounded like a useful option, and at first even sounds like it may be a safer option for our data, and we all want that of course.  But I don't think it stands up under a more complete analysis.

[bonienl]

This is an interesting (and long) story

 

So here is my point of view (a short version)!

 

The parity disk is there to be able to replace a faulty data disk. In that respect the parity disk must have always the correct parity set, or in other words if a parity mismatch is detected it should be corrected on the parity disk immediately otherwise the concept of a data disk replacement will fail.

 

The parity disk is not there to correct individual corruptions on the data disks. If you suspect a data disk to be faulty, better replace it, the same would apply if the parity disk is suspected to be faulty. One should never end up in the situation where one (or more) disks are unreliable and raise questions about the parity check outcome.

 

[chickensoup]

This is an interesting (and long) story

 

So here is my point of view (a short version)!

 

The parity disk is there to be able to replace a faulty data disk. In that respect the parity disk must have always the correct parity set, or in other words if a parity mismatch is detected it should be corrected on the parity disk immediately otherwise the concept of a data disk replacement will fail.

 

The parity disk is not there to correct individual corruptions on the data disks. If you suspect a data disk to be faulty, better replace it, the same would apply if the parity disk is suspected to be faulty. One should never end up in the situation where one (or more) disks are unreliable and raise questions about the parity check outcome.

 

I agree for the most part. Personally I would always want my parity drive to be correct and therefore have no real reason/want for a nocorrect option. Also, statistically speaking I also agree that the likelyhood that the parity is invalid outweighs the likelyhood of actual data corruption. I would also remind users that parity is NOT a "backup" system however it is one form of data protection. Any vital data (things you simply cannot afford to lose/become corrupt) should be backed up twice, three, four times, offsite in an underground environmentally controlled bunker (you get the point).

 

Some (advanced) users may feel differently though and prefer to dig a little deeper so perhaps one option, just to try and keep everyone happy might be to: include the nocorrect option, however when the box is ticked a messagebox/warning is displayed advising against the use of nocorrect for "etc" reasons -> Are you sure you wish to proceed?

[lionelhutz]

I do agree that the default should be to correct. Otherwise, it can get a user who doesn't understand or follow-up on the errors into trouble.

 

However, I simple don't agree with your arguement that a drive will always return the correct data or an error when it is read. I've helped a few people here who would have lost data using a correcting parity check.

 

[S80_UK]

I do agree that the default should be to correct. Otherwise, it can get a user who doesn't understand or follow-up on the errors into trouble.

Agreed.

 

BUT , having said that, I am also unclear as to the rationale behind the statement...

 

Also, statistically speaking I also agree that the likelyhood that the parity is invalid outweighs the likelyhood of actual data corruption.

It seems to me (I am more of a hardware guy than a software guy) that any drive may become corrupt for a variety of reasons at any time, whether through hardware or software failure.  A fair number of errors reported on the forums are caused by cabling problems and they may affect any drive.  My view is that it is never safe to assume that the data read from one drive will be more likely to be correct than the parity that is read from another.  Without knowing a great deal more about the parity scheme employed I would not wish to speculate further.

[Frank1940]

I am going to take the contrarian view with this question:

 

WHY IS THE PARITY WRONG?

 

From a theoretical standpoint, the data on the parity disk is always correct.  The algorithm used to calculate parity is perfect.  It is written on the parity drive with error detecting and correcting data encoding. If the parity drive reads at all, it returns the proper parity code.  If a parity check fails, the failure must lie outside of the parity drive and its data.  Without this premise, you might as well throw out the entire concept of using any parity scheme.

 

Let's have a closer and more practical look at this statement.  I am going to make an assumption about Tom's parity algorithm: It  is correct and always generates the proper parity from its input.  If it were not, everyone would be seeing parity checking failures. In my opinion, the parity software is the least likely source of any parity error!

 

All hard disks use error detecting and correcting encoding of data.  (In fact, it is safe to say that every read of ANY hard disk will have an errors in the read that can and will be corrected!)  Once this is done, the corrected data is stored in the cache memory on the hard disk.  I believe this is the last time, this data is provided with error detection.  (I did a quick check on the SATA spec and didn't see any provision for error detection.) 

 

So we have the disk cache memory, SATA cable, SATA controller, the RAM on the computer, CPU caches, CPU all essentially without error detection.  There is always the possibility of electrical noise, powerline transients that brute force their way through the power supply, a burst of cosmic radiation that 'flips' a bit of RAM, an EMF event, fan or hard disk vibrations that cause intermittent connector issues.  I  have probably missed more possible reasons for bad parity than I have listed.

 

My point of view is that a parity check failure is an event which should not be simply ignored by writing what is perceived to be the 'correct' parity data!  The new calculated value is as likely to be wrong as what was read from the parity disk!  (And I will concede that the converse is also true!)

 

The true problem is that finding the cause of the error is not easy.  In fact, it is extremely difficult!!!  It is time consuming and is, often as not, a matter or trial and error replacement of parts until the culprit is found. 

 

I would suspect that a 'slow' (or 'undetectable')  failure of one of the hard disks is actually rather remote.  (Although, I suspect that the cache memory on the hard disk is actually the culprit in most of these cases!)  I suspect most failures of the platter portion of the hard disk is a complete failure to initiate a read sequence or a hard read error.  Both of these events are relativity easy to detect.  (I would hope that Tom's parity checking algorithm would detect such a condition and terminate immediately with a error message identifying the problem!)  Even if such a disk did exist, from a statistical standpoint, the bad disk is more likely to be in the data array then to be the parity drive.

 

Following the line of reasoning that I have outlined above, I would always do a parity check without correction.  If an error(s) occurred, I would do a second check without correction.  If this check should return without an error, I would bury my head in the sand and wait a few days before checking again.  However, if the second check returned errors, I would try not to panic and begin to analyze the problem.  Writing out a new parity database would be my last resort as I would know that there is a problem somewhere and I would not want that problem incorporated into the parity database!  I now know that I have a problem and I don't want to compound the issue by correcting the parity database with data that I know I can't trust at this point!

 

My reason for not doing this is as follows:  Let's assume that we have some non-recurring (or intermittent) problem that causes completely random parity errors and have 'corrected' the parity which will incorporate that error(s) in the parity database . If I should have a complete failure of a drive in the array (which is totally independent of the first problem)  and rebuild the replacement hard disk using that parity database, the data reconstructed on the replacement hard drive would now be wrong! 

 

In conclusion, my point is that any parity check error must be completely investigated and the root cause of the problem should be found because the source of the problem lies outside of the parity database!  Correcting the parity database is simply throwing a coat of paint on an already rusting and peeling structure.  At a minimum, a second non-correcting parity check should be run and the results of both checked.  It is my contention, that a correcting parity check should only be run after the cause has been found or all other avenues exhausted.  In other words, it should be used as the last resort, not the first one.

 

[chickensoup]

I am going to take the contrarian view with this question:

 

WHY IS THE PARITY WRONG?

 

From a theoretical standpoint, the data on the parity disk is always correct.  The algorithm used to calculate parity is perfect.  It is written on the parity drive with error detecting and correcting data encoding. If the parity drive reads at all, it returns the proper parity code.  If a parity check fails, the failure must lie outside of the parity drive and its data.  Without this premise, you might as well throw out the entire concept of using any parity scheme.

 

Let's have a closer and more practical look at this statement.  I am going to make an assumption about Tom's parity algorithm: It  is correct and always generates the proper parity from its input.  If it were not, everyone would be seeing parity checking failures. In my opinion, the parity software is the least likely source of any parity error!

 

All hard disks use error detecting and correcting encoding of data.  (In fact, it is safe to say that every read of ANY hard disk will have an errors in the read that can and will be corrected!)  Once this is done, the corrected data is stored in the cache memory on the hard disk.  I believe this is the last time, this data is provided with error detection.  (I did a quick check on the SATA spec and didn't see any provision for error detection.) 

 

So we have the disk cache memory, SATA cable, SATA controller, the RAM on the computer, CPU caches, CPU all essentially without error detection.  There is always the possibility of electrical noise, powerline transients that brute force their way through the power supply, a burst of cosmic radiation that 'flips' a bit of RAM, an EMF event, fan or hard disk vibrations that cause intermittent connector issues.  I  have probably missed more possible reasons for bad parity than I have listed.

 

My point of view is that a parity check failure is an event which should not be simply ignored by writing what is perceived to be the 'correct' parity data!  The new calculated value is as likely to be wrong as what was read from the parity disk!  (And I will concede that the converse is also true!)

 

The true problem is that finding the cause of the error is not easy.  In fact, it is extremely difficult!!!  It is time consuming and is, often as not, a matter or trial and error replacement of parts until the culprit is found. 

 

I would suspect that a 'slow' (or 'undetectable')  failure of one of the hard disks is actually rather remote.  (Although, I suspect that the cache memory on the hard disk is actually the culprit in most of these cases!)  I suspect most failures of the platter portion of the hard disk is a complete failure to initiate a read sequence or a hard read error.  Both of these events are relativity easy to detect.  (I would hope that Tom's parity checking algorithm would detect such a condition and terminate immediately with a error message identifying the problem!)  Even if such a disk did exist, from a statistical standpoint, the bad disk is more likely to be in the data array then to be the parity drive.

 

Following the line of reasoning that I have outlined above, I would always do a parity check without correction.  If an error(s) occurred, I would do a second check without correction.  If this check should return without an error, I would bury my head in the sand and wait a few days before checking again.  However, if the second check returned errors, I would try not to panic and begin to analyze the problem.  Writing out a new parity database would be my last resort as I would know that there is a problem somewhere and I would not want that problem incorporated into the parity database!  I now know that I have a problem and I don't want to compound the issue by correcting the parity database with data that I know I can't trust at this point!

 

My reason for not doing this is as follows:  Let's assume that we have some non-recurring (or intermittent) problem that causes completely random parity errors and have 'corrected' the parity which will incorporate that error(s) in the parity database . If I should have a complete failure of a drive in the array (which is totally independent of the first problem)  and rebuild the replacement hard disk using that parity database, the data reconstructed on the replacement hard drive would now be wrong! 

 

In conclusion, my point is that any parity check error must be completely investigated and the root cause of the problem should be found because the source of the problem lies outside of the parity database!  Correcting the parity database is simply throwing a coat of paint on an already rusting and peeling structure.  At a minimum, a second non-correcting parity check should be run and the results of both checked.  It is my contention, that a correcting parity check should only be run after the cause has been found or all other avenues exhausted.  In other words, it should be used as the last resort, not the first one.

 

That is a really solid argument. Although you are assuming that the cause of bad parity would mostly be from hardware failure?

[Joe L.]

It is my understanding that the "writes" to both the parity disk and the data disks are performed in parallel.  unRAID does not write the data disks first, and then the parity disk.

 

The "writes" are always through the underlying linux buffer cache.  Therefore, depending on the timing of the flushs of the buffer cache to the physical disk, you may have actually update the physical parity disk before the physical data disk.

 

Once written to the physical disks again the data is in its "cache" waiting for the platter to spin for the correct sector.  Again, it is impossible for us to know what any given "write" will finally be physically on the disk platter, for either the data or parity disk.

 

All it takes for a parity error to occur is a unexpected power loss with data still in one of the disk caches or in the on-disk RAM cache....  At that point, it is impossible to know which disk is correct, if either.  The journaling file-system used on the data disks seem to protect against the data disks being corrupted... as the journal will be re-played once power is restored.  I think parity is kept in sync with this... but again who knows.  it may not replay partial transactions, but those might have been written to the disks before power was lost.

 

I have read that there is a CRC checksum when transferring data across SATA cables.  I do not suspect the SATA cables ever cause parity errors.  They can cause read errors, but not parity errors with successful reads  (statistically, odds of a failure matching the CRC checksum is just too high).

 

The internal cache RAM on the disk is a very likely source of parity errors.  All it would take is for one bit to be read incorrectly (intermittently).  We've seen hard disks with this class of error... occasional inconsistent returned data, but no error otherwise.  It might be very difficult to detect this class of error, even when looking for it, as it could easily be triggered by a marginal or noisy power supply voltage to the drive electronics, or, vibration, or heat, or the phase of the moon. :-)

 

It is my understanding that read errorsare supposed to invoke a process where the sector that could not be read is replaced with the simulated/re-constructed content from the other working drive.  This of course assumes a read error, not a read with a bit flipped somewhere in it.

 

What can be improved is the error handling when a parity error is detected.  If unRAID were to re-try, perhaps several times, perhaps even re-setting the disk/disk controller in between, and then if correct parity is subsequently returned, dump something to the syslog that helps to helps to isolate what is happening.

 

Obviously, diagonal parity is part of a solution.  That would assist in determination of the correct contents of the data block.

 

Joe L.

 

[Frank1940]

 

 

That is a really solid argument. Although you are assuming that the cause of bad parity would mostly be from hardware failure?

 

Most likely if you consistently get parity check failures. Having it happen once might be a failure induced into the computer by an isolated random outside event like a nearby lightning hit or a burst of cosmic energy.  Having it happen consistently is an indication of bigger problem.  Finding it is the problem because it is an intermittent event.  (If the hardware would completely fail, it would probably have made its presence known long before a parity check uncovered it.) 

 

I have converted my FreeNAS server to unRAID in the past week.  I went from a 3TB FreeNAS server to a 7TB unRAID server.  After getting the server working, I copied all of the data back to the new unRAID server.. approximately 2.1TB of data.  I then run a non-correcting parity check.  No errors were found.  I plan on running another non-correcting check again this weekend.  The reason for doing this is stress-test the hardware.  The parity check will exercise virtually all of the hardware and hopefully detect any possible problems. 

[Frank1940]

Response to Joe L. comments:

 

I agree with most of what you say.  A power interruption during a write operation will cause havoc with the data on both the data drive and the parity drive.  That is why all servers should have a UPS that will shut them down before the backup battery runs down!  (I can remember back in the olden days (early 1980's) when a quick power hit would nearly always cause a disk screw-up!  The power level would drop to the point where the computer became unstable but not low enough to reset everything.  As the power quickly came back up, the disk would be written to with whatever garbage was then in its buffer!)  If any server loses power during a write operation, you will have a screwed up storage system and probably have to rebuilt the disks in question.

 

You could be right about the SATA data transmission.  I read quickly through how the data is transferred.  It is complex as the scheme used attempts to limit the number of changes from '0' to '1' to reduce the cable bandwidth requirements. 

 

I think you are correct about unRAID attempting to replace unreadable data by using the parity data to reconstruct it.  However, there is going to be a rather lengthy initial delay as all of the drives are going to have to spin up first!  It is my observation, the parity drive is never spun up during normal read operations.  (Someone please correct me if I am wrong!) 

 

As I see it, the parity data in unRAID is basically intended to be only used when a disk is replaced and the data needs to be restored to that disk.  I doubt if you really want to run very long in a emulated disk mode!  The parity check feature is only there to provide some level of confidence that can be done. 

[RobJ]

However, I simple don't agree with your arguement that a drive will always return the correct data or an error when it is read. I've helped a few people here who would have lost data using a correcting parity check.

 

You have helped many people!  I'm hoping you can help me too, to understand how a correcting parity check can cause data loss.  It's probably my ignorance, and I have not been around as much in awhile, so I've missed the situations you are referring to.  If you have time, could you point me to a case or two?  Or if you prefer, explain how it can happen.

[RobJ]

It seems to me (I am more of a hardware guy than a software guy) that any drive may become corrupt for a variety of reasons at any time, whether through hardware or software failure.  A fair number of errors reported on the forums are caused by cabling problems and they may affect any drive.  My view is that it is never safe to assume that the data read from one drive will be more likely to be correct than the parity that is read from another.

 

I'm more of a software guy, but I love helping others, and that means learning a fair amount about both hardware and software, so you can help them when they bring their computer to you!

 

You are certainly correct that there are a variety of reasons that data can be corrupted, but when you narrow the focus to problems related to parity protection, it is surprisingly hard to corrupt data.  Software (at least at the application level) is not a source of corruption for us, because parity protection works at a low level.  It ensures the integrity of data AS WRITTEN.  If your song player messes up your song and saves it to disk, parity protection doesn't care about the song, only cares that that corrupted song is kept safe.  As to the hardware side, there are so many checks and tests, watching for corruption and blocking it, retrying each transmission from device to device, and either succeeding or giving up and returning an error.  How often do we actually see real corrupted data?  We see error messages indicating a problem, we see evidence in SMART reports of past problems, etc.  But our systems catch the problems, and deal with them.  If you see a number in a SMART report indicating a sector was remapped, do you actually ever see the damaged data there.  NO, all we really see is possibly a little lost time, and a number change on the SMART report.  Our data is safe, even the data from that sector.  I'm not saying this very well, but I'm trying to point out that our perceptions of the dangers of data corruption are partly colored by our knowledge of all the little corruptions that are constantly happening and being caught and corrected by our systems.

 

You mentioned cable problems, and I would like to make a comment about that.  In my experience, once you have diagnosed an issue as a cable problem, the end user should really breathe a sigh of relief, because he has just experienced the cheapest and easiest to fix and safest issue there is!  Cables are inexpensive.  But more importantly, you know that if it was a cable problem, then your drive is probably completely fine, and your data is also probably completely fine.  Cable problems can start with what looks like a serious drive failure, if the Linux kernel gives up trying to communicate across that bad cable, and disables the drive.  But the drive is fine, just out of contact.  And drive data is fine, untouched, because any attempted data transmissions (reads or writes) that are not perfect - are trapped by CRC tests and rejected.

 

A bad cable will not cause bad data.  If the bad cable causes a bad transmission of a block of data, then that message is rejected and the data is retransmitted, multiple times if necessary until it is perfect, all of which happens transparently to you.  You may only know about it by seeing exception handler messages in your syslog, with either the ICRC or BadCRC error flags.  If your drive was not disabled afterward, then your data was safely read or written, even if it took multiple tries.

 

I'd like to respond to more of the great responses to my initial post, but I'm too tired, so it will have to wait a bit.

[lionelhutz]

I don't really see the point you are trying to make. If there is so much protection on the data being placed onto drives that errors can never happen, then parity errors should never occur and either check should be fine.

 

I have no examples to link without searching back through previous posts.

[dgaschk]

When a data drive is corrupt but not red-balled, a correcting parity check would be the wrong thing to do.

[Joe L.]

You could be right about the SATA data transmission.  I read quickly through how the data is transferred.  It is complex as the scheme used attempts to limit the number of changes from '0' to '1' to reduce the cable bandwidth requirements. 

A similar scheme is used when writing to the disks platters.  The object there is to keep a regular series of transitions, since read heads on disks can ONLY detect transitions, and not steady states.  Even though we think we are storing all zeros when clearing the drives, the platters themselves have a regular series of transitions in magnetic state.  The transition points allow the disk to sync up with what it is reading. The pattern of transitions, the presence or absence of a transition when expected determines if a "1" or "0" is stored on a disk at a certain spot. 

I think you are correct about unRAID attempting to replace unreadable data by using the parity data to reconstruct it.  However, there is going to be a rather lengthy initial delay as all of the drives are going to have to spin up first!  It is my observation, the parity drive is never spun up during normal read operations.  (Someone please correct me if I am wrong!) 

True, the parity disk is not read unless the array is in a degraded state, OR when a read of a data disk fails.  In that situation the code is supposed to spin up all the other drives, read the corresponding sectors, return the re-constructed sector to the OS to be passed on to whatever is reading the data and ALSO write the same (re-constructed) sector back to the disk that had the read failure.    In theory, (assuming a sector was marked as un-readable) this should allow the SMART firmware on the disk to re-allocate the sector automatically.  In other words, self-repairing of the sectors marked as un-readable.

 

I've seen many instances though of smart reports on data disks where many sectors are marked as un-readable and are pending re-allocation.  I have no explanation other than perhaps those sectors were discovered by internal tests of the disk  (and smart "long" or "short" test), and not by "read" failures.  Otherwise, I'd almost never expect to see a sector pending re-allocation.

As I see it, the parity data in unRAID is basically intended to be only used when a disk is replaced and the data needs to be restored to that disk.  I doubt if you really want to run very long in a emulated disk mode!  The parity check feature is only there to provide some level of confidence that can be done.

Actually, parity is also used for any read-failure, even if just to re-construct the failed read and write the sector back to the failed disk.  Disks are not taken off-line (disabled) for read failures.  They are disabled only for "write" failures.

 

Again, as far as cabling goes, either it is seated correctly, and has good connectivity and shielding or not.  The CRC checksum  on the SATA communications will prevent parity errors from being caused by a bad cable or cable connection.  I see many cases where tight bundling of multiple SATA cables causes errors (CRC errors) as one cable induces noise into an adjacent one.

The fix for that is easy... don't make it so neat, or get better SHIELDED SATA CABLES.

 

I still feel the majority of random parity errors from a specific hard disk are caused by flaky electronics in the disk, most likely the internal RAM cache.

 

Joe L.

[Joe L.]

I'm hoping you can help me too, to understand how a correcting parity check can cause data loss.  It's probably my ignorance, and I have not been around as much in awhile, so I've missed the situations you are referring to.  If you have time, could you point me to a case or two?  Or if you prefer, explain how it can happen.

Let me try.

 

Let's say only correcting parity checks exist.

Assume we have a data disk will flaky internal electronics.  It returns bad data every few hours of operation, but no other errors.

We perform a parity "check" which reads the bad data, and then updates parity.  Parity is now correspondnig to the bad data.    Parity will now re-construct the bad data that was read, not the original contents for that disk.

In addition, for ANY other disk, for that sector, a re-construction will also result in the incorrect re-constructed sector.

 

Once we have written parity with the wrong value, it basically prevents the reconstruction of that same sector on any disk.

All it would take to lose data is for ANY disk to to fail.  You take your chances...  It the sector part of the un-used space.. or in a precious file, or in the internal superblock on the file-system.  The effect could be anything from the loss of a file, to the loss of the entire disk's set of data. (assuming the user cannot recover with a file-system-repair, or just gives up and pre-clears again and re-formats.)

 

If non-correcting "checks" are available, the first read of bad data will result in the inconsistent parity being detected and reported.  Parity is NOT changed.

If we investigate and determine it is data disk2 that occasionally returns random values, we can un-assign disk2.  While un-assigned, the re-constructed sector from parity and the other disks can be used to copy your precious data from disk2 elsewhere.  It can also be used to re-construct the correct data onto a replacement disk2.   

 

Joe L.

[bonienl]

I'm hoping you can help me too, to understand how a correcting parity check can cause data loss.  It's probably my ignorance, and I have not been around as much in awhile, so I've missed the situations you are referring to.  If you have time, could you point me to a case or two?  Or if you prefer, explain how it can happen.

Let me try.

 

Let's say only correcting parity checks exist.

Assume we have a data disk will flaky internal electronics.  It returns bad data every few hours of operation, but no other errors.

We perform a parity "check" which reads the bad data, and then updates parity.  Parity is now correspondnig to the bad data.    Parity will now re-construct the bad data that was read, not the original contents for that disk.

In addition, for ANY other disk, for that sector, a re-construction will also result in the incorrect re-constructed sector.

 

Once we have written parity with the wrong value, it basically prevents the reconstruction of that same sector on any disk.

All it would take to lose data is for ANY disk to to fail.  You take your chances...  It the sector part of the un-used space.. or in a precious file, or in the internal superblock on the file-system.  The effect could be anything from the loss of a file, to the loss of the entire disk's set of data. (assuming the user cannot recover with a file-system-repair, or just gives up and pre-clears again and re-formats.)

 

If non-correcting "checks" are available, the first read of bad data will result in the inconsistent parity being detected and reported.  Parity is NOT changed.

If we investigate and determine it is data disk2 that occasionally returns random values, we can un-assign disk2.  While un-assigned, the re-constructed sector from parity and the other disks can be used to copy your precious data from disk2 elsewhere.  It can also be used to re-construct the correct data onto a replacement disk2.   

 

Joe L.

I can understand the situation you describe above and to a certain extend can go along but - at least in my view - parity should not be used to find faulty disks. There should be other means of warnings telling a particular disk is generating read or write errors. These 'early' detections can be used to either repair or replace the disk in question. In the latter case parity is used to reconstruct the content on a newly inserted disk.

[Joe L.]

I can understand the situation you describe above and to a certain extend can go along but - at least in my view - parity should not be used to find faulty disks. There should be other means of warnings telling a particular disk is generating read or write errors. These 'early' detections can be used to either repair or replace the disk in question. In the latter case parity is used to reconstruct the content on a newly inserted disk.

Unfortunately, most people do not watch the syslog that closely.  The first indication of an "error" is when either

1. A "read" error occurs (and it shows as an error on the web-management page),

or

2.  A "write" error occurs (and the disk is taken off-line and is shown as disabled on the web-management page)

or

3. A "parity check finds a parity error. (and it shows on the web-management page)

 

Joe L.

[Frank1940]

I can understand the situation you describe above and to a certain extend can go along but - at least in my view - parity should not be used to find faulty disks. There should be other means of warnings telling a particular disk is generating read or write errors. These 'early' detections can be used to either repair or replace the disk in question. In the latter case parity is used to reconstruct the content on a newly inserted disk.

Unfortunately, most people do not watch the syslog that closely.  The first indication of an "error" is when either

1. A "read" error occurs (and it shows as an error on the web-management page),

or

2.  A "write" error occurs (and the disk is taken off-line and is shown as disabled on the web-management page)

or

3. A "parity check finds a parity error. (and it shows on the web-management page)

 

Joe L.

 

+1

 

A parity check looks at all of the data in the complete array.  It tests every bit of every file in the entire array.  It is the perfect test to verify that unRAID can read EVERYTHING that is currently stored on the array with resorting to data recovery.  What kind of a test would you (bonienl) propose to be able to do this and not look like a non-correcting parity check? 

 

 

[bonienl]

I created the "Health" add-on for simpleFeatures. When that page is opened it will start monitoring in the background the SMART status of each active disk (optionally spun down disks too).

 

You need to stay on that page though to keep monitoring. I can think of a system which is always running in the background and give notifications (also part of SF) as soon as something is out of the ordinary. This might be SMART monitoring or whatever clever thing we can bring up.

 

Personally I don't want to wait until my (monthly) parity check runs to find out there is an issue lingering, potentially present for days.

[chickensoup]

I created the "Health" add-on for simpleFeatures. When that page is opened it will start monitoring in the background the SMART status of each active disk (optionally spun down disks too).

 

You need to stay on that page though to keep monitoring. I can think of a system which is always running in the background and give notifications (also part of SF) as soon as something is out of the ordinary. This might be SMART monitoring or whatever clever thing we can bring up.

 

Personally I don't want to wait until my (monthly) parity check runs to find out there is an issue lingering, potentially present for days.

 

Is this something you would integrate with email notifications? Sorry I don't currently use SF but will probably run it when i jump up to v5.

[bonienl]

Continuous disk monitoring needs to be developped, but yes it can (will) be used together with email notifications, hence the name

[Frank1940]

I created the "Health" add-on for simpleFeatures. When that page is opened it will start monitoring in the background the SMART status of each active disk (optionally spun down disks too).

 

You need to stay on that page though to keep monitoring. I can think of a system which is always running in the background and give notifications (also part of SF) as soon as something is out of the ordinary. This might be SMART monitoring or whatever clever thing we can bring up.

 

The problem that I see with SMART is that each disk manufacturer reports back different attributes and there are differeances in the way that they report the ones that are common between them.  The board is filled with requests from users asking whether the information in the reports that they have means anything and what does it tell them about the disk that the report is for. 

 

Plus, if you have a large aray, it takes a long time to just get the reports on each individual disk and even longer to interpet each one.

 

As I see it, SMART is a tool that is most useful once you are aware of a problem.  The report MIGHT help you identify a hard disk that is giving a problem...

 

Personally I don't want to wait until my (monthly) parity check runs to find out there is an issue lingering, potentially present for days.

 

If you look at the automated parity check in Simple Features, you can set the interval as 'weekly' or even 'daily' if you desire.  I have a new unRAID build and I will be running the tests (manually) at this point at approximately weekly intervals for six to eight weeks.  I don't know where the idea that a 'monthly' parity check interval should be the 'standard' came from.  I think each user should set to the interval by knowing the amount, the type and difficulty of reconstructing his data and he is comfortable with that interval since he is the only one who knows how much risk he is willing to assume.

[bonienl]

The problem that I see with SMART is that each disk manufacturer reports back different attributes and there are differeances in the way that they report the ones that are common between them.  The board is filled with requests from users asking whether the information in the reports that they have means anything and what does it tell them about the disk that the report is for. 

 

Plus, if you have a large aray, it takes a long time to just get the reports on each individual disk and even longer to interpet each one.

 

As I see it, SMART is a tool that is most useful once you are aware of a problem.  The report MIGHT help you identify a hard disk that is giving a problem...

Its true that no full standardization exists in the SMART world. Though there are a number of 'common' denominators, checking these in the background at regular intervals with a simple good/bad result might help already a lot. Just a flag to say attention is required. This kind of warning system is not present today, but it can be envisioned.

 

If you look at the automated parity check in Simple Features, you can set the interval as 'weekly' or even 'daily' if you desire.  I have a new unRAID build and I will be running the tests (manually) at this point at approximately weekly intervals for six to eight weeks.  I don't know where the idea that a 'monthly' parity check interval should be the 'standard' came from.  I think each user should set to the interval by knowing the amount, the type and difficulty of reconstructing his data and he is comfortable with that interval since he is the only one who knows how much risk he is willing to assume.

There is no definite "best" interval to run parity checks. I guess the monthly period comes from previous experiences that people have with unRAID. And at some point the monthly period was considered a good trade off.

 

Of course feel free to change to your own desires. As the author of the "scheduled parity check" in SF I have made it extremely easy to set whatever schedule fits best to your needs.

 

There is no default interval given in the settings, people are - more or less - obliged to chose something of their own, and yes for the very brave there is even a 'yearly' schedule available

[vca]

I created the "Health" add-on for simpleFeatures. When that page is opened it will start monitoring in the background the SMART status of each active disk (optionally spun down disks too).

 

You need to stay on that page though to keep monitoring. I can think of a system which is always running in the background and give notifications (also part of SF) as soon as something is out of the ordinary. This might be SMART monitoring or whatever clever thing we can bring up.

 

Personally I don't want to wait until my (monthly) parity check runs to find out there is an issue lingering, potentially present for days.

 

Personally I don't put much faith in the SMART reports, I've had a few drives replaced and if I recall correctly the SMART reports showed them all as good - but the manufacturer's testing tool showed issues. 

 

A case in point, about 2 months back I had a WD 2TB green go bad, I ran a parity check (non correcting) on a Thursday and all ran fine.  I copied some files to the array on Friday (and some copied to the drive that eventually failed).  Occasionally (say every 2 to 3 months) I run a short SMART test on all the drives and save the reports (I'm still trying to see if SMART ever predicts a failure), and on the Saturday morning I decided to do this.  So I launched all the short tests, waited a few minutes and started gathering SMART reports and saving them.  I was able to do this for all but one drive.

 

For some reason that drive would not respond to the system.

 

Unraid had now marked it as failed and when I looked in the log file the first time the drive had reported any issues (remember I had copied files to it the day before and it had passed a full parity check the day before that) was when it was accessed to do the smart test.

 

So, I concluded, SMART kills drives...

 

I had a hot spare in the array so I unassigned the bad drive, assigned the spare drive and rebuilt without incident.

 

Then I removed the bad drive from the unraid box and put it in my Windows desktop machine, I got the smart report there (which didn't show anything odd) and then ran WD's diagnostic on it.  WD's diagnostic did find issues with the drive, in fact it found blocks that needed remapping and when it tried to do so gave me a message that the remapping had failed.  Writing zeroes to the drive seemed to work and another read worked and then writing zeros ran into the same issue. 

 

At this point the SMART report still said the drive was fine...

 

I RMA'd the drive without any question from WD.

 

Oh, and my vote is for the NOCORRECT option being the default.  One of my previous drive failures would show the parity being checked and a few blocks (say 3) being corrected and then the next parity check I did it would say 3 blocks needed correcting and then the new few checks would be fine and then the issue would repeat (where there were two parity checks in a row that corrected the same number of blocks).  Turned out that one data drive would occasionally return different values for some of its blocks and this would cause the parity to get flipped and then the next time it would get flipped back again.  You can read that unhappy tale here:

 

http://lime-technology.com/forum/index.php?topic=11515.msg109840#msg109840

 

 

Regards,

 

Stephen