unRAID 6.2 - Screenshot Dual Parity support

How about using the second parity to protect part of the array? Lets say you have a 6 drive array (1x parity, 5x data) consisting entirely of 8TB drives. Adding a 2TB second parity would then provide 10TB of "double" protection and 30TB of normal protection. Seems possible based on my rudimentary understanding of parity calculation. Key word here being "rudimentary"... 

Rationale:

1) It's usually a lot easier to squeeze an extra 2.5" hdd into a case than a full-size spindle.

2) Most arrays are probably filled only partway with important content.

3) It's a way to repurpose smaller hdd's laying around from previous upgrade cycles

How about using the second parity to protect part of the array? Lets say you have a 6 drive array (1x parity, 5x data) consisting entirely of 8TB drives. Adding a 2TB second parity would then provide 10TB of "double" protection and 30TB of normal protection. Seems possible based on my rudimentary understanding of parity calculation. Key word here being "rudimentary"... 

 

Rationale:

1) It's usually a lot easier to squeeze an extra 2.5" hdd into a case than a full-size spindle.

2) Most arrays are probably filled only partway with important content.

3) It's a way to repurpose smaller hdd's laying around from previous upgrade cycles

The real question for me personally with dual parity coming soon, is if parity+btrfs disks can be used to repair bitrot.

this is my main interest in dual parity - fingers crossed

I suppose if LT told us what was in 6.2 the next question would be WHEN.. So at least this way, they can leave us to guess what it contains rather than pester them with WHEN is it out!

at 6:33m in Linus's video it shows a login screen to unRAID webui?  I wonder if that means another change?

Woah, I didn't notice this when I first watched the Video. I was just discussing unRAID with a friend and we watched the video again together and I noticed!

FWIW I hope that the Security that this image seems to imply is being added is "Optional" and we can enable / disable as we see fit. I access the GUI so much (probably far more than I should  ) so I hope we are not forced to log in each time!

I don't know what others think. Never seen security of the GUI (other than general advice to new users not to expose the GUI of their unRAID Server to the Internet) discussed before.

I would imagine that it'll be like telnet access.  Password protected if you set it..

I would imagine that it'll be like telnet access.  Password protected if you set it..

If I understand you correctly I do hope not. That would mean there is an extra step to access information on the GUI even if there is not a password to type (even if when all said and done it is just one click or a key press).

Thinking out loud, perhaps unRAID could use a principal built into dd-wrt (which those familiar will know) which allows for a "status" page which (can be in itself password protected if need be) by default is accessible without a password and allows for a quick view and check of status without having to log in. To make any changes or to access other more sensitive areas of the router config a userid and password is requried.

I thought the only thing you had to do to require login was set a password for root. Haven't done it recently but I know I have gotten a login before.

trurl you're correct.  If you set a root password, as I always do, then I would imagine that would carry over to the gui.  If you don't then I would not expect the gui to be password protected either.

Also your browser could be set to remember the password. So then the only "extra" step would be clicking "OK".

I'd assumed that if a root password was set, the GUI would ask also, and if there isn't one you don't get asked.  However, I've always assumed that, so I could be very wrong.

Given the amount of power the GUI offers the user, the more things you can now do with v6, and the fact that unRAID is appealing to broader audiences (corporate archives, etc.), I would say that the option to hide the GUI behind a login is essential.

I understand that you need to get to it as quickly as possible (and a summary page, or the option to permanently skip the login, or a remember me option would solve these), but when somebody can start, stop, add and remove virtual machines, create, delete and modify network shares, unassign drives, power down, etc., it would seem scary to leave it wide open.

Anyways, I came here to work out what dual parity was, and I now know, I think, so thanks one and all.  About £180.

trurl you're correct.  If you set a root password, as I always do, then I would imagine that would carry over to the gui.  If you don't then I would not expect the gui to be password protected either.

Yeah, this is the way it works.

The real question for me personally with dual parity coming soon, is if parity+btrfs disks can be used to repair bitrot.

Any details on this?

I may be totally missing the mark here but why is dual parity necessary for bitrot protection? Is not having a btrfs formatted drive enough to protect against it? We already have (quasi) basic bitrot protection in the case of a bit flipping on a single drive in the form of parity checks (although this is made tricky by not knowing whether the parity drive or the array drive had its bit flipped).

I don't think dual parity has anything to do with bitrot protection but I could be wrong, my understanding is that dual parity will just protect against more then one drive failure. Am I missing something?

So going from memory and summarizing, currently if your drives are btrfs and a file gets bitrot, unRAID has the ability to inform you about it but it does NOT have to ability to correct the corruption. You would still need to use checksums to find out which file is bad and restore it from backups. In order for btrfs to restore your data it needs another disk (like our parity disk?, correct me if I am wrong??). The theory was with dual parity you could somehow incorporate both protecting unRAID's data disks and also use it to correct the corrupt bitrot data. AFAIK, this was just a theory and may or may not even be possible.

So going from memory and summarizing, currently if your drives are btrfs and a file gets bitrot, unRAID has the ability to inform you about it but it does NOT have to ability to correct the corruption. You would still need to use checksums to find out which file is bad and restore it from backups. In order for btrfs to restore your data it needs another disk (like our parity disk?, correct me if I am wrong??). The theory was with dual parity you could somehow incorporate both protecting unRAID's data disks and also use it to correct the corrupt bitrot data. AFAIK, this was just a theory and may or may not even be possible.

Ohhh this makes sense now! I was under the impression btrfs could both detect a checksum mismatch AND correct it. Apparently it can only warn you of a checksum mismatch and you need another copy of the data on another disk to fix the data.

This does still mean that you could fix the bitrot with current versions of unRAID though, if you run a parity sync and untick the 'Write corrections to parity disk' option to tell unRAID you trust the data on the parity disk and not the other way round. This is a very hokey method though, so I can see the interest now in whether dual parity can help edge cases of bitrot (where multiple disks have bitrotted, the parity drive has bitrotted, more than one drive has bitrotted in the exact same bit, etc).

Also, BTRFS can do snapshots, so if you have a snapshot of your user dir say and then at some point you are informed of bit rot corruption, you could theoretically restore files from a snapshot.

unRAID parity doesn't know anything about the data or the filesystems on the drives, so I don't expect that unRAID parity will ever be able to work together with BTRFS checksums to fix problems.

With one parity disk unRAID can detect a parity error, but it doesn't know for sure which disk the error is on.  unRAID assumes the error is on the parity drive, because true bitrot is assumed to be extremely rare.  Hard drives are pretty good at what they do, after all

If my understanding is correct, with two parity disks it should be possible to pinpoint which disk has the error and therefore correct it.  So if you run a parity check with two parity drives, in theory, unRAID should be able to find and fix bitrot.  But Limetech has not actually come out and said that yet.

unRAID parity doesn't know anything about the data or the filesystems on the drives, so I don't expect that unRAID parity will ever be able to work together with BTRFS checksums to fix problems.

 

With one parity disk unRAID can detect a parity error, but it doesn't know for sure which disk the error is on.  unRAID assumes the error is on the parity drive, because true bitrot is assumed to be extremely rare.  Hard drives are pretty good at what they do, after all

 

If my understanding is correct, with two parity disks it should be possible to pinpoint which disk has the error and therefore correct it.  So if you run a parity check with two parity drives, in theory, unRAID should be able to find and fix bitrot.  But Limetech has not actually come out and said that yet.

Yet ....

Thanks for the explanation. I appreciate it. Would be a nice feature if they do implement it. Would make using 3rd party methods / plugins / software redundant!

When dual parity is enabled and running how would a parity check work? Just an example, I have a 4TB parity drive and that takes some where from 10-18 hours to complete. If I drop another 4TB in for dual parity does is check them both at the same time and double the time it takes? I can see parity checks taking several days with larger drives, and I always try not to use the array when a parity check is going because I do see a performance decrease and didn't want to screw up the parity check. If it takes days for a parity check I assume there will be some users that will be forced to use their system as normal while doing a parity check. Read/write speeds would drop considerably, especially an all mechanical system. Is this correct thinking?

EDIT: I read in this thread concern about unraid moving towards favoring a gaming OS. Maybe put up a poll and take a vote on if this is what users really want.

Dual Parity is built and checked at the same time.

I read in this thread concern about unraid moving towards favoring a gaming OS. Maybe put up a poll and take a vote on if this is what users really want.

I feel that their is a need for a separate "I want to be just like Linus" subforum. 

When dual parity is enabled and running how would a parity check work? Just an example, I have a 4TB parity drive and that takes some where from 10-18 hours to complete. If I drop another 4TB in for dual parity does is check them both at the same time and double the time it takes? I can see parity checks taking several days with larger drives, and I always try not to use the array when a parity check is going because I do see a performance decrease and didn't want to screw up the parity check. If it takes days for a parity check I assume there will be some users that will be forced to use their system as normal while doing a parity check. Read/write speeds would drop considerably, especially an all mechanical system. Is this correct thinking?

 

EDIT: I read in this thread concern about unraid moving towards favoring a gaming OS. Maybe put up a poll and take a vote on if this is what users really want.

A useful explanation of dual (or P+Q) parity protection. It is the same algorithm used in RAID6 - although RAID6 stripes the data while unRAID keeps parity data on dedicated disks.

DUAL PARITY EXPLANATION

A useful explanation of dual (or P+Q) parity protection. It is the same algorithm used in RAID6 - although RAID6 stripes the data while unRAID keeps parity data on dedicated disks.

 

DUAL PARITY EXPLANATION

Oh interesting, I didn't realize it was possible to calculate a second parity value using the same bits.  I thought we'd being doing diagonal parity, which would require a different set of bits to be used. This is more mathematically intense, but wouldn't need any additional disk reads.

This method would be able to detect and correct a single bit of bitrot, right?  Seems like it just comes down to some math.