Parity drive..... can it be a few bites smaller than a data drive?

Sorry if this a repeat question ( i tried searching, but got 18 pages of matches for "parity drive" ) 

 

I'm just taking the leap to 2TB drives and as many do, I just buy what I need at the moment ( as prices always get cheaper per GB over time ).

 

My current setup I have Seagate and WD Green drives in it.    I used my Seagate for parity, when I set things up, simply because it was faster than the WD Green drive.  When I look at my Main UnRaid Menu I can see the size of the Seagate is just a tiny bit larger:

 

Seagate: 1,465,138,552

WD Green: 1,465,137,496

 

I suspect I will likely be getting a mix of sizes of 2TB drives, all depending on what is on sale when I need them.    What if I was to get a WD Green drive now and set it as my parity, then in a month get a Hitachi 2TB as a data drive and discover its a few bits smaller in size?

 

Does UnRaid just block off those last few bites and not use them? 

 

I'm still uncertain which drive I am getting for my 2TB parity.  My gut tells me to stay away from the "LP" or "Green" drives for parity, but they are ok for data.    I presently have 6 drives in my array and only one is "Green", the other 5 are regular Seagates.

Your WD Green drive has has an HPA (host protected area) added by your Gigabyte BIOS.  It uses this for the Xpress BIOS Rescue feature and writes a copy of the BIOS tothe area it reserves for itself.  It artificially makes the drive appear smaller to the operating system.

 

This "feature" of the BIOS has caused lots of problems in unRAID servers since the BIOS will add the HPA if it does not see one that already exists.  If it does this to a data drive, the drive will be invalidated and unRAID will consider it to be a new drive.  If it does this to the parity drive, it will be smaller than the data drives and the array will not start.

 

If the WD drive were to fail, and eventually all disks fail, the BIOS will at random pick the first drive it can and add one at the next boot.  It usually does not overwrite a formatted drive, but the parity drive is not formatted, therefore it is a potential target.    Now, think what happens when one of the data drives in your array fails...  You put in a new drive of equal size.  Oops.. the HPA is added and now it is smaller.  Can't use it to rebuild onto.  Or, worse, the HPA is added instead to the parity drive... now it is invalidated since it is the wrong size...  Can't restore anything, as you have no valid parity.

 

Therefore the answer to your question is:

No, the parity drive may not be smaller than the data drives.

AND

You need to update the BIOS on your motherboard to one where the HPA feature is disabled by default.  And, after doing that, you need to remove the existing HPA from ALL your disks and make all your disks appear their true size.

 

The BIOS must have this feature disabled by default.  Otherwise, you have a ticking time-bomb.  On newer motherboards Gigabyte learned their lesson and it is now disabled.  On some older BIOS it is enabled by default and may be disabled, and on others there is not a way to disable it at all.

 

You have a bit of reading ahead of you.  The HPA is covered in the wiki here:

http://lime-technology.com/wiki/index.php?title=UnRAID_Topical_Index#HPA

 

The size difference has nothing to do with brand, or being green.

 

Joe L.

 

Good thing I asked.  I prefer to have no bombs in my machine

 

If the motherboard is responsible for adding the HPA, then why didn't it do it to my Seagate drive(s)?

 

My purchase plan is to get a 2TB drive now to replace my 1.5TB Seagate drive.    I have made no decision on what brand to buy.

 

The WD Green Advanced Format drive was looking as a possible option, however with the HPA needing to be disabled and Advanced Format Feature, it make me think: why not just get a different brand of drive that works out of the box.

 

I only have the one WD Drive in my server at present and its a Data Drive.  Hopefully I can disable the HPA while its full of data.  I will read up on the Wiki to get more details.

 

Thanks!

Good thing I asked.   I prefer to have no bombs in my machine

 

If the motherboard is responsible for adding the HPA, then why didn't it do it to my Seagate drive(s)?

It only needs one.  It will only add one if one does not currently exist.  The danger is if the existing drive with it dies, then it will choose a different drive, and you'll find yourself in a messy situation since it will then be disabled by unRAId since the size is different.

My purchase plan is to get a 2TB drive now to replace my 1.5TB Seagate drive.    I have made no decision on what brand to buy.

 

The WD Green Advanced Format drive was looking as a possible option, however with the HPA needing to be disabled and Advanced Format Feature, it make me think: why not just get a different brand of drive that works out of the box.

 

I only have the one WD Drive in my server at present and its a Data Drive.   Hopefully I can disable the HPA while its full of data.   I will read up on the Wiki to get more details.

 

Thanks!

Yes, you can disable the HPA but then you'll need to reset the partition size to use the full drive.  See this thread

http://lime-technology.com/forum/index.php?topic=5072.0

for a script I wrote to help one user.  The unraid_partition_disk.sh  script can be used after the HPA is removed, but before you next start the array.

 

The HPA has absolutely nothing to do with the brand of disk drive... It is a Gigabyte BIOS issue.  It will add an HPA to any brand drive if it is enabled and it does not currently detect one on one of the disks.  It will do it without you knowing, and before you can even boot unRAID as it does it when the BIOS is first detecting the hardware.

 

Joe L.

Apparently I was mistaken about which drive was smaller.  I thought it was my WD drive, however I actually have 2 Seagate drives that are smaller than the WD drive.  It looks like the HPA is on two.    Odd... i thought it just had to save to one drive?

 

Perhaps one of the drives didn't show up on reboot, so the HPA feature selected a different drive to infect. This is why anyone running RAID of any sorts loathes HPA. It's a ticking time-bomb unless the BIOS has it disabled by default.

I read through that one users issues with HPA ( he was in a bad way as he couldn't load his array and access important data plus his drive size was a fraction of what it should be ).

 

Should I be concerned about squeezing a few more bytes out of the 2 drives with the HPA on them?

 

I stand to gain:

1,465,138,552 -

1,465,137,496

----------------

            1056 (bytes I think)

 

Or would the simplest thing be to just shut HPA off on my motherboard ( assuming I can figure that out ) to prevent any possible future writing to my new drives or parity drive?

I read through that one users issues with HPA ( he was in a bad way as he couldn't load his array and access important data plus his drive size was a fraction of what it should be ).

 

Should I be concerned about squeezing a few more bytes out of the 2 drives with the HPA on them?

 

I stand to gain:

1,465,138,552 -

1,465,137,496

----------------

            1056 (bytes I think)

 

Or would the simplest thing be to just shut HPA off on my motherboard ( assuming I can figure that out ) to prevent any possible future writing to my new drives or parity drive?

simplest, yes, but best, no...

 

It is a ticking time-bomb.  Unless the feature is disabled by default it is just waiting for the battery on the motherboard to fail and the bios revert to its factory defaults.

 

Yes, do get a replacement/upgrade of the BIOS, yes, get rid of the existing HPAs (although that is not as simple as getting rid of the HPA alone, since when you do it unRAID will see the disk as a different size and think it has changed... So do them one at a time and allow unRAID re-construct the contents onto the "new drive.  Once one is done, you can do the other.

 

Joe L.

If the battery was to die ( or, in my case, I was about to replace it as I have an issue with my server not holding the time ), even if the HPA was disabled on the Bios, the fact that its still done it's "thing" to 2 of my drives, I still run a risk of the "time bomb" ?

Yes, unless the bios defaults the settig to DISABLED. When/if the battery is replaced or dies, BIOS will reset to its default values and on boot it may pick a different hard-drive to infect with HPA. There is no guarantee as to which drive it will infect, it could be an already infected one or a non-infected one.

What board is it?

Ahhhh....that makes perfect sense.

 

So...no easy way around this then.

 

I have a few tasks to perform, but before I start them I figure I best get this sorted out.

 

I have one of the 8 port SATA cards to install as well as two new hard drives.  I also want to replace the motherboard battery ( although I doubt this is my problem with my server losing a second every few minutes , but to satisfy Gigabyte, i will replace it ).

 

Having the spare hard drives now, would it be easier to install the new drive into the array, move all the data off the HPA infected drive and then just pre-clear it?  ( Repeating that process twice as I have 2 drives infected, both 1.5TB drives ).   My new drives are also 1.5TB drives.

 

EDIT:  The board is the Gigabyte MA74GM-S2

Having the spare hard drives now, would it be easier to install the new drive into the array, move all the data off the HPA infected drive and then just pre-clear it?  ( Repeating that process twice as I have 2 drives infected, both 1.5TB drives ).   My new drives are also 1.5TB drives.

The Host-Protected-Area is completely hidden.  The drive actually reports is is smaller than it actually is physically.  You cannot "clear" it until AFTER you eliminate the HPA area.

 

You must run one of the utilities to reset the HPA.

Some have had luck with the "hdparm -N" command that exists in unRAID, other use MS-DOS based utilities, others use tools provided by the disk manufacturers.  In ALL cases, once the HPA has been set, it cannot be changed again unless you power cycle the drive.  (It is allowed only once each time you apply power)

 

Once the disk is reporting its true physical size, then you can put it back in the server and do whatever you want with it.  Since unRAID tracks the mode/serial number of the drives in addition to the size, a change in size will be treated as if the drive is a replacement for the existing drive.  unRAID will offer to re-construct the old contents onto the new.  You should let it.

 

You can only do this to one drive at a time, otherwise 2 new disks would not allow any re-construction, as it would be the equivalent of having 2 failed disks at the same time. 

Just confirming this is what i need to do, to complete this operation:

 

1.  Reboot Server and log into the BIOS and look for an option to disable HPA

My motherboard is running rev.2.0 , so hopefully that will give me the option to disable it

 

2.  Let Unraid load, however Stop the array

 

3.  Telnet into the server

 

4.  Run the command:

smartctl -a -d ata /dev/sda

 

5.  Run the command (not sure if I need to change directories to run it or if it will be in the root folder after I telnet in):

hdparm -N p1465138552 /dev/sda

( the size was taken from the image I posted earlier as I have 3 of these same drives, 1 larger than the other as it doesn't have HPA on it )

 

6. Restart the array and let UnRaid detect the drive has changed ( because of the size difference ) and click the appropriate option that lets the drive be rebuilt.

 

7.  (possible option) Run a parity check before starting process on 2nd drive with HPA

 

8.  Repeat steps 2 through 7, however this time change it to dev/sdf

 

 

If the default state, for HPA in the BIOS is "Enabled", then what's to stop it from being re-activated again later, as suggested "If the battery dies on the Bios" ?

 

Just confirming this is what i need to do, to complete this operation:

 

1.  Reboot Server and log into the BIOS and look for an option to disable HPA

My motherboard is running rev.2.0 , so hopefully that will give me the option to disable it

 

2.  Let Unraid load, however Stop the array

 

3.  Telnet into the server

 

4.  Run the command:

smartctl -a -d ata /dev/sda

 

5.  Run the command (not sure if I need to change directories to run it or if it will be in the root folder after I telnet in):

hdparm -N p1465138552 /dev/sda

( the size was taken from the image I posted earlier as I have 3 of these same drives, 1 larger than the other as it doesn't have HPA on it )

Close... but the argument to hdparm is NOT the number of bytes... but instead, the number of sectors.

Use the following command to learn the correct number to use

hdparm -N /dev/sda

 

See the manual page here for details: http://www.unix.com/man-page/Linux/8/hdparm/

 

6. Restart the array and let UnRaid detect the drive has changed ( because of the size difference ) and click the appropriate option that lets the drive be rebuilt.

You got it. 

Make sure you click on "Start" to begin the re-construction.

Do NOT click on the button labeled as "restore" as that immediately invalidates parity and sets a new initial disk configuration.

 

7.  (possible option) Run a parity check before starting process on 2nd drive with HPA

 

8.  Repeat steps 2 through 7, however this time change it to dev/sdf

Sounds good.

If the default state, for HPA in the BIOS is "Enabled", then what's to stop it from being re-activated again later, as suggested "If the battery dies on the Bios" ?

You've finally caught on...  That is the big problem... You will, if you care about your array, upgrade the BIOS with one that has the feature disabled by default, if Gigabyte makes it available (call their Tech support and complain loudly) or as a number of unRAID users have already done, replace the motherboard.

 

Joe L.

From Server:

 

Linux 2.6.31.6-unRAID.

root@Tower:~# hdparm -N /dev/sda

/dev/sda:

max sectors   = 18446744072344859375/2930277168, HPA setting seems invalid

 

root@Tower:~# hdparm -N /dev/sdb

/dev/sdb:

max sectors   = 18446744072344861488/2930277168, HPA setting seems invalid

 

root@Tower:~# hdparm -N /dev/sdc

/dev/sdc:

max sectors   = 1465147055/1465149168, HPA is enabled

 

root@Tower:~# hdparm -N /dev/sdd

/dev/sdd:

max sectors   = 1953525168/1953525168, HPA is disabled

 

root@Tower:~# hdparm -N /dev/sde

/dev/sde:

max sectors   = 18446744072344861488/2930277168, HPA setting seems invalid

 

root@Tower:~# hdparm -N /dev/sdf

/dev/sdf:

max sectors   = 18446744072344859375/2930277168, HPA setting seems invalid

 

 

So, the commands I would run are:

hdparm -N p2930277168 /dev/sda

 

(Rebuild Parity then):

hdparm -N p2930277168 /dev/sdf

 

Based on this explanation, the second number is what I want.

 

For those that understand linux better, why is the first number so high?   Is it just a really really high number so that no drive ever gets limited by it?

 

 -N     Get/set  max  visible  number of sectors, also known as the Host

     Protected Area setting.  Without a parameter,  -N  displays  the

     current  setting,  which is  reported  as two values: the first

     gives the current max sectors setting, and the second shows  the

     native  (real)  hardware limit  for  the  disk. The difference

     between these two values indicates how many sectors of the  disk

     are currently hidden from the operating system, in the form of a

     Host Protected Area (HPA).  This area is often used by  computer

     makers  to hold diagnostic software, and/or a copy of the origi-

     nally provided  operating  system  for  recovery purposes.

 

 

 

So, the commands I would run are:

hdparm -N p2930277168 /dev/sda

hdparm -N p2930277168 /dev/sda

Yes.

Thanks for confirming.  One more thing ( yes, I am being overly careful as I have 1.5TB of data I could lose, should I mess up a drive ) 

 

 

You can see I ran the hdparm -N /dev/sdx command on all my drives.    The two drives that it shows "HPA is disabled" beside are my two smaller drives ( 1TB and 750GB ).  Of the other 4 drives (all 1.5TB), sda and sde were the data drives that I could see in my UnRaid menu were slightly smaller than my others (1,465,137,496 when the other two were 1,465,138,552). 

 

Summary:

sda - Seagate 1.5TB -1,465,137,496

sdb - WD 1.5TB drive -1,465,138,552

sdc - Seagate 750GB

sdd - Seagate 1.0TB

sde - Seagate 1.5TB -1,465,138,552 - PARITY

sdf - Seagate 1.5TB -1,465,137,496

 

Does the comment "HPA setting seems invalid" mean that even though my sdb and sde drives show a slightly larger size, that the HPA may have wrote to them also?

 

 

Thanks for confirming.  One more thing ( yes, I am being overly careful as I have 1.5TB of data I could lose, should I mess up a drive ) 

 

 

You can see I ran the hdparm -N /dev/sdx command on all my drives.     The two drives that it shows "HPA is disabled" beside are my two smaller drives ( 1TB and 750GB ).   Of the other 4 drives (all 1.5TB), sda and sde were the data drives that I could see in my UnRaid menu were slightly smaller than my others (1,465,137,496 when the other two were 1,465,138,552). 

 

Summary:

sda - Seagate 1.5TB -1,465,137,496

sdb - WD 1.5TB drive -1,465,138,552

sdc - Seagate 750GB

sdd - Seagate 1.0TB

sde - Seagate 1.5TB -1,465,138,552 - PARITY

sdf - Seagate 1.5TB -1,465,137,496

 

Does the comment "HPA setting seems invalid" mean that even though my sdb and sde drives show a slightly larger size, that the HPA may have wrote to them also?

 

 

They should all show the same size once you remove the HPA.

 

The HPA command to reset it can only be run once per power cycle.

 

Joe L.

I ran the command:

smartctl -a -d ata /dev/sda

and just saved the info to a text file.

 

I then ran the:

root@Tower:~# hdparm -N p2930277168 /dev/sda

 

And I got instantly:

 

/dev/sda:

setting max visible sectors to 2930277168 (permanent)

Use of -Nnnnnn is VERY DANGEROUS.

You have requested reducing the apparent size of the drive.

This is a BAD idea, and can easily destroy all of the drive's contents.

Please supply the --yes-i-know-what-i-am-doing flag if you really want this.

Program aborted.

root@Tower:~#

 

I'm guessing the "program aborted" without me having a chance to enter Yes is an issue.

 

Going to bed.  Will do battle again tomorrow

 

I have the same question running in two threads.  The other is:

http://lime-technology.com/forum/index.php?topic=2642.75

Sorry... different people reading different threads potentially, so I figured I would continue until the end in both.

 

Hey Guys, can someone recommend what to do?

 

Reading on this topic because due to what I'm about to explain, (and before reading posts here) I bought a Gigabyte motherboard...

 

Had a simple 3 disk array, 512 Gb each disk, all WD. Turns out one of my disks was always failing (not recognized, had to do the unplug/replug/rebuild parity every so often). Thinking it was the disk, I bought a new 1TB drive, to start upgrading, and I swapped this for my parity disk, and the old parity disk replaced my "failing" data disk.

 

After a while, I started to get the same problem, only this time, on what used to be the parity disk (the one that replaced my "failing" disk...). So now I said: Ah ha... Seems the disk mught not have had anything wrong, maybe it's the SATA port on the motherboard...

 

So taking advantage of a trip to the states, I bought one of those CPU/Mobo combos at fry's thinking to replace my current setup and be done with it.  Since I waqs there I also bought one more pair of 1TB drives, to bring my array to an upgraded "full" 2TB of data...

 

So anyway, before leaving I had additionally, since that current motherboard only had 2 SATA connectors and I had needed to buy an add-on card for the third drive, I disconnected the "failing" drive from the motherboard, and connected it to the add-on card. Problem seemed to be solved.

 

Except till a few days ago when it happened again.... SO all my theories out the window... It's not the disk, it's not the controller, can it be the cable?  I replaced it, and for now it *seems* to work...

 

So, if your still reading this far, since the new motherboard I bought is Gigabyte, I started reading on this, to see what I needed to do to not have an additional problem, and well.. it seems I already have it!! Even though my current motherboard is NOT Gigabyte, (It's an ASUS A7NE8-X Deluxe) looking through my last pulled syslog just now, I just saw I DID have and HPA problem.. Great...

 

So, here's the recommendation I need (FInally!):

 

Current Situation:

 

- Asus motherboard + Sempron 2800 2.0 Ghz processor + 1Gb RAM

- 1TB PArity disk, 2 x 512GB Data disks

- Unraid Server 4.2.4

- Seems to have HPA problem

 

Desired Situation:

 

- New Gigabyte motherboard (G41M-ES2L) + Celeron E3200 Processor 2.4 GHz + 2GB RAM

- 3 WD Green 1TB disks

- Unraid Server 4.5.6

- NO HPA Problems...

 

What would the best / correct steps be to upgrade / Fix HPA problems, in the correct order to have the right end result and not mess anything up?

Latest_Syslog_With_Error.zip

Your choice.  I'd fix the HPA issue first, since you are not on a Gigabyte MB, it should not come back.  the disk must have been connected to a Gigabyte MB at some point in its life.

 

Jun 16 15:44:15 Tower kernel: ata5.00: HPA detected: current 976771055, native 976773168

Jun 16 15:44:15 Tower kernel: ata5.00: ATA-8: WDC WD5000AAKS-00D2B0, 12.01C02, max UDMA/133

Jun 16 15:44:15 Tower kernel: ata5.00: 976771055 sectors, multi 16: LBA48 NCQ (depth 0/32)

Your choice.  I'd fix the HPA issue first, since you are not on a Gigabyte MB, it should not come back.  the disk must have been connected to a Gigabyte MB at some point in its life.

 

Jun 16 15:44:15 Tower kernel: ata5.00: HPA detected: current 976771055, native 976773168

Jun 16 15:44:15 Tower kernel: ata5.00: ATA-8: WDC WD5000AAKS-00D2B0, 12.01C02, max UDMA/133

Jun 16 15:44:15 Tower kernel: ata5.00: 976771055 sectors, multi 16: LBA48 NCQ (depth 0/32)

 

Now that you mention it, it is possible... I *think* I pulled at least one I had on a gaming machine or an HTPC machine which I left with a 160Gb or 250Gb one (can't remember now), and that may have happened... So how do I fix that quickly if it's one disk? Just format and recreate or something like that?

 

Thanks!

 

Edit: Well, Just thinking, if I did after all buy the 1TB disks to replace the 500Gb ones, if I just replace the disk with the HPA issue with one of the 1TB ones, that should do it just fine, right?

 

Is there any sort of checker or something that I could use to see if the motherboard I bought has HPA enabled (or if I can remove it completely with another BIOS version?).

Your choice.  I'd fix the HPA issue first, since you are not on a Gigabyte MB, it should not come back.  the disk must have been connected to a Gigabyte MB at some point in its life.

 

Jun 16 15:44:15 Tower kernel: ata5.00: HPA detected: current 976771055, native 976773168

Jun 16 15:44:15 Tower kernel: ata5.00: ATA-8: WDC WD5000AAKS-00D2B0, 12.01C02, max UDMA/133

Jun 16 15:44:15 Tower kernel: ata5.00: 976771055 sectors, multi 16: LBA48 NCQ (depth 0/32)

 

Now that you mention it, it is possible... I *think* I pulled at least one I had on a gaming machine or an HTPC machine which I left with a 160Gb or 250Gb one (can't remember now), and that may have happened... So how do I fix that quickly if it's one disk? Just format and recreate or something like that?

 

Thanks!

 

Edit: Well, Just thinking, if I did after all buy the 1TB disks to replace the 500Gb ones, if I just replace the disk with the HPA issue with one of the 1TB ones, that should do it just fine, right?

Yes, that would remove the disk with the HPA from the array, but not remove the HPA from the disk you remove.

Is there any sort of checker or something that I could use to see if the motherboard I bought has HPA enabled (or if I can remove it completely with another BIOS version?).

Look for a way to enable/disable the "BIOS Quick Recovery" feature in the BIOS.  It should be disabled by default.

(otherwise, it will be a problem when the CMOS battery dies and it gets enabled again)

 

Newer Gigabyte BIOS have it disabled by default and configurable.

Older Gigabyte BIOS have it enabled, and NOT configurable  (These you really want to avoid)

Even older Gigabyte BIOS do not have the "feature" at all.

 

Joe L.

 

Thanks Joe L., so yeah, I know that won't take the HPA off the drive, but since I will reformat that disk afterwards (or even use it in the unraid server anyway), I don't think it'll be a problem (even if it goes back to a Gigabyte motherboard, as it wont be part of the array anymore).

 

Are there any issues / precautions / steps to be taken when one changes the hardware? Or is it as simple as just powering up after the hardware change and that's it? (even though there's plenty about changing DISKS, I couldn't find anything in the wiki about hardware changes / upgrades).