HPA a.k.a. why are two identical drives different sizes?


Recommended Posts

What is HPA?

HPA (Host Protected Area) is a 'feature' (read: curse) of certain motherboards that is supposed to help recover from a corrupted BIOS.  At the time of this writing (Feb. 2011), only certain older Gigabyte motherboards are known to force an HPA on hard drives.  I don't know the exact cut-off date, but if you are using a Gigabyte motherboard manufactured before 2010 then you should definitely check to see if you have HPA on any of your drives.  Also, it is definitely possible to have an HPA from a previous motherboard.  If a drive has at any point in time been used with one of these HPA-inducing boards, then there's a good chance that it still has a vestigial HPA, even if it has been formatted since.

 

What is the danger?

HPA essentially shrinks the capacity of the drive.  As far as unRAID or any other OS is concerned, the drive is now smaller than it used to be.  In short, this causes your unRAID array to be unreliable in the event of a disk failure.  Read on to see why.

 

Let's consider an example.  Say you are building a new server out of a mixture of 1 TB drives and an old Gigabyte motherboard that has HPA enabled by default.  Say your parity drive is a 1 TB drive with a capacity of 1953525168 sectors.  All your data drives must be 1953525168 sectors or smaller.  As the motherboard boots for the first time, it will first search all the disks connected directly to it via the onboard SATA ports (not on disks connected via a SATA expansion card) for an HPA containing a current copy of its BIOS.  Since this is a brand new build, it won't find one.  It will then create an HPA on the first drive that initialized.  This isn't dependent on the number of the SATA port (SATA0, SATA1, etc.).  This may or may not be the parity drive, it depends on how long each drive took to initialize.  If there are six drives connected to the motherboard, any one of those six could be a potential target for an HPA.  Let's say for example that the motherboard chooses the parity drive for its HPA.  The parity drive now reports a size of 1953523055 sectors to unRAID, instead of its native size of 1953525168 sectors.  All of a sudden the parity drive is not the largest drive in the array, at least from unRAID's point of view, and the 1 TB drive is an invalid candidate for parity.  This is about the best possible scenario, since as this is a new build it would be fairly easy to identify the problem and rectify it.

 

Now let's consider the worst possible scenario.  Say you built the server above and just allowed the HPA to reside on one or more of your data disks.  As long as the parity disk is the largest, unRAID will still function normally even if multiple disks have an HPA.  For this example let's say that your array has only one disk with an HPA - disk1.  The parity and all the other data disks are HPA free.  Every time the server boots the motherboard searches for an HPA, finds it on disk1, and then contented it takes no further action.  Then..disaster strikes - disk1 fails.  You shut down the server, replace disk1 with another 1 TB drive, then boot up expecting to be able to recover disk1's contents onto the new disk.  Alas, you are greeted with a 'parity is not the largest disk' message.  What happened?  When you booted the server after replacing the failed disk, the motherboard again searched for a valid HPA, but this time didn't find one, as disk1 was no longer present.  So it created a new HPA on another disk, which just so happened to be your parity disk.  Your array is now essentially missing two disks - disk1 has failed, and the former parity disk is now smaller than it should be, so it isn't a valid parity disk.  At this point you would be unable to recover the data from disk1 without the careful guided help from members of this forum.  Even with that help, there's still a chance of permanent data loss.

 

How will I know it when I see it?

HPA goes by many names in the BIOS settings.  It has been called:

  • Save a Copy of BIOS to HDD
  • BIOS Backup
  • Backup BIOS to HDD
  • Xpress BIOS Rescue

...and I'm sure there are other names as well

 

unRAID identifies any HPA it finds in the syslog.  It looks like this:

Feb  8 07:41:22 Tower kernel: sd 1:0:0:0: [sda] Attached SCSI disk

Feb  8 07:41:22 Tower kernel: ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 0)

Feb  8 07:41:22 Tower kernel: ata2.00: HPA detected: current 1953523055, native 1953525168

Feb  8 07:41:22 Tower kernel: ata2.00: ATA-7: SAMSUNG HD103UJ, 1AA01118, max UDMA7

Feb  8 07:41:22 Tower kernel: ata2.00: 1953523055 sectors, multi 16: LBA48 NCQ (depth 31/32)

Feb  8 07:41:22 Tower kernel: ata2.00: configured for UDMA/100

 

So you can always just search your syslog for the phrase 'HPA' to see if you have any in your system.

 

I've identified one or more HPAs in my array, what can I do about it?

For the short term, you can see if you can disable the BIOS feature that is creating a Host-Protected-Area on your disks.  This is NOT a good solution since some day, when your CMOS battery dies, the feature will re-enable itself, and it WILL change the size of one of your disks, and it will cause you to all kinds of grief.

 

If you cannot disable the feature, you basically either need to get a different motherboard, or a BIOS upgrade.  The newest of Gigabyte BIOS have the feature disabled by default.  This is what you want.   The older of Gigabyte BIOS have no way to disable the feature, those are not acceptable for unRAID use (or ANY raid use for that matter)

 

Then, you need to remove the HPA from your parity drive.  This will them make it appear as its correct full size.

 

Do not be fooled into thinking just disabling the feature in the BIOS is enough...  It must be disabled by default.  Other unRAID users have actually swapped out motherboards because the feature either was not disabled by default, or could not be disabled at all.

 

How do I remove HPA from a drive?

The HDAT2 Method (requires the ability to hook up a CD drive to your server - an external optical drive is fine)

HDAT2 Guide

 

The hdparm -N method (uses either the unRAID system console or telnet)

hdparm -N Guide

 

More Info

Wiki HPA Topic

 

Special Note regarding unRAID 4.7

If you are upgrading from unRAID 4.6 (or below) to unRAID 4.7 (or above) you MUST remove any HPAs on any of your drives BEFORE the upgrade can be successful.  See this thread for more information.

Link to comment

Another possible means of determining if you have HPA is manually issuing the following command. This will probe all of your drives, including your USB Flash drive, so it's natural to see an HDIO_DRIVE_CMD failed for that drive. This takes the guesswork out of knowing what your drives are.

 

hdparm -N /dev/[hs]d[a-z]

 

/dev/sda:
max sectors   = 312581808/312581808, HPA is disabled

/dev/sdb:
max sectors   = 3907029168/3907029168, HPA is disabled

/dev/sdc:
max sectors   = 3907029168/3907029168, HPA is disabled

/dev/sdd:
max sectors   = 3907029168/3907029168, HPA is disabled

/dev/sde:
max sectors   = 3907029168/3907029168, HPA is disabled

/dev/sdf:
HDIO_DRIVE_CMD(identify) failed: Invalid argument

/dev/sdg:
max sectors   = 3907029168/3907029168, HPA is disabled

/dev/sdh:
max sectors   = 3907029168/3907029168, HPA is disabled

 

Link to comment

Let's consider an example.  Say you are building a new server out of a mixture of 1 TB drives and an old Gigabyte motherboard that has HPA enabled by default.  Say your parity drive is a 1 TB drive with a capacity of 1250263728 bytes.  All your data drives must be 1250263728 bytes or smaller.  As the motherboard boots for the first time, it will first search all the disks connected directly to it via the onboard SATA ports (not on disks connected via a SATA expansion card) for an HPA containing a current copy of its BIOS.  Since this is a brand new build, it won't find one.  It will then create an HPA on the first drive that initialized.  This isn't dependent on the number of the SATA port (SATA0, SATA1, etc.).  This may or may not be the parity drive, it depends on how long each drive took to initialize.  If there are six drives connected to the motherboard, any one of those six could be a potential target for an HPA.  Let's say for example that the motherboard chooses the parity drive for its HPA.  The parity drive now reports a size of 1250261615 bytes to unRAID, instead of its native size of 1250263728 bytes.  All of a sudden the parity drive is not the largest drive in the array, at least from unRAID's point of view, and the 1 TB drive is an invalid candidate for parity.  This is about the best possible scenario, since as this is a new build it would be fairly easy to identify the problem and rectify it.

 

Just wanted to clarify one point.  Gigabyte motherboards will NOT put an HPA on a disk that is already partitioned by unRAID (including the parity disk, data disks, and the cache disk).  Instead it will place an HPA on a BRAND NEW DISK.  So if you have an array of formatted disks, than then replace your motherboard with one of the problematic Gigabyte boards (not that you'd ever do this, but this is just an example) - it will not place an HPA on any disk.  All will be fine until you add a new disk to the computer.  Then, when you put a new disk in the machine and turn the computer on, the BIOS will place the HPA on the disk before you have a chance to preclear or do anything else on the disk.

 

See next post.

Link to comment

I need to issue a retraction of my last post.  It is not so cut and dry the conditions under which Gigabyte boards lay down an HPA.  Refer to this link to see that users are seeing HPAs added to disks that are already formatting for RFS and ZFS file systems in place.  There may be special logic to not corrupt a Windows NTFS partition (or I expect Windows users would be screaming bloody murder), but I am not 100% sure of even this.

 

The only thing I know is that these guys are idiots to have implemented HPA in this way!

 

Bottom line - HPA is a bad thing.  Do not use a motherboard with HPA enabled.

 

I am working on an update to myMain to tell you if you have HPAs on your disks.  Stay tuned - hopefully have it ready this weekend.

 

 

Link to comment

Another possible means of determining if you have HPA is manually issuing the following command. This will probe all of your drives, including your USB Flash drive, so it's natural to see an HDIO_DRIVE_CMD failed for that drive. This takes the guesswork out of knowing what your drives are.

 

hdparm -N /dev/[hs]d[a-z]

 

/dev/sda:
max sectors   = 312581808/312581808, HPA is disabled

/dev/sdb:
max sectors   = 3907029168/3907029168, HPA is disabled

/dev/sdc:
max sectors   = 3907029168/3907029168, HPA is disabled

/dev/sdd:
max sectors   = 3907029168/3907029168, HPA is disabled

/dev/sde:
max sectors   = 3907029168/3907029168, HPA is disabled

/dev/sdf:
HDIO_DRIVE_CMD(identify) failed: Invalid argument

/dev/sdg:
max sectors   = 3907029168/3907029168, HPA is disabled

/dev/sdh:
max sectors   = 3907029168/3907029168, HPA is disabled

 

 

I tried this method but I'm getting crazy results...

 

I can't copy and paste my telnet window, but every drive says HPA is invalid and gives something like this:

 

/dev/sdb

max sectors =18446744072344861488/11041584, HPA setting seems invalid

 

Four of the discs in my array return that identical result, even though two are 2TB drives, one is 1.5TB and one is 1TB.  First I think that sector count seems invalid, and two, it shouldn't be the same for all four discs.

 

I've got three other discs that return an equally puzzling result, but gives a tiny sector count despite being on 1 and 2 TB drives.

 

UPDATE:

 

I updated to version 4.6 of unRAID.  I ran the same command and got a different result that still seems wrong:

 

/dev/sdb

max sectors =3907029168/14715056(18446744072344861488?), HPA setting seems invalid (buggy kernel device driver?)

 

A check of my syslog shows no mention of HPA and reports the sector count of this drive as 3907029168, which is correct.

 

 

 

 

 

 

 

Link to comment
  • 2 weeks later...

I updated my BIOS on my P35-DS3L a few weeks ago and today noticed an option to disable the BIOS saved to HD disk. I'm happy I was able to disable it, but I'm unhappy since I don't know which drives got the HPA crap installed on them. This is in my main PC, but at least I know I can use this board for an unRAID server if need be. This is with the F9 BIOS.

Link to comment

I updated my BIOS on my P35-DS3L a few weeks ago and today noticed an option to disable the BIOS saved to HD disk. I'm happy I was able to disable it, but I'm unhappy since I don't know which drives got the HPA crap installed on them. This is in my main PC, but at least I know I can use this board for an unRAID server if need be. This is with the F9 BIOS.

 

A problem is that the HPA is not off by default. If for some reason your BIOS gets reset, e.g., dead battery, you may be surprised when your drives become corrupted by HPA.

Link to comment

I updated my BIOS on my P35-DS3L a few weeks ago and today noticed an option to disable the BIOS saved to HD disk. I'm happy I was able to disable it, but I'm unhappy since I don't know which drives got the HPA crap installed on them. This is in my main PC, but at least I know I can use this board for an unRAID server if need be. This is with the F9 BIOS.

 

A problem is that the HPA is not off by default. If for some reason your BIOS gets reset, e.g., dead battery, you may be surprised when your drives become corrupted by HPA.

 

Ughh, good point. I really wanted to avoid buying TWO new motherboards, but I guess  could always sell the DS3L. It's a great board as long as you're not using that system for RAID/unRAID.

Link to comment
  • 8 years later...

Shucked a Seagate Backup Plus Desktop Hub 8TB STEL8000200 today, and put in in Fujitsu Siemens Celcius W360, motherboard D2587, bios Phoenix V6.00 R1.18.2587_A1.

Presented itself as 5.3T. :oTook hours before I got it fixed, HPA is a truly a curse!

 

Can I expect this to happen again with this motherboard?

 

 

Link to comment
6 hours ago, reboot81 said:

Can I expect this to happen again with this motherboard?

If it did it once then it can do it again.  So before connecting drives I would check in the BIOS for an option to disable any saving to hard disk of the BIOS itelf or its configuration.  If you cannot disable it then I'd start looking for a different board.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.