unRAID Development

We have plenty of drive options:

2 TB

WD EARS with a jumper

WD EADS

Seagate LP with a firmware update

Hitachi

<2 TB

WD EARS with a jumper

Any other WD

Any Seagate

Any Samsung

Any Hitachi

Incompatible with unRAID

Samsung F4s (2 TB)

Any 3 TB drive

Even if they did work with our current hardware, I personally wouldn't buy 3 TB drives until they offer the best bang for your buck, which I expect won't be for another 6 months at least.

Are you suggesting that Samsung, and the other manufacturers, will come around and start including a jumper setting too?

I'm saying we don't know what they will do... and there are more options than the WD jumper to skin that cat.  What about Hitachi and other manufacturers?

This is not just a tweak to the UI or a new version of Samba.... this is deep-in-the-brain neurosurgery w/r/t unRAID.  The alignment issue can't be addressed in a vacuum.  It is also wrapped up with drives >2TB, GPT partitions, EFI BIOS, 4K sectors being exposed to the OS.  This all needs to shake out because we are going to have different solutions from different vendors.  Trying to do just the alignment issue doesn't make sense, and trying to do all of it at this early stage of the game is premature -- it is a moving target.

Who knows if other vendors will 1) have a jumper like WD, or 2) make the offset standard, so you need a jumper to turn it OFF, 3) have no option to turn it off or on, 4) continue 512 byte sector translation or expose 4K sectors.... or even something else.  Who knows if that they do today, will be what they do in 6 months?  Will there be a way to detect it by the OS?

And I sure hope that not all paths forward involve getting new hardware

Uhhhh... drives are new hardware.  And you most assuredly need new controllers (or mobos with new controllers/BIOS) for SAFE usage of >2TB drives.

OK, I see your point.  Hitachi and Seagate haven't come out with their Advanced Format drives yet, so it might be premature to roll out a new feature aimed at dealing with 4k sector drives.  But I'm still very concerned, as things can move quickly.  Western Digital seemed to carry EARS and EADS drives together for a while (though now it seems to be hard to get a 2TB EADS drive), but Samsung seemed to pretty much decide one day that they would stop making 2TB HD203WIs and switched to the HD204UIs.  How much warning are we going to get before Hitachi and Seagate roll out their Advanced Format drives?  It's certainly going to happen.  The major hard drive manufacturers all agreed to only use Advanced Format in all new drive products.

Plus, isn't this a solved problem?  This shows my ignorance on the subject, but I thought Windows Vista SP1, Windows 7, MacOS, and newer versions of Linux all play nicely with Advanced Format drives.  Why can't unRAID?  Why can't LimeTech just update unRAID to a newer kernel and have things simply magically work? Even if there is a moving target, it seems like hard drive manufacturers are going to make sure they stay compatible with Windows 7.

Rajahal-

Are you sure <2TB Samsung drives are safe?  I think they moved their 1.5TB drives over to 4k sectors too.

The unRaid alignment issue (sector 64 instead of 63) is NOT in the kernel. It's in how the new filesystem is created, specifically how the drive partitions are created. The part that needs to be updated is in the management console / unraid drivers in how it detects valid unraid drives. There is no "magical" update. As already outlined and explained earlier in this thread, there are several issues to deal with that shouldn't just be hack-n-slashed developed.

If Samsung has moved their lesser size drives over to advanced format too, then the immediate solution is obvious . That issue along with their well known and documented below-ambient drive temp reporting is more than enough to not be concerned with what they do. Just avoid Samsung drives.

Another word on Linux Kernel updates, the unRAID 5.1 will be moving over to a recent version of Slackware that does use a recent version of the Kernel. This is required for AFP support at a minimum. The Slackware 13.2 makes use of 2.6.35.7 as of last week. The current version of Linux Kernel is 2.6.35.8 and 2.6.36 has also been released.

Regardless of how much of an issue *we* think it is or opinions on the matter, it would be nice to have a statement of intent towards the issue from Limetech. Otherwise we're all left in the dark which will start to get awkward as time marches on.

We're already discussing if certain drives should be avoided.

Rajahal-

Are you sure <2TB Samsung drives are safe?  I think they moved their 1.5TB drives over to 4k sectors too.

Source?  I've only seen 2 TB F4s.

Samsung Electronics has today (8/3/10) unveiled to the world its latest EcoGreen series storage products, the 1.5TB and 2TB EcoGreen F4 3.5-inch hard drives featuring three 667GB platters and a 23% lower power consumption in standby compared to their predecessors (F3EG models).

http://hothardware.com/News/Samsung-Introduces-15TB-and-2TB-EcoGreen-F4-Hard-Drives/

Just haven't released the 1.5TB for sale...yet.

Plus, isn't this a solved problem?  This shows my ignorance on the subject, but I thought Windows Vista SP1, Windows 7, MacOS, and newer versions of Linux all play nicely with Advanced Format drives.  Why can't unRAID?

Those are not RAID appliances.

Why can't LimeTech just update unRAID to a newer kernel and have things simply magically work?

I can't believe someone even typed that question.

Even if there is a moving target, it seems like hard drive manufacturers are going to make sure they stay compatible with Windows 7.

You like Win7 so much, keep your data on it.

We're already discussing if certain drives should be avoided.

We've discussed avoiding certain drives ages ago before this greater than 2TB item was even a concern. You make it sound like it's a bad situation. It is not.  It's a matter that all mass storage forums should have as a live topic. How else would the community figure out reliability issues whether it's high Load Cycle Counts (WD), bogus Firmware (Seagate), unusually high early failure rates, or click of deaths (IBM Dethstars), or incorrect Temp Reporting (Samsung), or consumer fixable performance issues (WD EARS) or even consumer unfixable performance issues (Samsung)?

BRiT-

I don't know what Windows 7 does to take care of the alignment issue, but it seems like by the very fact that Windows 7 does play nicely with 4k sectors there must be an existing solution to this problem.  I don't know what they do.  Do they detect the drive is a 4k drive and treat them differently?  Or do they work with all drives differently?  I see some vague references to disk accesses in Win7, Mac, and Linux using "atomic" accesses, where they're accessing a specific byte on the drive instead of the entire sector.  I'm not really sure why that helps, but is it something that could be done in unRAID?  There might be several things in unRAID that have to change to support 4k drives, but it seems like if Microsoft has a permanent solution, there's a fixed target for a solution.  Are there specific things about the way unRAID works that makes it more difficult for unRAID to deal with 4k drives?

As it stands, it seems like the Western Digital approach with the jumper is the crudest hack job for compatibility.  You're tricking the OS so its not writing to the sector is thinks it is.  That seems like a bad long-term solution, and the one that could easily create more compatibility problems down the line.

Rajahal-

It doesn't look like its for sale yet, but Samsung's website talks about the 1.5 TB HD155UI.  It seems like a safe bet that Samsung will stop making the older HD154UI in the not-too-distant future.  But, as is described in this writeup on the 4k transition, all new product lines will be "Advanced Format compatible" after January 1st.

We're already discussing if certain drives should be avoided.

 

We've discussed avoiding certain drives ages ago before this greater than 2TB item was even a concern. You make it sound like it's a bad situation. It is not.  It's a matter that all mass storage forums should have as a live topic. How else would the community figure out reliability issues whether it's high Load Cycle Counts (WD), bogus Firmware (Seagate), unusually high early failure rates, or click of deaths (IBM Dethstars), or incorrect Temp Reporting (Samsung), or consumer fixable performance issues (WD EARS) or even consumer unfixable performance issues (Samsung)?

The difference being that's always come from the angle of individual drives (or at an extreme, brands) being problematic (and alot of the issues you've listed above don't really fall under reliability concerns). But unraid still being happy to use them should you decide to trust the hardware.

Now we're talking about the other way round. The drives being ok but Unraid not supporting them.

That is 'a' situation. I don't think I'm intoning it's bad either way right at this moment in time. But if unraid doesn't address it then things *could* end up where we have limited or no new drive support.

This will have to be addressed at some point and in some way. All I suggest is that a hint as to how that may happen and when would be a very useful thing. Or it's not addressed, whilst every other storage appliance and OS out there will have done so, if they haven't already.

If we're all looking to make / keep unraid useful and competitive going forward then this big 'shakeup' in storage has to be looked at. Even if that's just a case of limetech saying 'not doing anything about it just now'. Otherwise we'll keep speculating ad infinitum on the forum like this. I don't think that's an unreasonable thing to be putting out there.

As already explicitly explained in this thread and the other thread where Limetech has direct communication with us, some of the magic is starting the partition table on sector 64 instead of sector 63. I suggest you search, find, and read that other thread and this thread again to get a better comprehension of the larger picture ramifications involved with simply changing the starting partition sector.

As for dealing with larger than 2TB drives, that is an entirely larger can of worms that the current hardware and drive firmware and bios firmware makers have not fixed yet. It would be premature to even consider supporting larger than 2TB drives until a long-term industry standard solution is delivered by those other players.

This will have to be addressed at some point and in some way. All I suggest is that a hint as to how that may happen and when would be a very useful thing. Or it's not addressed, whilst every other storage appliance and OS out there will have done so, if they haven't already.

Nothing can be done by a software-only maker until the current hardware and drive firmware and bios firmware makers have delivered an industry standard solution. In this case, software only can not fix physical issues.

I have every confidence that once the needed prerequisites are delivered and solved, then unRAID will readily and easily support larger than 2TB drives. The issue will be addressed at some point, but not until the physical issues are dealt with first.

Nothing can be done by a software-only maker until the current hardware and drive firmware and bios firmware makers have delivered an industry standard solution. In this case, software only can not fix physical issues.

Other software makers seem to have managed...

I'm also not expecting an out the air fix tomorrow. I'm just suggesting it would be nice to know how / if / when this might be addressed.

I have every confidence that once the needed prerequisites are delivered and solved, then unRAID will readily and easily support larger than 2TB drives. The issue will be addressed at some point, but not until the physical issues are dealt with first.

That's a bit too bury your head in the sand for me, but I guess we have little choice!

For the record the Guys from FreeNAS made there appliance compatible with the 4k drives.

New Features

==========

 

- Samba 3.5.5.

- AIO settings from the WEBGUI.

- AMD CPU on-die digital thermal sensor.

- Advanced format 4kb sector (UFS/GPT data partition)

(data partition in the boot disk is always aligned to 32KB)

- Virtual machine guest support (VMware and Virtialbox)

While some may have early support for 4K drives, all software makers are still at the mercy of hardware makers for physical support including SATA controllers, motherboard, and BIOS for larger than 2TB drive support. It does little to no good for the consumer if the rest are missing support.

While some may have early support for 4K drives, all software makers are still at the mercy of hardware makers for physical support including SATA controllers, motherboard, and BIOS for larger than 2TB drive support. It does little to no good for the consumer if the rest are missing support.

 

 

Aren't they two separate issues given we've already established <=2TB drives are going 4k as of early next year?

I'm with you on the 3TB stuff but the 4k sectors is here and now, and fixable (though I can see why it will be a pig of a job for unraid)

From what I've seen the 4K sectors are not exposed yet outside of the hard drive. The existing drives (Samsung and WD) use 512 byte sector emulation.  So it's not an issue, yet.

Other software makers seem to have managed...

Name one that is an unstriped RAID5 implementation, which runs on off-the-shelf generic hardware.

I don't understand why people are getting spun up in this thread.  We all want basically the same thing, and that is to continue to be able to support a diverse set of hardware, including drives from various manufacturers, in our unRAID arrays.

unRAID is using an RFS format that has data sectors starting at sector 63.  It needs to start at 64 (or some number divisible by 4).  I believe that ALL our drives should be migrated to a sector 64 alignment - new ones, old ones - all of them.  If people are using the jumper to change the alignment, they will need to remove the jumper.  It may be something of a PITA to get from here to there, but with a consistant format all of the features we've come to expect will continue to work - including rebuilding an old disk on a new disk without disturbing parity.

The alternative is to depend on a klugey fix being provided by a single vendor (WD), and take choice out of the consumer's decision as to what drives to purchase.

Tom recognized this 4 months ago, and decided to wait until more drives came on the market.  Look at this post by Tom.  He realized back then that this was an issue he'd need to address.  If you rewind in that thread a little you'll see he was on the verge of implementing a fix at that time, but he decided to give it some time.  

But some time has passed, and soon the time will come when the fix is needed.  Debating what that fix should be should be openly discussed here.  It will give us something to do and may help Tom decide on the ultimate solution.

Agree 100%.  My point was that once you raise the hood to change the timing belt, don't button it back up just to undo it the next day to change the air filter.

The answer to >2TB drives may be GPT, so why mess around with manipulating offsets in an MPR partition, when you may be going to GPT.... or even partitionless drives?

But I'm interested in what Tom comes up with in any event.... particularly what happens if you replace a failed 2TB WDEARS with the jumper, with a different 2TB advanced format drive w/o the jumper.

Changing sector alignment has huge ramifications for existing systems, doesn't it? Wouldn't you essentially have to rebuild your entire array drive by drive?

And I believe fdisk doesn't support GPT, so those tools would have to be added to the package, necessitating probably quite a bit of testing and integration.

I expect that it's all going to take some time to figure out. I'm sure we'll get there in the end.

Other software makers seem to have managed...

 

Name one that is an unstriped RAID5 implementation, which runs on off-the-shelf generic hardware.

None? Do I win a lollypop?

I'm not sure what you're getting at. I'm fully aware it will be a pain to 'fix' unraid, but it's all in the software. Not fixing it because 'its hard' isn't really a very good excuse.

I should also add, as bjp999 said I'm not sure why people are getting so irate and argumentative.

a) Whatever we discuss here doesn't fix or solve anything

b) people seem to be diving down my neck for suggesting it would be nice to have a better idea of how limetech will handle it. The post linked to by bjp is almost 5 months old and we're now about to hit the conditionals for not doing it laid out in that post. I think that's a fairly reasonable thing to *ask* (note, not *demand*) an update for.

To be honest I'm not in hurry. I'll rather wait longer, and have a good solution.

Of course unRAID has to support 4k drives in the future also drives >2TB.

Other software makers seem to have managed...

 

Name one that is an unstriped RAID5 implementation, which runs on off-the-shelf generic hardware.

 

None? Do I win a lollypop?

 

I'm not sure what you're getting at. I'm fully aware it will be a pain to 'fix' unraid, but it's all in the software. Not fixing it because 'its hard' isn't really a very good excuse.

 

I should also add, as bjp999 said I'm not sure why people are getting so irate and argumentative.

 

a) Whatever we discuss here doesn't fix or solve anything

 

b) people seem to be diving down my neck for suggesting it would be nice to have a better idea of how limetech will handle it. The post linked to by bjp is almost 5 months old and we're now about to hit the conditionals for not doing it laid out in that post. I think that's a fairly reasonable thing to *ask* (note, not *demand*) an update for.

 

 

Agreed, way too much hostility here over a simple question.  If people forgot the original question that started this...

Now we just need to hear about the possible inclusion of 3 TB hard drives, I have heard some of the more notable members say why it wont work now, but I'd like to hear from Tom and what his plans are.

The question wasn't will current hardware work with it, why won't AF drives work with unraid, what do you guys think about the problems, what drives can we just not use at all, or whatever other answers have been brought up.  It was simply what is his plans for supporting hard drives that cannot currently be used (without workarounds).