Guest

SSDs as array drives question

34 posts in this topic Last Reply

Recommended Posts

6 minutes ago, SSD said:

 

Would Plex metadata create a significant write amplification issue?  Plex metadata files are tiny, but there are a zillion of them!

Most probably. But as long as they aren't constantly updated they probably can't combine and reach into TB-class disk wear.

Share this post


Link to post

Just an addendum.

 

While enterprise disks can perform at optimum speed without trim, they can still take advantage of trim to reduce the write amplification.

If a 128 kB flash block contains 64 kB erased data and 64 kB current data and the OS wants to rewrite the 64 kB of current data then the disk has two options

- if the 64 kB erased data has been reported through trim, then the drive can erase all content on that flash block while finding a suitable flash block for the new 64 kB write. So 64 kB of file writes results in 64 kB of flash writes.

- if the 64 kB erased data hasn't been reported, then the drive will pick a suitable spot for the 64 kB write. Then it will still think 64 kB data is valid on the original block. So when it later decides to erase that flash block, it first has to copy 64 kB to another block. So a single 64 kB file write results in 64 + 64 kB of flash writes. So without the trim, there was a larger write amplification.

 

Another thing to consider here is that unRAID, like SnapRAID, doesn't stripe.

 

So with 4+1 system with a traditional striped RAID 5, a 1 TB file write will result in 1.25 TB of flash writes since the parity adds 25% additional surface.

 

With unRAID, a 1 TB file write means the parity drive also has to handle a 1 TB write - so in total 2 TB of SSD writes.

 

In a traditional RAID 5, all drives gets the same amount of wear. You can select enterprise or normal drives depending on amount of writes the system needs to handle.

 

In unRAID, the parity drives would need to be expensive enterprise drives to be able to handle the additional number of TB of writes. Or the user would have to replace the parity drives 5-10 times more often depending on number of data disks.

 

One strong part about unRAID is that the unstriped data means only needed HDD has to be spinning. This is excellent for media consumers. With SSD this doesn't really matter because they don't produce noise and consume as much power which means that one of the unRAID advantages would no longer hold true. unRAID would still have a big advantage that each drive holds a valid file system so you can't get a 100% data loss because you lost one drive more than the number of parity drives.

 

In the end, I think it would be better if unRAID could add support for multiple arrays - mixed freely between unRAID arrays and normal RAID-0, RAID-1, RAID-5, ... than to focus on running a strict SSD main array. Especially if unRAID could still merge multiple arrays into unified user shares. That would allow people to select two-disc SSD RAID-1 for applications and special data while keeping the huge bulk data on much cheaper HDD.

Share this post


Link to post
50 minutes ago, pwm said:

Especially if unRAID could still merge multiple arrays into unified user shares. That would allow people to select two-disc SSD RAID-1 for applications and special data while keeping the huge bulk data on much cheaper HDD.

I think I know what you mean in the big picture, but you do know you just described exactly to a T what currently is in place with the normal array + BTRFS RAID 1 cache that is set up by default if you add 2 SSD's to the cache pool, right?

 

Or were you being funny and making a point that we already have exactly what is needed without additions?

 

Sorry if I misinterpreted your intent, it just came across to me as amusing.

Share this post


Link to post
Just now, jonathanm said:

I think I know what you mean in the big picture, but you do know you just described exactly to a T what currently is in place with the normal array + BTRFS RAID 1 cache that is set up by default if you add 2 SSD's to the cache pool, right?

Yes, I know about the cache pool. But that's just a single additional mirror.

 

My main storage server is currently running 3 separate mirrors besides the main disk array. And I plan to add a fourth mirror and possibly a second full disk array when I continue to throw in more disks. The only way I can merge unRAID with the other functionality is if I run unRAID virtualized to handle a single array in the main server. Many separate mirrors means I can get huge, disk read/write speeds to keep a 10Gbit/s network happy streaming media concurrently with making larger backups.

 

So currently, I have multiple unRAID for secondary tier storage pools. Which also means I can't take full advantage of the Docker/VM support in unRAID because all the nice SSD mirrors are on the main server.

Share this post


Link to post

I had never thought about that approach before. Since I only want my music and docs on an SSD, do you recommend using Sync to move files from the SSD over to the main array for backup as a better option?

Share this post


Link to post
On 12/31/2016 at 11:50 PM, antaresuk said:

I read in the getting started guide that this isnt a good idea because of no TRIM support. What do people think of this, is this a big issue? Do others have all SSD as their array? Please forgive the newbie question  :)

 

I had a chat recently with the guys who develop all-flash NAS, they told me that there must be specific algorithms developed for this kind of media. Optimize writes to minimize wear out.

 

Besides you need specialized hardware, and honestly I don't think unRaid was built for speed

Share this post


Link to post

So, if TRIM issues only come up with parity RAIDs, would I be safe with unRAID making a RAID 1 array with my SSDs? If unRAID is my main OS, could I boot Windows from that array despite having a X99 system?

Share this post


Link to post

The way I am handling this is by having my SSDs setup in a btrfs RAID-0 cache pool, then syncing the information off onto the array for redundacy nightly. Of course, between syncs&movers the data on this cache pool is vunerable, but having 1TB of space available at SSD access speeds is worth it to me. I really only use it for cacheing and my Steam games library. So if you are just using it for your audio and video files, something like this might be a viable option for you.

Edited by ryoko227

Share this post


Link to post
On 1/2/2017 at 1:59 PM, limetech said:

A file system issues TRIM commands as a result of deleting a file to tell the ssd that the set of blocks which previously made up the file are no longer being used.  The sdd can then mark those blocks as 'free'.  Later when the ssd internal garbage collection runs, then it knows that it doesn't have to preserve the contents of those blocks.  This makes garbage collection more efficient.  There are lots of articles that explain this.

 

The trouble this causes for parity-based array organizations is that the data returned from a TRIM'ed data block can be indeterminate.  This paper is a bit wordy but lays it out on p. 13:

 

 

 

To boil this down for unRAID: it should work to use SSD's in an unRaid P or P+Q array if TRIM is not used.  This is current behavior.  However note that:

a) Write performance can degrade faster on data disks depending on how many file deletions take place.

b) The parity disk is also written for each data disk write.

c) The data disks really should be completely written first because theoretically a block that was never written from the point of view of the SSD, can return non-deterministic data for those blocks.  We have not seen this happen, but then again we have not run too many SSD arrays (it would show up as parity sync errors).  This is pretty undesirable thing to do however since it will guarantee slowing down subsequent writes.

d) If you don't want to pre-write the disks as above, then only use SSD's that support "DX_TRIM" or "DZ_TRIM", and instead of writing the disks with zeros, simply use 'blkdisard' command to first TRIM the entire device instead.

 

You can use the 'hdparm' command to determine if your SSD's have this support:

 

 


hdparm -I /dev/sdX   # substitute X for your ssd device assignment
 

 

 

You want to look near the end of the "Commands/features:" section for:

 

          *    Data Set Management TRIM supported

 

Following this will either see this:

 

          *    Deterministic read data after TRIM

 

or you will see this:

 

          *    Deterministic read zeros after TRIM

 

or you won't see either of the above (if this is the case, do not use in unRAID P or P+Q array).

 

In a future release we do plan to add proper TRIM support to array disks.  Here's a heads-up on that.  In order to support TRIM in unRaid P or P+Q array, we must add code to the md/unraid driver and all SSD's in the array must support either "DX_TRIM" or "DZ_TRIM" mode as described above.  In addition there's a really good chance we will only support SSD's that support "DZ_TRIM" since to support "DX_TRIM" is a lot more work  ;-)

 

Thanks for this post.  Right now I have one SSD in my array that has been there for years and has not been at all problematic.  It uses "Deterministic read data after TRIM."  However, this drive is getting too small for my needs and I want to get a larger drive now, so naturally I want to make sure it is compatible with unRAID TRIM in the future. 

 

If only DZ_TRIM is going to be supported, I want to find a disc that supports this.  I however can't figure out a way to discover what type of TRIM any given SSD supports short of buying it and putting it into a Linux machine and checking.  Is there a list or any reference available to figure out what SSD drives support DZ_TRIM TRIM?  That said, there are also several hardware versions of the 840 PRO so who knows if they all handle TRIM the same way?!?

 

I read a random post in some random Linux forum that the Samsung EVO 840 PRO supports DZ_TRIM but that the 850 does not.  That is literally all the info I can find about SSD drive trim types after a lot of Googling.  Anyone have any guidance?

 

Thanks,

craigr

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


Copyright © 2005-2018 Lime Technology, Inc.
unRAID® is a registered trademark of Lime Technology, Inc.