Ultimate Unraid Server - The Sequel

I am planning to build the fastest UNRAID server possible using latest generation hardware and am looking for advice.

I would like to have a discussion on the most modern system components to achieve the following goals (In decreasing order of importance):

NEW SYSTEM GOALS

• Fastest available throughput (read AND write) on Gigabit LAN
  
• Dual Gigabit LAN Ports (for future aggregated/parallel data transfer - not sure if this is supported yet by UNRAID)
  
• At least 6 onboard SATA ports (I see some available boards with 10)
  
• Minimium Power Consumption ** without Sacrificing SPEED **
  
• Minimum Number of Add On Cards/Fans - I believe the most reliable system will be one with minimum components (Cards, Fans, Etc).
  
• Small Form Factor with Capacity for Minimum 5 Drives (I have a 5 in 3 ICYDOCK, but would happily replace it with a better solution if there is something better out there).  More capacity would be better, but I don't see myself using more than 5 drives for the forseeable future.
  

My previous UNRAID server was built to optimize performance with the equipment which was available at that time (2006).  I am planning to retire this setup and replace it with the current "state of the art".  It has run reliably for 4 years.  No issues except for a bad SATA port on the motherboard which I identified when replacing the original 3 hard drives with two 1.5TB Seagate Drives and a 1TB Western Digital Black.

OLD SYSTEM (See My Original Thread and Benchmarks Here:  http://lime-technology.com/forum/index.php?topic=378.0

• ASUS P5LD2-VM R2.0
  
• Cooler Master Centurion 5 Tower with 380 Watt Cooler Master Power Supply
  
• Intel Pentium D 820 Dual Core 2.8GHz, 800MHz FSB
  
• 1GB Dual Channel Corsair XMS2 PC5400 DDR2 675 SDRAM
  
• Lexar Lightning High Speed (100x! 22MBps Read, 15Mbps Write) USB 2.0 Flash Drive
  
• ICY Dock 5 in 3 SATA III Bay (Fits 5 SATA III Drives in 3 Full Height Bays)
  
• Seagate Barracuda 400GB 16MB 7200 RPM SATA/300 HDD (Parity) (Upgraded to 1.5TB Seagate)
  
• Maxtor 300GB 16MB 7200 RPM SATA/150 Drive 1 (Upgraded to 1.5TB Seagate)
  
• Maxtor 300GB 16MB 7200 RPM SATA/150 Drive 2 (Upgraded to 1TB Western Digital Black)

So far, I haven't learned much - but I am beginning to wonder if the new LGA 1156 Intel P55 boards may be worth considering.  Here is an example:

GIGABYTE GA-P55-UD5 LGA 1156 Intel P55 ATX Intel Motherboard

Key Specifications:

Onboard LAN

• LAN Chipset Realtek 8111D x 2
  
• Max LAN Speed Dual 10/100/1000Mbps
  

Storage

• SATA 3Gb/s 10 PORTS
  
• SATA RAID 0/1/5/10
  
• PATA 1 x ATA133 2 Dev. Max
  

Memory Standard

• DDR3 2200/1333/1066/800
  
• Dual Channel (Could go Tripple Channel with an x58 Board)
  

Has anyone done any benchmarking to identify how UNRAID read/write performance scales with the following parameters:

• Processor Performance and Architecture/Generation (Atom, Core 2, i3, i5, i7)?
  
• How does it scale with Memory Performance (DDR2 vs DDR3, Single vs Dual vs Tripple Channel, Memory Bandwidth)?
  
• How does it vary with different chipsets?  (HDD Controller, Memory Controller)
  
• How does it vary with different read/write speeds of USB Flash Drives?
  

This is the motherboard I'm going to buy when it is released.  Low power next generation Atom 510, dual Intel GigE nics, 6 SATA 2 and 1 PCI-E 4x.  Seems to me to be the perfect board for this application.  The Intel LAN is what did it for me, because there are plenty of people on here who have expressed the superior traffic management of that chipset over the Realtek / Marvell.  Power consumption according to Supermicro is about 15 W for the CPU, so the board probably eats around 25 when going full tilt.  The biggest downside to this board is the rather odd SO-DIMM choice.  But from what I've gathered here, RAM speed is not nearly as critical as I/O to the drives.  Oh, and my Hitachi Deskstar drives get about 130 MB/s on the edge and about 60 MB/s on the inside of the platter.

Wait for SATA-3?

I am planning to retire this setup and replace it with the current "state of the art".

Your "old system" is 10 times more powerful than what you need for unRAID server.

If you are looking for a way to spend some money, then spend it on some disks, like the 2TB WD RE-4.

That appears to be a reasonable choice, but I would like to see some quantitative data on how UNRAID READ/WRITE network performance varies with processor performance and architecture (particularly on large file transfers). 

It is actually the newer dual core Atom processors that started me thinking about an UNRAID update, but I would only go this route if it produced the exact same performance as a faster processor/chipset/RAM configuration. 

I would gladly burn more watts for faster transfer performance. 

I wish there was someone around here with a variety of different motherboards and processors sitting around (which would include Dual Core Atom all the way up to Core i7).  The test I would do is to load the system with the fastest practical HDD available (an Intel SSD) and show how gigabit network READ WRITE speeds vary among the processors and chipsets with UNRAID.

A great start would be to see the extremes with those SSDs alleviating the HDD bottleneck:

Dual Core Atom and Dual Core DDR2 vs x58 Overclocked i7 Processor with Tripple Channel DDR3.  Perhaps as a baseline, an older Dual Core Intel system thrown in for good measure.

Once the optimal motherboard and processor are identified, we could do some tests where the SSDs are traded out with various large capacity HDDs to identify how that then reduces the best case performance.  Unfortunately, I don't have all of this hardware at my disposal.

 

Your "old system" is 10 times more powerful than what you need for unRAID server.

 

If you are looking for a way to spend some money, then spend it on some disks, like the 2TB WD RE-4.

 

 

Since when does "what you need" enter into the equation for a performance enthusiast?

This is the motherboard I'm going to buy when it is released.

That's a great find!

 

Your "old system" is 10 times more powerful than what you need for unRAID server.

 

If you are looking for a way to spend some money, then spend it on some disks, like the 2TB WD RE-4.

 

 

 

Since when does "what you need" enter into the equation for a performance enthusiast?

I would like to see some quantitative data on how UNRAID READ/WRITE network performance varies with processor performance and architecture (particularly on large file transfers).

You can spend $10K on hardware and still your UNRAID READ/WRITE network performance won't be any better than mine.

(running a $300 server here)

A "performance enthusiast" would normally look at other RAID solutions, not unRAID.

This is the motherboard I'm going to buy when it is released.

 

That's a great find!

 

 

Indeed it is! If/when I build another machine or one that needs to be smaller then my current one then this may just be the board that I go with!

Now, back onto the original topic.  unRAID is not processor bound and newer faster processors make little to no difference (short of using less power possibly).  Your bottlenecks are going to be LAN speed, Bus speed and disk speed.  When doing parity checks it is best to have all of your drives on some kind of PCI-e bus.  SSD might speed up a parity check but only if all of the drives are SSD, and frankly SSD are FAR to expensive to create massive arrays with right now.  When doing a parity check you are saturating the bus so much and the hard drives are spinning as fast as they can that that is what becomes your bottleneck, not the speed of the processor.  The only thing you would gain from a faster processor is if you plan to run other things (like VMWare) on your server.

Your components seem perfectly fine the only thing I would look at upgrading is the motherboard. Find one that has more onboard SATA ports and plenty of PCI-e x1, x4, and x8 slots.

A "performance enthusiast" would normally look at other RAID solutions, not unRAID.

Agreed to that.  UnRAID is not the fastest parity arrangement, because it relies on one drive to do it all, instead of spreading the parity wealth to all the drives in the array.  No striping = slower speed.  That simple difference is what makes unRAID superior for protection and inferior for speed (write speed anyway).  When I think of a performance box, I think RAID 0 + 1 on 4 10k SATA drives or 15k SAS drives.  While that is awesome for raw speed, it's terribly expensive and limits you to lower capacities per drive.

This is the motherboard I'm going to buy when it is released.

That's a great find!

Indeed it is! If/when I build another machine or one that needs to be smaller then my current one then this may just be the board that I go with!

Thanks guys.  I was really excited when I found it.  I have plenty of space for a normal sized board, but I don't want the power draw on a 24/7 server.

So far, I haven't learned much

That, is an understatement.

Adding CPU horsepower past a Pentium III is WASTED effort.  It has no appreciable impact on unRAID read or write performance.  If you can't understand that NATIVELY and without having to "prove" it with bench testing, then you have, indeed, not learned much.

Because of how unRAID works (reading a sector, then writing back the SAME sector on the disk) you have a huge performance limit due to the use of rotating media.

You also have a performance limit on 1 gigabit ethernet. 

The first step is to eliminate rotating media.... use good SSDs.  You can simulate this and test it by installing 4GB of RAM, and write a 2GB file to the ramdisk (root file system).

If you want "state of the art" then use 10Gbps Ethernet cards and some Fusion IO SSDs for data and parity drives.... and be prepared to replace the parity drive once a year or so.

As long as you want to stick to unstriped, rotating, magnetic media, you have an anchor around your neck.

 

Adding CPU horsepower past a Pentium III is WASTED effort.  It has no appreciable impact on unRAID read or write performance.  If you can't understand that NATIVELY and without having to "prove" it with bench testing, then you have, indeed, not learned much.

 

Feel free to lose the attitude.  If the processor/RAM performance have zero impact on UNRAID speed, great! (this is why I am asking the question)  

Is there data to support this fact?

Regarding why I want to optimize UNRAID vs purchasing another faster solution - I like the features of UNRAID:  

• The fact that only one extra disk is necessary to produce a fully redundant system
  
• The fact that I don't have to worry about a proprietary operating system (which may also be stored on a rotating disk) to worry about as a failure mode.
  
• The fact that it is "portable" and allows me to easily migrate to better and faster hardware as the technology evolves.
  
• The hardware can be reused as a standard PC, server, etc if your needs change.
  

Is there data to support this fact?

Yup... all over the forum and in the Wiki....  You say you have an unRAID server -- have you never profiled it?  Never even checked CPU utilization while calculating parity?  No, you charge into the forum with your wizz-bang ideas and shoot from the hip.

 

You also have a performance limit on 1 gigabit ethernet. 

 

...

 

You can simulate this and test it by installing 4GB of RAM, and write a 2GB file to the ramdisk (root file system).

 

...

 

If you want "state of the art" then use 10Gbps Ethernet cards and some Fusion IO SSDs for data and parity drives.... and be prepared to replace the parity drive once a year or so.

 

Some further thoughts...

Based on what I have been hearing on the other discussions, noone has passed the limits of Gigabit ethernet performance (125MB/s) with UNRAID, so 10GB cards would not help even if practical.  Even $1500+ hardware seems stuck below 90MB/s.

SSD is obviously not a practical solution for large capacity storage, so we are stuck with rotating media for the time being.  

I do like your idea of doing network writes to the RAMDISK for testing purposes to check the theoretical limit of your hardware.  Can you easily expose the RAMDISK to the network through UNRAID?  

unRAID is a proprietary operating system - it is not open source.  The rest of your reasons for sticking with unRAID stand, though.

As others have explained, there is no need to benchmark or 'prove' that unRAID isn't bottlenecked by the CPU or RAM because of the underlying way that unRAID works (which has already been described).

If you want some real-life accounts, here's mine:

I initially built my unRAID server with dual core 2.0 GHz AMD CPU.  I later downgraded it to a single core 1.7 GHz AMD CPU.  My third and final change has been to upgrade it to a low power (45 W) single core 2.7 GHz AMD CPU.  At no point in this process did I notice or clock any difference in read/write speed - it stayed consistent throughout both CPU swaps (downgrade and upgrade).

I have also downgraded and upgrade my server's RAM at various points.  I have gone as low as 1 GB of RAM (2 x 512 mb), and as high as 5.5 GB of RAM (2 x 2 GB + 1 x 1 GB + 1 x 512 MB).  My server currently runs with 4 GBs of RAM (2 x 2 GB).  Again, at no point during these upgrades and downgrades have I noticed or clocked any discernible differences in read/write speeds.  Likewise, RAM tweaks and extras (such as dual channel versus single channel) haven't made any difference whatsoever.

Keep in mind that I run unRAID completely stock, with no add-ons.  If you plan on using add-ons (specifically torrents), then upgraded hardware (RAM, at least) may improve your server's performance.

I'm curious about the impact of a cache drive on the performance of a thoroughly benchmarked unRAID setup.  From what I can gather from posts on here, the cache drive can give you a real boost in speed if you're willing to sacrifice a drive.

unRAID is a proprietary operation system - it is not open source.  The rest of your reasons for sticking with unRAID stand, though.

 

If you want some real-life accounts, here's mine:

I initially built my unRAID server with dual core 2.0 GHz AMD CPU.  I later downgraded it to a single core 1.7 GHz AMD CPU.  My third and final change has been to upgrade it to a low power (45 W) single core 2.7 GHz AMD CPU.  At no point in this process did I notice or clock any difference in read/write speed - it stayed consistent throughout both CPU swaps (downgrade and upgrade).

 

I have also downgraded and upgrade my server's RAM at various points.  I have gone as low as 1 GB of RAM (2 x 512 mb), and as high as 5.5 GB of RAM (2 x 2 GB + 1 x 1 GB + 1 x 512 MB).  My server currently runs with 4 GBs of RAM (2 x 2 GB).  Again, at no point during these upgrades and downgrades have I noticed or clocked any discernible differences in read/write speeds.  Likewise, RAM tweaks and extras (such as dual channel versus single channel) haven't made any difference whatsoever.

 

Thanks for taking the time to provide a useful answer.  Sorry for my word choice - yes, I understand that unraid is somewhat proprietary - but I was trying to relate it to a system like an INTEL or NETGEAR NAS box where they have specific hardware which runs the array.  If that hardware should fail, you may find yourself in a situation where you have to find that specific box to restore access to your data.  UNRAID is portable to new hardware if your motherboard decides to die tomorrow.

What benchmarks did you use through your various changes in hardware?  I have always used IOZONE on my UNRAID since I can compare the results to this database of commercially available systems:

http://www.smallnetbuilder.com/component/option,com_nas/Itemid,190

So far, I haven't learned much

 

That, is an understatement.

 

Adding CPU horsepower past a Pentium III is WASTED effort.  It has no appreciable impact on unRAID read or write performance.  If you can't understand that NATIVELY and without having to "prove" it with bench testing, then you have, indeed, not learned much.

LOL!  You kill me.    

+1

Is there data to support this fact?

Purko and I have the Norco DS-520. This runs on a 1ghz celeron-m.

We get good transfer rates.

My 2.6ghz ABIT AB9 Pro is only slightly faster because it has PCI-E rather then PCI-X.

If I drop the 2.6 down to 1.6 I get the same transfer speed. All limited by disk I/O on parity.

Going from a 5400 RPM drive to a 7200 RPM 1.5tb parity raised my write rates by a great deal.

I suppose dropping an intel network adapter might increase performance too but at this stage of the game I'm in good enough zone.

A Newer, faster CPU or memory architecture will not affect performance as much as a faster disk subsystem.

A faster CPU/Memory come into play is when other applications are run on the system.

I.E. vmware, torrent, etc, etc.

 

...so 10GB cards would not help even if practical.

 

...SSD is obviously not a practical solution

 

Given that, about all you can do is buy a new server with gold-plated side panels and ivory buttons.

I'm curious about the impact of a cache drive on the performance of a thoroughly benchmarked unRAID setup.  From what I can gather from posts on here, the cache drive can give you a real boost in speed if you're willing to sacrifice a drive.

Perceived boost in speed.

The initial transfer is actually faster to a temporary unprotected holding place.

Then later, the data is moved to the protected array at the normal. data to protected array,  transfer rate.

noone has passed the limits of Gigabit ethernet performance (125MB/s) with UNRAID

Uhhh... the limit on CSMA/CD transport like gigabit Ethernet is only 1/4 to 1/3 of the theoretical limit, or about 40MB/s.

See this post:

  http://lime-technology.com/forum/index.php?topic=787.msg21068#msg21068

so we are stuck with rotating media for the time being.  

So you don't want "State of the art" you want "State of the cheap?"

Do the calculation yourself.  7200 rpm = 120 r/sec.  Minimum (perfect world) of 1 revolution to process each sector by unRAID.  That's at best 120 sectors per second, or 60 MB/sec.  Caching, track buffering, and read-ahead can help, but in practice, 1 rev per sector is about the best you can get.

Now what does this tell you?  If you want speed, go to 10,000 rpm drives.

But don't take my word for it... install 4GB of RAM in your desktop and in unRAID,and write RAMdisk to RAMdisk.  Copy from hard drive to/from RAMdisk in unRAID itself.  That will answer your questions.

Thanks for taking the time to provide a useful answer.

You are welcome, glad it was helpful.

 Sorry for my word choice - yes, I understand that unraid is somewhat proprietary - but I was trying to relate it to a system like an INTEL or NETGEAR NAS box where they have specific hardware which runs the array.  If that hardware should fail, you may find yourself in a situation where you have to find that specific box to restore access to your data.  UNRAID is portable to new hardware if your motherboard decides to die tomorrow.

Very true, and one of the primary reasons that I went with unRAID over all the other options out there.

What benchmarks did you use through your various changes in hardware?  I have always used IOZONE on my UNRAID since I can compare the results to this database of commercially available systems:

 

http://www.smallnetbuilder.com/component/option,com_nas/Itemid,190

I didn't use any specific benchmark software, I just observed the transfer speed using Windows Explorer.  While this may not be the most accurate method, I prefer it since those are the 'real' numbers that are going to matter in the end while I'm sitting there waiting for a transfer to finish.

If you haven't already, you may want to take a look at this section of the wiki:

Improving unRAID Performance

Specifically, these are the two changes I made that dramatically improved my server's performance:

Move Largest and Fastest Data Drives Off PCI Bus

Add a Cache Drive

You may also find this thread helpful:

Performance Differences With/Without Cache Drive

Is there data to support this fact?

 

Yup... all over the forum and in the Wiki.... 

 

I felt guilty and did additional searches (I had done so before posting).  I continue to find a lot of anecdotal answers explaining that the CPU and Memory do not contribute to performance - but there is no good data posted to back many of these up.  It may very well be there, but I'm not finding it with any of my search terms.  If there is a good (recent) thread with this information, I would appreciate it if someone can direct me to it.  

Keep in mind that the Atom is a fairly weak processor by today's standards.  Just because a system doesn't show the processor being saturated, it doesn't mean that response times from that slower processor don't contribute to slower application performance.

I actually do have a dual core atom system and another i7 system which is partially built (waiting to find a decent price on the processor).  They are destined for other purposes, but perhaps before I send them off I will load a pair of WD Raptor 300GB 10,000 RPM drives I have laying around into one system and then the other to measure transfer performance.  I was planning to wait to purchase one of the new 930 CPUs, which is not scheduled to ship until the end of February, so it may be a while before I can do this test.

I felt guilty and did additional searches (I had done so before posting).  I continue to find a lot of anecdotal answers explaining that the CPU and Memory do not contribute to performance - but there is no good data posted to back many of these up.  It may very well be there, but I'm not finding it with any of my search terms.  If there is a good (recent) thread with this information, I would appreciate it if

someone can direct me to it.

What kind of data are you looking for?

Keep in mind that the Atom is a fairly weak processor by today's standards.  Just because a system doesn't show the processor being saturated, it doesn't mean that response times from that slower processor don't contribute to slower application performance.

I've read that the atom is about equal in performance to a 900Mhz PIII.

Quite frankly, The response times I feel from the atom seem to be way snappier then that.

Maybe it's the dual core architecture, FSB and/or memory transfer speed.

It sure is a capable processor.

I think the Supermicro ITX board posted earlier is going to be quite the lil performer for an environment like unRAID.  Thanks to Aiden for posting that link.