Any hardware testing done for performance?

I have scanned lots of posts about performance, done my share of searching, but am still left wondering about this topic ...

 

Which components are most critical to performance and at what point is additional component speed/size wasted?  For example, has anyone built a server running Unraid and swapped out components to see the impact?  Iozone or similar results would be useful.

 

I have seen the PATA vs. SATA numbers and 100Mbit vs. Gigabit in TheMaster's most excellent "Ultimate" post, but I am looking for others:

 

* 512MB vs. 1GB vs. 2GB

* Single processor core vs. multi

* AMD vs. Intel (tough to swap, obviously)

* CPU speeds

* Different mobos (also tough to swap)

* Brand/speed of hard drive (5400 vs. 7200, cache sizes, etc.)

* ...

 

Given that parity calculations can be somewhat intensive and I will be doing multiple read streams (potentially while writing), I am wondering if I should be spending more on a multicore CPU to help out the write speeds, but I don't want to waste money on a Core Duo if a $50 Celeron will do the trick.

 

 

Bill

I have scanned lots of posts about performance, done my share of searching, but am still left wondering about this topic ...

 

Which components are most critical to performance and at what point is additional component speed/size wasted?  For example, has anyone built a server running Unraid and swapped out components to see the impact?  Iozone or similar results would be useful.

 

I have seen the PATA vs. SATA numbers and 100Mbit vs. Gigabit in TheMaster's most excellent "Ultimate" post, but I am looking for others:

 

* 512MB vs. 1GB vs. 2GB

* Single processor core vs. multi

* AMD vs. Intel (tough to swap, obviously)

* CPU speeds

* Different mobos (also tough to swap)

* Brand/speed of hard drive (5400 vs. 7200, cache sizes, etc.)

* ...

 

Given that parity calculations can be somewhat intensive and I will be doing multiple read streams (potentially while writing), I am wondering if I should be spending more on a multicore CPU to help out the write speeds, but I don't want to waste money on a Core Duo if a $50 Celeron will do the trick.

 

 

Bill

I'm pretty sure the bottleneck for most will be the PCI buss used for the disk controllers.  Only way to make that faster is to get the disk IO on its own faster buss.

 

I'm using the originally specified Intel motherboard and all PATA drives.  The motherboard uses the $50 Celeron you described...  Recently I did an experiment where I simulated a failure of one of my 500 Gig drives.  I then replaced it in the array and started the unRaid array in rebuilding it.

 

To rebuild, it had to read the other 7 drives in my array, calculate the contents of the disk being reconstructed, and write the blocks to the disk being rebuilt.  So... 7 drives being read sequentially, and one being written to sequentially and it was doing it at a rate somewhere near 18K Mb/Sec.

 

Now... this did not involve any network traffic, so I decided to try to work it a bit harder.  I used one of the PCs on my LAN to play an ISO image of a DVD that was stored on the drive being re-built.  Now, this image had to also be re-constructed from by reading the 7 working drives in my array and calculating the original contents of the (simulated) failed disk.  OK, now I'm reading 7 disks and writing the results to the network card.... and to my PC to play the DVD image.... at the same time I'm reading the same 7 disks and re-building the failed disk.

 

I did this 3 more times, each with a different ISO image from the failed disk.  Now I'm playing 4 different DVD images, to 4 different PCs and media players on my LAN, and rebuilding a failed drive at the same time.  The re-build had slowed to 10K MB/Sec, the DVD images all played perfectly... apparently reads of the drives took priority over the writes to rebuild the failed drive.

 

The UNIX "top" command showed about 16% of the time the OS was waiting on I/O.  About 70% of the time, the CPU was idle, and about 12% of the time it was actually doing something. (Calculating the missing blocks of data based on the read of data from the other 7 disks)

 

So... with the CPU 70% idle I would think a faster/More expensive CPU would be wasted.    It might make the 12% of time dong parity calcs go faster, but the biggest improvement might be if we could implove the 16% of the time we are waiting on I/O.

 

To improve the time waiting on I/O we need a faster buss to the disk controllers, and a faster buss to the network card.  PCIe based disk controllers might be one answer... SATA disks might be another. A read-ahead buffer in the disk itself would help.  A motherboard that keeps network traffic off of the same PCI buss as the disk-IO would help.

 

There is plenty to experiment with.  I would look at PCIe bus SATA disk controllers for max IO speed first and a motherboard with a faster CPU last.

 

Joe L.

Good info.  I am trying to bridge the gap between the limits I know of:

 

* Gigabit ethernet: ~125MB/sec

* PCI: ~125MB/sec

* A SATA drive's sustained read speed: 150MB-200MB/sec

* SATA interface: 150MB-300MB/sec

 

... and the Unraid's measured speed with SATA drives (25-30MB/sec)

 

The test you performed bumped up against several of the limits above, because you had seven drives pumping data, but what about when fewer drives are involved, as in with two or three videos are being streamed?  Heck, in a typical iozone read test, only one drive is involved.

 

What am I missing?  Perhaps I am underestimating the traffic over PCI?

 

 

Bill

Good info.  I am trying to bridge the gap between the limits I know of:

 

* Gigabit ethernet: ~125MB/sec

* PCI: ~125MB/sec

* A SATA drive's sustained read speed: 150MB-200MB/sec

* SATA interface: 150MB-300MB/sec

 

... and the Unraid's measured speed with SATA drives (25-30MB/sec)

 

The test you performed bumped up against several of the limits above, because you had seven drives pumping data, but what about when fewer drives are involved, as in with two or three videos are being streamed?  Heck, in a typical iozone read test, only one drive is involved.

 

What am I missing?  Perhaps I am underestimating the traffic over PCI?

 

 

Bill

I know I was probably bumping up against the PCI buss limits. The point I was trying to make was that a faster CPU would not be likely to change performance much.  Odds are it would just be idle for a higher percentage of the time.  The PCI buss is involved twice.... If you can get the network traffic or the disk IO traffic on a different bus, performance will be improved.

 

My test was artificial...to test the performance of the server. In my house there is usually only one movie being served at a given time.  We would never have 4 ISO images being simultaneously served, as there are only two of us.    I'll bet the disks were seeking like mad in my test.  I was very impressed they could keep up with the demands I placed on them.

 

99.9% of the time, when my unRaid server is active, only 1 ISO image is being read from the server.  The server is written to only when I'm adding media, or doing a backup of another PC on my LAN, and again I almost never would be serving up a movie to view at the same time. 

 

I'm really glad you are looking to find the best hardware.    For most unRaid users, the only time the current bottleneck is noticed is when they are initially migrating their media collections to the unRaid server over their LAN.  Copying hundreds of gigabytes over the LAN will never be fast.  This page describes how motherboards with a faster PCI-X buss will eliminate the PCI buss as an issue.  It is currently available, but on motherboards designed for high-end servers, usually with multiple CPUs... perhaps you can find an affordable PCI-X based motherboard and use it with only 1 CPU.  You will make many very happy if you can find currently available/affordable hardware that performs well under unRaid.

 

Joe L.

 

Adding to my own post ... folks should check out this article:

 

http://smallnetbuilder.com/content/view/29936/79/

 

Several factors were considered, including drive rotational speed, drive cache, SATA level, and NAS memory.  The one that seems to be "the biggie" is memory.  While not tested in an Unraid configuration (they used the Thecus 5200 and a Dlink box), the learnings may be applicable to all NAS solutions.

 

On a related note: Infrant claims that moving from 256MB to 1GB results in a ~12% performance boost.

 

Comments encouraged.

 

To restate my objective/goal: If I am to spend more than the minimum for my hardware, where (if anywhere) should it be spent to maximize the performance benefit? 

 

 

Bill

 

 

What a strange test.

The objective was to test, how difference in harddrive specs would result performance in NAS'es.

But most of the artichle is spent on comparing the NAS'es and speculate on the effects of cache, and really gets funny when Firewire and Esata discs got into the test.

/Rene

 

 

 

It was a bit unfocused, but not as bad as you stated.  He starts testing drive performance and actually completes that, but also spiders into other areas.

 

I would not solely rely on that one article, but as we collect other similar studies, a picture will develop that is accurate and focused.

 

 

Bill

The reviewer's talk of 'cache' is simply the RAM in the faster NAS. He also notes that a faster processor* (when file sizes exceed 'cache'/RAM) helps as well.

 

So, I would suggest the following to boost unRAID performance:

 

1. PCIe bus (Not from the NAS article, but from discussions here on the forums.)

2. SATA drives (Ditto)

3. RAM (This from the NAS article)

4. Fast processor (Ditto*)

 

Note that you may never reach the theoretical throughput of Gig Ethernet due to performance issues of other components, but the PCIe bus will help that. Hard drive performance could still be a bottleneck, but the review suggests that 8 MB cache in a 7200 RPM SATA drive is the bang-for-the-buck mark, as 16 MB cache or 10,000 RPM Raptors aren't worth the price - the performance boost isn't high enough.

 

If you want to put extra money into your hardware for performance, here are my recommendations:

 

1. Get a MB with a PCIe bus. Make sure any onboard Gig Ethernet or SATA controller is on the PCIe bus, or get PCIe cards as necessary

2. Use SATA drives (obviously)

3. Pump up the RAM according to your budget

4. I think a current Celeron (-D or -M) at 2+ GHz  would be sufficient processor*, based on the forums here. If you really want something better, the dual core Pentium 805D is running about $65 at Newegg.

 

*The faster processor in the NAS article was a 600 MHz Celeron, and the processor speed was only relevant when file size exceeded 'cache'/RAM size. If you put 4 GB RAM (the reviewer's 'dream' amount for a NAS) in your unRAID box, processor speed will likely only become significant if you are dealing with files larger than 4 GB. And with today's Celerons, you are well beyond the scale of what the reviewer experienced.

Peregrine,

 

Thanks for the post.

 

I am thinking along similar lines.  I may actually start with something like 1GB of RAM then beef it up temporarily with memory from other machines to see if it makes much of a difference in the real world.  For the CPU, I am thinking of a low-end multicore (like the 805D you recommend) just because the prices are so reasonable.

 

I may get some IDE/PATA only if the prices are fantastic and I would use those particular drives for data where QOS is less critical (like music or backups).

 

How do I find out if a mobo's built-in SATA is sitting on PCI or PCIe?  I peeked around at various examples and they never stated what I was looking for.

 

 

Bill

 

How do I find out if a mobo's built-in SATA is sitting on PCI or PCIe?  I peeked around at various examples and they never stated what I was looking for.

 

Bill

 

Hard question; I hadn't looked before I posted that suggestion.

 

I would check the manufacturer's support site for documentation. That might show MB layout and what's connected to what.

I may actually start with something like 1GB of RAM then beef it up temporarily with memory from other machines to see if it makes much of a difference in the real world.  For the CPU, I am thinking of a low-end multicore (like the 805D you recommend) just because the prices are so reasonable.

 

Note Tom's post quoted below (Jan 2007) that indicates 1GB is maximum supported by unRAID.  Dual channel mode certainly makes sense to get the most out of the processor.

However, a simple test as you describe would certainly be interesting (in case something has changed with V4.0beta / 2.6 kernel).

 

unRAID requires minimum 512MB, maximum 1GB (it won't recognize any more than 1GB).  Definitely use 2 identical sticks if possible in order to run in dual-channel mode.

 

I may get some IDE/PATA only if the prices are fantastic and I would use those particular drives for data where QOS is less critical (like music or backups).

 

This would seem to defeat the goal of seeking best performance. Remember that even if you have a SATA parity and some SATA data disks, any PATA drives in the array will still not provide best performance when parity builds / checks are required. (eg. when upgrading a drive etc.)

 

How do I find out if a mobo's built-in SATA is sitting on PCI or PCIe?  I peeked around at various examples and they never stated what I was looking for.

 

Not entirely sure on this myself.  Note that PCIe connected GigaLAN is also important due to the PCI GigaLAN bottleneck.  All I can report is that I am very happy with the performance of my recent 100% SATA300 Asus P5B-E unRAID build.  This uses PCIe GigaLAN, and Intel southbridge ICH8 onboard 6x SATA300 ports in AHCI mode (instead of IDE mode, for full Linux driver support of NCQ etc.).

 

Another link in the chain to discuss: Gige switches/routers.  I have been doing a lot of reading about performance testing and they are all over the map.  Performance seems to be somewhere between 4X 100Mbit and 0.9X 100Mbit - that's right, SLOWER.  If you have one of those, it doesn't really matter how fast your NAS is, the straw is just too small.

 

I am running 100Mbit right now but am looking to upgrade it.  So add another item to performance tweak ...

 

 

Bill

That's right - the #1 bottleneck is GigE performance.  We've spent quite a bit of time in the past trying to squeeze more performance out of GigE - including tweaks to the network driver stacks on both Windows side & linux side, jumbo frames of various sizes, etc.  Nothing ever produced anything close to theoretical GigE performance.

But for things like parity checks the network isn't involved. In this case I believe PCI is the #1 bottleneck. I read that in practice you only get 50% of the theoretical limit. I think that this is why people with PCIe are seeing so much better performance. Now, whether PCIe removes the bus as the #1 bottleneck I don't know. Perhaps some people with PCIe can do a few memory/CPU tests for us?