Performance: IBM M1015 and Intel RES2SV240 vs M1015x3

Just like the title states.. looking for any reviews of performance differences between the two setups....

Or at least  is the difference between them worth only using 1 PCIe slot vs 3..

 

I might have an example in a week or two once I get the problem drives out of the unRAID system I was going to test with.  It is an array of ~12 ST3000DM001 drives on a RES2SV240 currently but will attempt to put those 12 on 2 M1015s and run a second parity check that way as well.  It is an ESXi server with an unRAID VM.  I want to get a completely apples to apples comparison with RES2SV240 connected with a single cable to a single M1015 speeds on a full parity check and then again with 2 M1015s passed through for a second test.  Basically get the total seconds back on a NOCORRECT for each.

 

I've been trying to get this done for two or more weeks now for another thread but something always seems to come up.  I do have the ESXi VMs setup correctly now to make this easier just have to get time to actually run the tests in between the scheduled recordings on the Windows VM also on the server.  Thankfully I should just be able to shutdown to switch cables and start back up now so just an hour between recordings (when I'm up) now instead of hours to days to adjust the tuners around on the Windows VM so I free up a slot for the M1015 and NOCORRECT check I was struggling with before.

 

So hopefully within a week or so I can get back to you.

 

sweet

How many drives total, and what make/model?

 

Be careful of extrapolating from BobP's 12-drive test to your (say) 20-24 drive set-up. My (gut) prediction is that BobP's comparison will show no (or very little--~10%) difference in (parity-check) performance. But, with 24 drives, the performance of 3 x M1015 will be ~twice as good as one expanded M1015. Bob's 12 (fast) drive scenario is right at the "tipping point" of the M1015.

 

--UhClem

 

 

20 drives

and what make/model?

 

The drives? Basically what's on sale. But a mix between seagate and WD anywhere from 1-4tb

Here is a Correcting parity check done on my WD Red array which has a mix of 3 and 4 TB Red drives.  Drive count is 1 parity, 18 data and 1 cache - M1015 with single cable connecting the RES2SV240 to the M1015.

May 11 03:49:41 unRAIDServer kernel: md: sync done. time=49839sec (unRAID engine)

 

The following is from my N40L with a SASLP-MV8 controller and drives connected off it for a total of 9 drives all 7200 Hitachi (REAL Hitachi drives too to give you age perspective) Mix of 2, 3 & 4TB drives and 2 more drives on SASLP-MV8 that are not in array (4 on MB, 7 on SASLP-MV8 which could be some additional speed loss)  Will be replacing the SASLP-MV8 with a Dell Perc H310 soon I hope.

May 19 05:50:58 N40L kernel: md: sync done. time=43704sec (unRAID engine)

 

So about the same speeds with a much weaker controller and older drives with smaller aerial density and drive  count.

 

Still want to see what I get on my 12 drive server.  My previous timing was ~28000sec and ~30000sec's right before I started having drive problems on it (12 ST3000DM001 drives).  I want to be able to have the same drives and perform 2 NOCORRECT's as back to back as possible with controller's only and controller with expander.

 

 

Edit:  Dang unMenu log colors don't paste very well hope the edits show up right.  Ok after lots of fixes after pasting in something I'm done editing post now I think. Guess I lied but this time the edit is for content.

Here is a Correcting parity check done on my WD Red array which has a mix of 3 and 4 TB Red drives.  Drive count is 1 parity, 18 data and 1 cache - M1015 with single cable connecting the RES2SV240 to the M1015.

May 11 03:49:41 unRAIDServer kernel: md: sync done. time=49839sec (unRAID engine)

That's about what I would expect for 19 of those drives on a single M1015 (note that 20th drive, the cache, is not a factor in a parity check). Because the (single) M1015 is saturated, you are not able to utilize the full speed of the outer ~50% of those drives.

 

If there were 3 x M1015 (and no expander), I'd expect it to complete in 36-38Ksec.

 

The following is from my N40L with a SASLP-MV8 controller and drives connected off it for a total of 9 drives all 7200 Hitachi (REAL Hitachi drives too to give you age perspective) Mix of 2, 3 & 4TB drives and 2 more drives on SASLP-MV8 that are not in array (4 on MB, 7 on SASLP-MV8 which could be some additional speed loss)  Will be replacing the SASLP-MV8 with a Dell Perc H310 soon I hope.

May 19 05:50:58 N40L kernel: md: sync done. time=43704sec (unRAID engine)

With that mix of drives, that's about as good as you can expect (regardless of MB and controller). You might get a slightly faster completion if you had 5 array drives on the N40L itself (and only 4 on the MV8), but logistics might preclude that. If you ever add a 6th array drive (of that same ilk) to the MV8, expect your time to increase 15-20%. With the current layout, I don't think a controller upgrade will buy you much performance (<5-10%) [pls let me know if I'm wrong].

 

So about the same speeds with a much weaker controller and older drives with smaller aerial density and drive  count.

Actually about 15% faster completion for the "weakling" ... which highlights the effect of "overburdening" a controller.

 

Still want to see what I get on my 12 drive server.  My previous timing was ~28000sec and ~30000sec's right before I started having drive problems on it (12 ST3000DM001 drives).

I presume that is with a single M1015 + expander. Because, with 2 x M1015, I'd expect 20-22Ksec. Those drives are capable of 160-195 MB/sec across the outer ~30-40%, but they are being limited to a max of ~150-160 (with 12 of them on a single M1015).

 

With that mix of drives, that's about as good as you can expect (regardless of MB and controller). You might get a slightly faster completion if you had 5 array drives on the N40L itself (and only 4 on the MV8), but logistics might preclude that. If you ever add a 6th array drive (of that same ilk) to the MV8, expect your time to increase 15-20%. With the current layout, I don't think a controller upgrade will buy you much performance (<5-10%) [pls let me know if I'm wrong].

Will do.  Should be getting the controller later this week then have to flash it to IT mode and put it in.  I have TWO more drives on the MV8 not in array but those 4TB drives were empty and were removed temporarily.  They will replace 2 3TB drives which I might remove.  The fact that the 2TB drives are genuine Hitachi drives can't be helping either with their lower density and potentially slower firmware.  But they seem to last forever so won't replace until they die.

I presume that is with a single M1015 + expander. Because, with 2 x M1015, I'd expect 20-22Ksec. Those drives are capable of 160-195 MB/sec across the outer ~30-40%, but they are being limited to a max of ~150-160 (with 12 of them on a single M1015).

Correct.  A single cable from the M1015 connected to the RES2SV240.

 

Supposedly if I only expanded the RES2SV240 to 16 drives and used two cables I would get better speeds.  But I've also seen posts saying the dual connection was really meant for two M1015 cards to be connected to one RES2SV240 so that an array would still function if one card died.  If that would increase the speed significantly that would be another way for me to get to 24 drives with only one additional card (8 directly connected to M1015s and 16 off sas expander).

Supposedly if I only expanded the RES2SV240 to 16 drives and used two cables I would get better speeds.  But I've also seen posts saying the dual connection was really meant for two M1015 cards to be connected to one RES2SV240 so that an array would still function if one card died.  If that would increase the speed significantly that would be another way for me to get to 24 drives with only one additional card (8 directly connected to M1015s and 16 off sas expander).

I just did a little reading on the RES2SV240 [http://www.intel.com/support/motherboards/server/res2sv240/sb/CS-033969.htm], and connecting it to a (single) controller with 2 cables will potentially allow for increased bandwidth IF the controller supports it. Intel says their controllers do; I don't know about the M1015, but it probably does. Of course, actually realizing that increased bandwidth is totally dependent on the controller having the oomph! to exploit it. I suspect that the M1015 is maxed out on a single cable [4x6Gbps] (but I hope I'm wrong!).

 

Maybe you could test the two-cable theory now, with just the existing 12 drives. There is no need to do a full parity check--just watch the speed report for the first five minutes, and note what it "averages" (it should be pretty steady) [then terminate it]. Then connect the 2nd cable and test again. I expect the current (single cable) setup will be 150-160 MB/s. If the 2-cable setup is "successful", the test should be 175-190.

 

Maybe you could test the two-cable theory now, with just the existing 12 drives. There is no need to do a full parity check--just watch the speed report for the first five minutes, and note what it "averages" (it should be pretty steady) [then terminate it]. Then connect the 2nd cable and test again. I expect the current (single cable) setup will be 150-160 MB/s. If the 2-cable setup is "successful", the test should be 175-190.

If I have a recording window available on the Windows VM I will try it tonight.  Because I will have to rearrange cables and I'm not confident I can do that with the ESXi server running I want to be able to shut down ESXi first.  Otherwise it may have to wait for another night or the weekend when I should have time during the day.  The summer lineup left a day during the week with nothing to record but it was probably yesterday.

Ok I have the results of the tests below.  But first I need to explain something.  I have several SAS expanders.  One of them I thought I was having problems with.  I have now found said problem expander the one on my ST3000DM001 based ESXi server.  It functions fine as long as I don't try to use all lanes from it.  I have one lane on one of the ports on the expander that doesn't function so my expander really is only 23 lanes not 24 like it is suppose to be.  This testing has now allowed me to confirm what I've thought since I bought it but never confirmed.  Also I apparently had faulty memory of the drive count on the server it was 14 array drives, 1 parity and 1 cache drive = 16 total drives.

 

 

Anyway I have tests for 3 conditions.  The number of bytes run through check is at the top of each code block below.

 

 

1) 1 M1015 with one cable to SAS Expander with 16 drives run NOCORRECT for 22 minutes (I.E. 16 drives off of 4 lanes of M1015):

Expander 1 cable 1 M1015
129.35GB


May 20 18:17:47 Media1 kernel: mdcmd (89): check NOCORRECT
May 20 18:17:47 Media1 kernel: md: recovery thread woken up ...
May 20 18:17:47 Media1 kernel: md: recovery thread checking parity...
May 20 18:17:48 Media1 kernel: md: using 3072k window, over a total of 2930266532 blocks.
May 20 18:39:48 Media1 kernel: mdcmd (90): nocheck 
May 20 18:39:48 Media1 kernel: md: md_do_sync: got signal, exit...

 

 

2) 2 M1015s with NO SAS Expander and 16 drives run NOCORRECT for 22 minutes:

2 M1015s
219.66GB


May 20 19:54:10 Media1 kernel: mdcmd (61): check NOCORRECT
May 20 19:54:10 Media1 kernel: md: recovery thread woken up ...
May 20 19:54:10 Media1 kernel: md: recovery thread checking parity...
May 20 19:54:10 Media1 kernel: md: using 3072k window, over a total of 2930266532 blocks.
May 20 20:16:19 Media1 kernel: mdcmd (62): nocheck 
May 20 20:16:19 Media1 kernel: md: md_do_sync: got signal, exit...

 

 

3) 1 M1015 with two cables to SAS Expander with 16 drives run NOCORRECT for 21 minutes:

THIS TEST IS SUSPECT because the only way I could run it was to connect one of the cables from the M1015 to the SAS Expander port with the bad lane and the other cable to a good port.  You can see that errors were produced in the unRAID log entries below.  I will rerun the test this weekend when I can swap the bad expander for one of my other ones that works.

Expander 2 cables one M1015
68.59GB


May 20 21:33:42 Media1 kernel: mdcmd (59): check NOCORRECT
May 20 21:33:42 Media1 kernel: md: recovery thread woken up ...
May 20 21:33:42 Media1 kernel: md: recovery thread checking parity...
May 20 21:33:42 Media1 kernel: md: using 3072k window, over a total of 2930266532 blocks.
May 20 21:33:57 Media1 kernel: mpt2sas0: log_info(0x31110d01): originator(PL), code(0x11), sub_code(0x0d01)
May 20 21:34:31 Media1 last message repeated 2 times
May 20 21:36:31 Media1 kernel: mpt2sas0: log_info(0x31110d01): originator(PL), code(0x11), sub_code(0x0d01)
May 20 21:38:51 Media1 last message repeated 2 times
May 20 21:40:01 Media1 last message repeated 2 times
May 20 21:41:32 Media1 kernel: mpt2sas0: log_info(0x31110d01): originator(PL), code(0x11), sub_code(0x0d01)
May 20 21:44:37 Media1 kernel: mpt2sas0: log_info(0x31110d01): originator(PL), code(0x11), sub_code(0x0d01)
May 20 21:47:55 Media1 kernel: mpt2sas0: log_info(0x31110d01): originator(PL), code(0x11), sub_code(0x0d01)
May 20 21:49:12 Media1 last message repeated 2 times
May 20 21:51:49 Media1 last message repeated 2 times
May 20 21:53:57 Media1 last message repeated 3 times
May 20 21:54:02 Media1 kernel: mpt2sas0: log_info(0x31110d01): originator(PL), code(0x11), sub_code(0x0d01)
May 20 21:54:19 Media1 kernel: mdcmd (60): nocheck 
May 20 21:54:19 Media1 kernel: md: md_do_sync: got signal, exit...

 

 

But by looking at the first two you can see exactly what you expected with two M1015s to 16 drives there is a very significant boost in speed on the parity check over using a SAS Expander and one M1015 with a single cable connecting them - can't say with 2 cables yet due to bad lane on expander.  I won't change my mind about using them because it is fast enough for me but it might convince someone else to use multiple controllers.

...

But by looking at the first two you can see exactly what you expected with two M1015s to 16 drives there is a very significant boost in speed on the parity check over using a SAS Expander and one M1015 with a single cable connecting them - can't say with 2 cables yet due to bad lane on expander.  I won't change my mind about using them because it is fast enough for me but it might convince someone else to use multiple controllers.

Nice. Good test data leads to good information ... which allows for informed decisions.

 

 

BobP,

 

Upon looking a little closer at the numbers for your recent Test#1 [M1015<=single cable=>expander w/15 array drives], I think they're a little low. With 12 drives, I had been expecting 150-160--with the (revised) 15 drives, I would reduce that to 120-130.

 

However, that test yielded a speed of only 100 MB/sec. [it smells like there might be a PCIe misconfig.]

 

If you have a chance to perform that same test on your 19-drive WD Red array, pls do so. That will help determine what the max bandwidth for a M1015<=>RES2SV240 set-up should be. If it doesn't jibe with the above Test#1, we can investigate the PCIe further. With 19 array drives, I expect ~100 MB/s; but if it's actually ~80 (which approximates Test#1's 15-drive @ 100), then I'm "wrong", and the "slowdown" is an inherent factor of using an expander.

 

BobP,

 

Upon looking a little closer at the numbers for your recent Test#1 [M1015<=single cable=>expander w/15 array drives], I think they're a little low. With 12 drives, I had been expecting 150-160--with the (revised) 15 drives, I would reduce that to 120-130.

 

However, that test yielded a speed of only 100 MB/sec. [it smells like there might be a PCIe misconfig.]

 

If you have a chance to perform that same test on your 19-drive WD Red array, pls do so. That will help determine what the max bandwidth for a M1015<=>RES2SV240 set-up should be. If it doesn't jibe with the above Test#1, we can investigate the PCIe further. With 19 array drives, I expect ~100 MB/s; but if it's actually ~80 (which approximates Test#1's 15-drive @ 100), then I'm "wrong", and the "slowdown" is an inherent factor of using an expander.

Will do.  Tomorrow night however as I'm moving files around on it currently (3TB - 1.5TB from multiple drives to cache then that 1.5 back to single array drive) tonight and tomorrow while I'm at work.

 

 

Edit: Sorry was busy last night will try tonight.

...maybe slightly off-topic but...

 

FWIW, I see a constant 113MB/sec for read or write over the NIC with my setup (16 Disks, 3TB segate+hitachi 7.2k) on a ZFS (on Linux) based server, including FullDiskEncryption with LUKS/dmcrypt)

The box uses an ASUS AM3+ board with Opteron CPU, 8GB RAM and 1xM1015 in IT and a RESCV240 (one cable connected).

 

This is a raidz3 array (triple parity) and an internal zfs scub reports a rate of >630MB/sec.

 

IMHO with unRAID you won't get any better in real life scenarios with a single NIC.

Edit: ...or with a single parity disk...so does this questioned variation of the setup *really* matter?..I think not.

...maybe slightly off-topic but...

 

FWIW, I see a constant 113MB/sec for read or write over the NIC with my setup (16 Disks, 3TB segate+hitachi 7.2k) on a ZFS (on Linux) based server, including FullDiskEncryption with LUKS/dmcrypt)

The box uses an ASUS AM3+ board with Opteron CPU, 8GB RAM and 1xM1015 in IT and a RESCV240 (one cable connected).

 

This is a raidz3 array (triple parity) and an internal zfs scub reports a rate of >630MB/sec.

 

IMHO with unRAID you won't get any better in real life scenarios with a single NIC.

Edit: ...or with a single parity disk...so does this questioned variation of the setup *really* matter?..I think not.

 

Ford Prefect, by FDE you mean your raidz3 pool is encrypted or you have an encrypted root pool ?

 

how much usable space are you getting out of those 48TB in raidz3 ?

Ford Prefect, by FDE you mean your raidz3 pool is encrypted or you have an encrypted root pool ?

 

how much usable space are you getting out of those 48TB in raidz3 ?

 

Yes my data pool is encrypted.

As my OS is Linux, I used LUKS/dmcrypt (which supports the AES-NI instruction set of my CPU) to encrypt the disks, then setting up the storage pool with ZFS-on-Linux on that.

The performance is simply awesome (with only 8GB of RAM)...I tried FreeNAS (with geli) and native Solaris but these cannot compete.

 

As for the usable pool size...I am currently traveling and don't have access to my server but I *think* it is something around 34TB+.

Edit: calculate it yourself with this tool: http://www.servethehome.com/raid-calculator/

Edit2: ZFS normaly reserves some space as "headroom" and recommendations are that you should not fill a pool above 80% or so....don't know what my pool-reservation size is, I kept the defaults.

BobP,

 

Upon looking a little closer at the numbers for your recent Test#1 [M1015<=single cable=>expander w/15 array drives], I think they're a little low. With 12 drives, I had been expecting 150-160--with the (revised) 15 drives, I would reduce that to 120-130.

 

However, that test yielded a speed of only 100 MB/sec. [it smells like there might be a PCIe misconfig.]

 

If you have a chance to perform that same test on your 19-drive WD Red array, pls do so. That will help determine what the max bandwidth for a M1015<=>RES2SV240 set-up should be. If it doesn't jibe with the above Test#1, we can investigate the PCIe further. With 19 array drives, I expect ~100 MB/s; but if it's actually ~80 (which approximates Test#1's 15-drive @ 100), then I'm "wrong", and the "slowdown" is an inherent factor of using an expander.

 

 

OK here are the figures for my 19 drive WD Red array (1 parity, 18 array, 1 cache) 3 & 4 TB drives:

111.25 GB


May 23 18:11:08 unRAIDServer kernel: mdcmd (73): check NOCORRECT
May 23 18:11:08 unRAIDServer kernel: md: recovery thread woken up ...
May 23 18:11:08 unRAIDServer kernel: md: recovery thread checking parity...
May 23 18:11:08 unRAIDServer kernel: md: using 1536k window, over a total of 3907018532 blocks.
May 23 18:33:37 unRAIDServer kernel: mdcmd (74): nocheck 
May 23 18:33:37 unRAIDServer kernel: md: md_do_sync: got signal, exit...

 

 

 

 

 

The following is from my N40L with a SASLP-MV8 controller and drives connected off it for a total of 9 drives all 7200 Hitachi (REAL Hitachi drives too to give you age perspective) Mix of 2, 3 & 4TB drives and 2 more drives on SASLP-MV8 that are not in array (4 on MB, 7 on SASLP-MV8 which could be some additional speed loss)  Will be replacing the SASLP-MV8 with a Dell Perc H310 soon I hope.

May 19 05:50:58 N40L kernel: md: sync done. time=43704sec (unRAID engine)

With that mix of drives, that's about as good as you can expect (regardless of MB and controller). You might get a slightly faster completion if you had 5 array drives on the N40L itself (and only 4 on the MV8), but logistics might preclude that. If you ever add a 6th array drive (of that same ilk) to the MV8, expect your time to increase 15-20%. With the current layout, I don't think a controller upgrade will buy you much performance (<5-10%) [pls let me know if I'm wrong].

Well it did get a little better with the Perc H310:

May 23 18:52:48 N40L kernel: md: sync done. time=41612sec

So a little faster 5% by my figuring or 2092 seconds so not really much difference

OK here are the figures for my 19 drive WD Red array (1 parity, 18 array, 1 cache) 3 & 4 TB drives:

111.25 GB


May 23 18:11:08 unRAIDServer kernel: mdcmd (73): check NOCORRECT
May 23 18:11:08 unRAIDServer kernel: md: recovery thread woken up ...
May 23 18:11:08 unRAIDServer kernel: md: recovery thread checking parity...
May 23 18:11:08 unRAIDServer kernel: md: using 1536k window, over a total of 3907018532 blocks.
May 23 18:33:37 unRAIDServer kernel: mdcmd (74): nocheck 
May 23 18:33:37 unRAIDServer kernel: md: md_do_sync: got signal, exit...

Interesting ... 111.25GB in 22:30 = ~85 MB/sec

 

Since that is well below the speed of those drives [in the outer zones (130-150 MB/sec)], that means the test result was constrained by the controller/expander combination. The result shows a max throughput of ~1600 MB/sec (19 x 85). Now, since the M1015 itself  has a maximum sustained throughput of 2000+ MB/sec, the expander is imposing a fairly significant inefficiency. Either that or Intel's claim of a single cable connection getting 4 x 6Gbps [theoretical] throughput--ie, 2000-2200 MB/s [realistic]--is optimistic/exaggerated. Maybe that test with 1xM1015<=>2xcables<=>RES2sv240<=>16x(fast)drives will pinpoint the source of the degradation.

 

[Note that seeing 2000+ MB/s from a ("stand-alone") M1015 requires using 4-8 SSDs, since the fastest mechanical drives, today, max out at ~200 ea.]

 

Ok here are times for the WHOLE 3TB on my 16 drive ST3000DM001 array:

 

 

Time with 16 drives on expander and one cable attached from M1015 to RES2SV240:

May 23 18:43:02 Media1 kernel: mdcmd (61): check NOCORRECT
May 23 18:43:02 Media1 kernel: md: recovery thread woken up ...
May 23 18:43:02 Media1 kernel: md: recovery thread checking parity...
May 23 18:43:02 Media1 kernel: md: using 3072k window, over a total of 2930266532 blocks.
May 24 03:01:01 Media1 kernel: md: sync done. time=29878sec

 

 

Time with 16 drives on 2 M1015s:

May 24 12:18:26 Media1 kernel: mdcmd (59): check NOCORRECT
May 24 12:18:26 Media1 kernel: md: recovery thread woken up ...
May 24 12:18:26 Media1 kernel: md: recovery thread checking parity...
May 24 12:18:26 Media1 kernel: md: using 3072k window, over a total of 2930266532 blocks.
May 24 18:28:49 Media1 kernel: md: sync done. time=22223sec

 

 

So 25% slower parity checks with an expander.