Time for an upgrade - suggestions?

I had a drive failure the other day and it reminded me it's about time for a server upgrade.  While I'm still on 5.0.6, I plan to make heavy use of the Docker containers when 6.0 is released.  My 5 year old i3 uATX with 4GB of RAM isn't up to the task.  I've done some preliminary research, and these are what I'd like to get out of the new build:

• IPMI - I run headless without it now, and it's terrible
  
• ECC RAM - The i3 I'm using didn't support ECC at the time, so this would be nice also
  
• Onboard SATA ports - I have 4 now.  It's not enough, and it seems like going out through PCIe slows parity checks/rebuilds down substantially
  

I currently am in a Norco 4220, which has 5 backplanes with 20 SATA ports total.  I have 10 data disks, 1 parity disk, 1 warm spare, and 1 additional mounted SSD where plugin data is stored.  The array may grow by another disk or two, but probably not more than that.  I'm not particularly interested in a cache disk.  I have an AOC-SASLP-MV8 for additional SATA ports.  It's all powered by a SeaSonic X650 Gold.

So with that background, I'm looking at the following:

Motherboard: Supermicro X10SL7-F

Processor: Intel Xeon E3-1231 and stock cooler

RAM: 16GB ECC RAM (unsure what brand/speed here)

I believe the SeaSonic X650 would still be sufficient.  I like that the motherboard essentially gives me 14 SATA ports, has Intel NICs vs my current Realtek, and with the added MV8 I would be able to support any drive configuration in my case.  I think 16GB would get me enough RAM for Dockers (my current apps often push me up to 4GB, and I can't run cachedirs because it puts me over).

I don't like that the board is old-ish.  Is there something more appropriate I've missed?  I also don't like the cost and would consider going with something cheaper if it met the above requirements.

Any thoughts?  Decent selections, or back to the drawing board?

That motherboard/CPU/memory combo is an excellent choice -- the v3 CPU is a Haswell chip, so it's plenty current.    And the SuperMicro board is a superb board with ECC support and an excellent server-grade chipset.

You could shave a few bucks off by using an i3 with ECC support, but I wouldn't recommend it.

If you want to be sure about the X650 PSU, pick up a Kill-A-Watt or its equivalent and measure the current draw of your system at bootup and during parity check.  No household should be without a Kill-A-Watt anyhow, so you'll just be fulfilling your civic duty if you don't already have one. 

If you want to be sure about the X650 PSU, pick up a Kill-A-Watt or its equivalent and measure the current draw of your system at bootup and during parity check.  No household should be without a Kill-A-Watt anyhow, so you'll just be fulfilling your civic duty if you don't already have one. 

That... is a fantastic idea.  I actually have a Kill-A-Watt and hadn't considered using it for that purpose.  I'll definitely do that.

That motherboard/CPU/memory combo is an excellent choice -- the v3 CPU is a Haswell chip, so it's plenty current.    And the SuperMicro board is a superb board with ECC support and an excellent server-grade chipset.

 

You could shave a few bucks off by using an i3 with ECC support, but I wouldn't recommend it.

Great, sounds like I may have a plan, then.  I'm not totally against saving a few bucks on an i3, especially given it may be somewhat lower power/easier to cool, but then again I have sooo much money tied up in drives now, it seems crazy to save a few dimes if it has the slightest chance of causing an issue down the road.

I have read this processor has some kind of issue with the board regarding clock speed readouts ( http://lime-technology.com/forum/index.php?topic=27800.135 ), but that seems minor.

Will the AOC-SASLP-MV8 be OK in this board?  It's a PCIe x4 card, and the X10SL7-F has a x4 port, but the specs say the slot is x8 ("1x PCI-E 2.0 x4 (in x8) slot").  Will that cause an issue?  I'm not sure I even know what that means.

... Will the AOC-SASLP-MV8 be OK in this board?  It's a PCIe x4 card, and the X10SL7-F has a x4 port, but the specs say the slot is x8 ("1x PCI-E 2.0 x4 (in x8) slot").  Will that cause an issue?  I'm not sure I even know what that means.

Yes, it'll work just fine.  The motherboard has a slot that's electrically a PCIe x4 slot; but uses a physical x8 connector.    That means you plug in any card with 8 or fewer lanes ... i.e. a PCIe x1, x2, x4, or x8 card => but an x8 card would only have 4 active lanes.

using an i3 with ECC support, but I wouldn't recommend it.

Why?

using an i3 with ECC support, but I wouldn't recommend it.

 

Why?

The Haswell i3's support ECC but don't support vt-d for I/O device passthrough.

In addition, the Xeon E3-1231v3 has far more performance than any I3 ... it scores 9568 on PassMark, whereas the highest-performing Core I3-4370 scores 5570 -- and as I noted does not support vt-d.

In addition, for not much more you can boost the performance even higher -- a 1241v3 only costs about $15 more than the 1231v3 and scores 10,068 on PassMark.

The performance and feature differences certainly seem worth the extra $100 or so for the Xeon.

I have just built a very similar build as an upgrade to my main server. I am documenting my journey as well as build notes and choices here: http://lime-technology.com/forum/index.php?topic=37567.0

 

RAM: 16GB ECC RAM (unsure what brand/speed here)

 

As for memory - remember do not buy Registered DIMMs! You need Unbuffered DIMMs. Also you need to buy 4GB or 8GB DIMM's. The board will NOT accept 16GB DIMM's.

Also the QVL on the Supermicro site would have you beleive that you are limited to Samsung, Micron or Hynix. That is not true. There are other options out there BUT for this board the only rule I have found is DON'T Buy Kingston! Their modules "used to work" BUT apparently in 2013 Kingston changed the supplier of the DRAM chips without changing the model number of the DIMM's. This meant that newer DIMMs with the same model number no longer worked properly in Supermicro X10 motherboards. Kingston then removed this board from the supported memory list on their site.

Full reference see here: http://webcache.googleusercontent.com/search?q=cache:Ot37IHGhOOYJ:forums.freenas.org/index.php%3Fthreads/ram-recommendations-for-supermicro-x10-lga1150-motherboards.23291/+&cd=1&hl=en&ct=clnk&gl=au&client=safari

If I was you I would also buy 32GB instead of 16GB. It's one of my own rules when it comes to system building (which I almost didn't follow when doing this build due to cost) and that is - if you can - FILL the board. I always find when I don't - some time down the line I always wish I had. BUT this is a personal opinion.

I chose: 2 x Crucial 16GB Kit (8GBx2) DDR3/DDR3L-1600MT/s (PC3-12800) DR x8 ECC UDIMM Server Memory CT2KIT102472BD160B/CT2CP102472BD160B

Manufacturer: http://www.crucial.com/usa/en/ct2kit102472bd160b

Vendor: http://www.amazon.com/dp/B008EMA5VU/ref=pe_385040_127745480_TE_item

Apparently Crucial is Micron's (who is on the Supermicro QVL for this board) consumer brand and the above model DIMM is apparently just a rebrand of Micron's MT18KSF1G72AZ-1G6E1.

They are working flawlessly in my setup.

Thanks for the details and RAM suggestions.  I've always had good luck with Crucial/Micron parts.

I'm not sure about the 16/32GB choice.  Once you have IPMI it's so easy to add later that I wonder if it's worth the cost if I never use it.  I only just hit 4GB occasionally now, and I don't have increased usage planned.  It'll just be the additional overhead of 6.x and converting my plugin installs to Dockers.

I'm also tempted by the 1241 over the 1231.  Not sure which is the better choice there.  I suspect still the 1231 as presumably it will run a touch cooler on average, but it's tempting.

... I suspect still the 1231 as presumably it will run a touch cooler on average, but it's tempting.

They'll both run at exceptionally low power (and temp) the vast majority of the time ... essentially no difference at all.    The difference is if you NEED more "horsepower" you've got it with the 1241.

In comparing the Xeon E3-1231 v3 against the E3-1241 v3, I looked at the cost per unit of performance.  Because prices are so dynamic this comparison will be stale in a day or two.

PC Partpicker shows the 1231 available at $232: http://pcpartpicker.com/part/intel-cpu-bx80646e31231v3

And the 1241 is available at $255: http://pcpartpicker.com/part/intel-cpu-bx80646e31241v3

Based on the Passmark numbers Gary quoted above, the Passmark/$ for each is:

1231 41.24

1241 39.48

If you don't see a need for that extra little bit of performance headroom, my recommendation would be to purchase the 1231 v3.

Thanks for all the insight, guys.  I plan to order all the parts this weekend.  I'm going to go with the Crucial RAM because I've had good luck with them, and it's relatively price-competitive and known to work with a board that seems to be a little picky about RAM.  Barring a price reduction on the 1241, I'll probably just stick with the 1231, but the prices have been steady for a week or so.

Based on some threads I've seen here, I'll need to change the LSI's controller firmware from IR mode to IT mode as discussed here (  http://www.supermicro.com/support/faqs/faq.cfm?faq=16810 ).  Does that sound accurate? 

I plan on exhausting the on-board ports first and can currently serve all my drives without going off-board, only using the SASLP-MV8 if I need more ports down the road (I'll install it now, of course, it just won't have any drives attached).

However, I also wonder if there's benefit to balancing the drives between the two controllers.  For example, connect half to the on-board ports and half to the SASLP-MV8.  Any thoughts on that?

However, I also wonder if there's benefit to balancing the drives between the two controllers.  For example, connect half to the on-board ports and half to the SASLP-MV8.  Any thoughts on that?

I don't think it matters at all.    Both have ample bandwidth to handle full speed on as many drives as they support; so it shouldn't make any difference.

Also note the E3-1270v3 is identical to the E3-1241v3 in every way.  I have no clue why they renumbered it, but sometimes the 1270v3 is cheaper than the 1241v3 as it's older stock.

Also note the E3-1270v3 is identical to the E3-1241v3 in every way.  I have no clue why they renumbered it, but sometimes the 1270v3 is cheaper than the 1241v3 as it's older stock.

Certainly looks that way -- the specs on Intel's ARK site are identical for both ... the only difference is the 1241v3 was released a year later than the 1270v3.    Interestingly, their PassMark scores are NOT identical ... the 1241 scores a bit higher than the 1270 =>  10068 vs 9789.  Only a 3% difference, but interesting nonetheless.

... also note that the 1270v3 has been replaced by the 1271v3 ... this scores 10303 on PassMark.

The only difference in the specs between this and the 1270/1241 is the clock speed is 0.1MHz higher (3.6GHz base, 4.0 Turbo  vs.  3.5 base, 3.9 Turbo).

Pricing on these is interesting ... clearly it depends on your supplier; but at Newegg the prices are:

E3-1231v3  $256.99

E3-1241v3  $274.99  (i.e. only an $18 bump from the 1231)

E3-1270v3  $334.99

E3-1271v3  $337.99

At those prices I'd go with the 1241, but clearly it depends on your supplier.

I don't understand why they shuffled the numbers all round.  Makes it pretty confusing, when the 1241v3 is suddenly the same as the 1270v3, and the 1241v3 is much faster than the 1240v3.

Sort of bizarre.

Despite the specs matching, I suspect there's a difference in the actual fabs between a 1241 and 1270.

Also note the E3-1270v3 is identical to the E3-1241v3 in every way.  I have no clue why they renumbered it, but sometimes the 1270v3 is cheaper than the 1241v3 as it's older stock.

 

Certainly looks that way -- the specs on Intel's ARK site are identical for both ... the only difference is the 1241v3 was released a year later than the 1270v3.    Interestingly, their PassMark scores are NOT identical ... the 1241 scores a bit higher than the 1270 =>  10068 vs 9789.  Only a 3% difference, but interesting nonetheless.

It is unusual.  I looked around and found other posts online wondering the same thing, most coming to the same conclusion.  It seems like it really is a simmilar part.  That said, I can't find it anywhere cheaper than the 1241v3, so it might not make any difference.  Good tip, though.

However, I also wonder if there's benefit to balancing the drives between the two controllers.  For example, connect half to the on-board ports and half to the SASLP-MV8.  Any thoughts on that?

 

I don't think it matters at all.    Both have ample bandwidth to handle full speed on as many drives as they support; so it shouldn't make any difference.

Hmm.  I always thought that was not the case.  I had presumed that was the cause for my parity sync/rebuild speeds being so low (~8MB/s) until I get above 2TB when I only have 3 drives with capacities that high (compared to 10 with 2TB or less).  I guess something else is the bottleneck.

... I guess something else is the bottleneck.

Are you sure the MV8 card is plugged in to an x4 slot?    If by chance you're using a PCIe x4 card in an x1 slot, then THAT would indeed be a significant bottleneck.    Won't, of course, be an issue with your new motherboard; but you didn't say what board you have now.

Another possibility is that you have an old 2TB drive with a very low areal density ... although even the early 4-platter 500MB/platter units should still have a far better sustained data rate than what you're seeing.

My guess is you're using the MV8 in an x1 port -- but without the motherboard details it's hard to say for sure.

... I guess something else is the bottleneck.

 

Are you sure the MV8 card is plugged in to an x4 slot?    If by chance you're using a PCIe x4 card in an x1 slot, then THAT would indeed be a significant bottleneck.    Won't, of course, be an issue with your new motherboard; but you didn't say what board you have now.

 

Another possibility is that you have an old 2TB drive with a very low areal density ... although even the early 4-platter 500MB/platter units should still have a far better sustained data rate than what you're seeing.

 

My guess is you're using the MV8 in an x1 port -- but without the motherboard details it's hard to say for sure.

Thanks.  I wouldn't be surprised if it was (a) drive(s).  Some of them are from when 2TB drives first started becoming feasible (a few WD20EARS and some Hitachis), and 3 of them have more than 4 years of power on hours.  No SMART errors or warnings on any of them, solid stuff.  I'm phasing them out and have a set of 4TB drives ready to replace them all.  I'll do some research to see if there's a good way to figure out which one/ones are slow and replace them first.

The current board is one of these: http://www.newegg.com/Product/Product.aspx?Item=N82E16813130656

The MV8 is in the x16 slot.

Edit:  Turns out every one of those Hitachis are ridiculously slow.  hdparm says between 15 and 20MB/s.  Nothing else is below 90 except for flash.  So, I guess I found the bottleneck(s).  I had actually planned to replace all of my WD and Seagate 2TBs because of various minor SMART warnings and their age, but the Hitachi's have reported no issues at all so I had no plan to replace them.  I'm not sure what I'll do now.  Anyway, thanks for the suggestions that ultimately led me to look at that.

Interesting -- I looked up the platter capacities for Hitachi's 2TB drives, and they actually made some 6-platter units that only used 333GB platters !!    I'm sure those are your bottlenecks.

They still shouldn't be as slow as what you're seeing ... but they're clearly slower than even the EARS units (which have either 500GB or 667GB platters, depending on the exact model).

I wonder if there's an interface issue with your Hitachi units => I'm not a "Linux guy", but I believe you can use the following command to see what the current interface settings are for a specified drive:

hdparm -i /dev/sda

Do that with a couple of your Hitachi units and then with one that's working fine and see if the resulting display is different.    The output should let you know what mode the drive is being accessed with.

Alright.  All parts ordered.  I went with 16GB of Crucial RAM for now, and the 1231v3.  Assuming everything arrives in time, I might swap all the hardware out this upcoming weekend.  I'll likely give it a few weeks on 5.x and then upgrade to 6.x when I'm sure the new hardware is stable.

I investigated the Hitachis a bit more, as I have two Monoprice 2-SATA PCIe 1x controllers I thought were maybe acting up, but two of the Hitachis are connected to the MV8, and one directly to the motherboard.  I have other drives on both of those hardware controllers performing at the expected 80-120MBps rates.  Further, both the fully-functional WD Greens on the MV8 and the Hitachis are running UDMA5 (info attached below).  So I'm not really sure what's up with them.  The fact that all three of them are slow across two different interfaces doesn't inspire confidence, but as I mentioned, they're absolutely solid from a SMART perspective where the similarly-aged Greens are in their warning thresholds for one or more parameters.  I'm not sure if I should mass replace everything (Greens for age, Hitachis to get some speed back), or what.  I'll guess I'll see how well everything plays together with the new motherboard.

root@Hyperion:~# hdparm -i /dev/sdb

/dev/sdb:

Model=WDC WD20EARS-00MVWB0, FwRev=51.0AB51, SerialNo=WD-WCAZA5693363
Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=50
BuffType=unknown, BuffSize=unknown, MaxMultSect=16, MultSect=16
CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=3907029168
IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
PIO modes:  pio0 pio3 pio4
DMA modes:  mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5 udma6
AdvancedPM=no WriteCache=enabled
Drive conforms to: Unspecified:  ATA/ATAPI-1,2,3,4,5,6,7

* signifies the current active mode

root@Hyperion:~# hdparm -i /dev/sdc

/dev/sdc:

Model=Hitachi HDS5C3020ALA632, FwRev=ML6OA580, SerialNo=ML0220F30U5Z7D
Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=56
BuffType=DualPortCache, BuffSize=26129kB, MaxMultSect=16, MultSect=16
CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=3907029168
IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
PIO modes:  pio0 pio1 pio2 pio3 pio4
DMA modes:  mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5 udma6
AdvancedPM=yes: disabled (255) WriteCache=enabled
Drive conforms to: unknown:  ATA/ATAPI-2,3,4,5,6,7

* signifies the current active mode

I would upgrade to v6 now while on a known stable hardware platform, but that's just me.

Not sure what's going on with the Hitachi's ... electrically they're interfacing fine (UDMA 5) => but clearly they're not transferring data from the platters nearly as quickly as I'd expect.

Won't hurt to get them on your new setup and confirm that things don't change before replacing them ... but if they're still slow I'd relegate them to off-line backups and pop in some new 1TB/platter units like WD Reds, Seagate NAS, or HGST NAS drives.

As to when to upgrade to v6 => either way is fine.  I'd tend to do as you're planning ... leave the configuration untouched until you move everything to the new hardware; THEN do the upgrade.    But if you want to upgrade first, that's okay too -- just be sure that if you do that you're confident that v6 is running stable on your current setup before you move everything to the new one.