ESXi/unRAID server

I've been lurking for over a year now and the time has finally come to start my build.

I started out wanting a simple unRAID machine but after reading Johnm's ATLAS thread (thank you), of course I wanted to virtualise. I don't really know if I need to, but I really want to.

It was quite some time ago that I did my research and I'm a little rusty but I'll go through the ATLAS thread again, at least, before I purchase/build to revise my knowledge. Any advice/suggestions though, would be greatly appreciated.

Hardware:

CPU: Intel Xeon E3-1230V2 - Are there better, reasonably priced, options for transcoding?

                  Intel Xeon E3-1240V2

Motherboard: Supermicro X9SCM-IIF-O

Storage Adapters: 2 x M1015's - I already have these and flashed them as required ages ago. Not sure which version of firmware I used. I can't quite remember how many HDDs I'll be able to run off these but I know it's more than I'll require. Might get a third M1015 or a RES2SV240 SAS Expander in future if required.

RAM:  2 x Kingston 8GB KVR13E9/8i (1333MHz) DDR3, ECC CL9

Case: Antec 1200 - no room for a Norco

                        Norco RPC-4224 Rackmount Server Chassis - http://www.techbuy.com.au/p/149123/CASINGS_SERVER_-_4U/Norco/RPC-4224.asp

                        w/ Norco 120mm Fan Wall Bracket - http://www.techbuy.com.au/p/152644/CASINGS_SERVER_-_ACCESSORIES/Norco/120mmFANBRACKET.asp

unRAID HDDs: Western Digital Red 3TB  EFRX - as many as required for my media. I have a few of 2TB Western Digital green drives in my HTPC, and some external drives, that stores it all at the moment so I'll have to migrate the data from there to unRAID and then I can add the WD's too.

Cache Drive: Western Digital Red 3TB  EFRX - not sure if this is a good choice but will research a bit more before purchase

                                    SAMSUNG 830 2.5 inch 256GB SATA III SSD

Parity Drive: 4TB Deskstar 3.5" SATA III Internal Hard Drive - Sale: http://www.ozbargain.com.au/node/88698

ESXi Datastore: SAMSUNG 830 2.5 inch 128GB SATA III SSD

                                    SAMSUNG 830 2.5 inch 512GB SATA III SSD

PSU:  Seasonic X-760 80plus Gold 760W

USBs: 2 x Sony Microvault 4GB - I have these already, the lime green ones of course.

Cables:  3Ware 1M mini-SAS(SFF 8087) to 4 x SATA, Discrete Cable (forward breakout)

                        4 x 1m 30AWG Internal Mini SAS 36pin (SFF-8087) Male to Mini SAS 36pin (SFF-8087) Male Cable - Black

                        2 x Norco 0.5M Mini-SAS(SFF-8087) to 4xSATA Discrete Cable

                        2 x BitFenix Sleeved 4-Pin Molex Power Splitter Black - http://www.pccasegear.com/index.php?main_page=product_info&cPath=19_197&products_id=17456

Heat Sink: Noctua NH-U9B

Case Fans: 3 x Noctua NF-P12-1300 - 120mm 1300RPM Pressure Optimised Fan

                2 x Noctua NF-R8 PWM - 80x80x25mm

UPS: Any suggestions?

Blu Ray Drive: I'll need a USB drive to pass through I think

If there are any suggestions or corrections, please let me know.

Thanks

i know its been a week or so since your post but im not sure if its worth the WD red's.

They are significantly more expensive compared to others (Green, Seagates, etc) and IMO don't offer a great deal of advantages for this sort of setup. They are definately what you'd want to put in your PVR if you replace its HDD, etc.

Not sure other people's thoughts on that.

for the cache/parity drive. The parity drive needs to be your biggest drive, if you can stretch, maybe got for a 4TB, that way when you want to put in other 4TBs, you won't need to upgrade your parity drive first. It might be $50-100 extra now, but if you do it in 6 months you will need to buy a whole new drive.

The cache drive doesn't need to be as big as your others, i have a 300GB Velociraptor as mine and have never had any issues.

As for Fans, i can't go past Noctua personally, they are definately more expensive, but are awesome fans...i also don't mind the colour

I am not 100% sure if anyone has built an ESXi / unRAID box on a X9SCM-IIF-O with IVYBridge CPU yet. on paper it should work. but, in the real world, until it has been tested, you never know.

if you have not yet purchased the parts, do a little research with google and make sure there are no hidden problems like using multiple SAS cards at once.

pantner:

I think you may be right about the reds. Might switch to greens and start getting reds if the greens fail. And I will get a red for my HTPC too.

I've thought the same thing about the parity drive. I haven't had the experience building one but a 4TB drive should hold me in good stead for a while. I've read that a faster parity drive can help performance so maybe I'll get a 4TB WD Caviar Black instead.

Good call about the cache drive. I really don't need one that big. I'm thinking a Samsung 830 256 GB SSD now. Think I'll upgrade the Datastore to a 528 GB too.

Johnm:

Yeah I remember there was a thread on here somewhere about that. I think someone had trouble adding the third m1015, from memory, and had to get an expander instead. Maybe it was a second m1015 though not sure. I will do some more research on that, cheers.

I'll have to add a UPS too I guess.

Thanks for the input both of you.

 

Johnm:

Yeah I remember there was a thread on here somewhere about that. I think someone had trouble adding the third m1015, from memory, and had to get an expander instead. Maybe it was a second m1015 though not sure. I will do some more research on that, cheers.

That was probably my thread you are talking about.  I built ans ESXi system on a X9SCM-IIF-O with threee M1015's.  I could never get all three to work in unRAID under ESXi with the Ivy Bridge CPU.  All three would work with the Ivy Bridge CPU in bare metal unRAID, but not running in ESXi.  I did, at one point, try running two M1015's with an expander, but parity sync speeds were atrocious (not sure why).  I tried everything I could think of and swapped out pretty much every component to rule out hardware failure and the pnly thing that fixed the problem was switch the Ivy Bridge CPU to a Sandy Bridge.  Once I did that everything worked great.

That was probably my thread you are talking about.  I built ans ESXi system on a X9SCM-IIF-O with threee M1015's.  I could never get all three to work in unRAID under ESXi with the Ivy Bridge CPU.  All three would work with the Ivy Bridge CPU in bare metal unRAID, but not running in ESXi.  I did, at one point, try running two M1015's with an expander, but parity sync speeds were atrocious (not sure why).  I tried everything I could think of and swapped out pretty much every component to rule out hardware failure and the pnly thing that fixed the problem was switch the Ivy Bridge CPU to a Sandy Bridge.  Once I did that everything worked great.

Yeah I did a bit of research last night. There seems to be a few issues floating around involving those components. Think I might end up with a Sandy Bridge too if I can find one. A shame because I put off building the thing to wait for it, then I put it off again due to your issue.

So how's your setup looking at the moment, RockDawg? X9SCM-IIF-O, Sandy Bridge and 3 M1015s? How many HDs are you running?

Thanks

It's been running great.  I have 18 drives plus parity in unAID, one drive as a datastore for ESXi, and two drives running Raid 0 in FreeNAS.  So glad I virtualized.

EDIT:  Oh Yeah, Johnm's Atlas thread is invaluable for setting everything up if you have any questions or get stuck.

I gotta' tell you - I'm really tempted to slap an Ivy in my machine and see what happens <sigh>. I have the same mobo as RockDawg but only need to pass thru 2 M1015s with the third being used by ESX. I figure worse comes to worse I can swap back but thus far haven't moved ahead buying a CPU. If it doesn't work I'm stuck with a CPU lol and these aren't cheap darn it! Wish there were some ESX boards that could do all of the nice IPMI stuff but were socketed for more common i7. <sigh> Ah well, I'm invested and will watch your build with interest. If I swap to an Ivy CPU I'll post results one way or the other, sadly it probably won't be fast enough to help you make your decision...

I gotta' tell you - I'm really tempted to slap an Ivy in my machine and see what happens <sigh>. I have the same mobo as RockDawg but only need to pass thru 2 M1015s with the third being used by ESX. I figure worse comes to worse I can swap back but thus far haven't moved ahead buying a CPU. If it doesn't work I'm stuck with a CPU lol and these aren't cheap darn it! Wish there were some ESX boards that could do all of the nice IPMI stuff but were socketed for more common i7. <sigh> Ah well, I'm invested and will watch your build with interest. If I swap to an Ivy CPU I'll post results one way or the other, sadly it probably won't be fast enough to help you make your decision...

Sorry for the delay but I've been slowly putting my system together. I ended up risking the Ivy in the X9SCM-iiF (bios 2.0a, unRAID Server Pro v5.0-rc11, ESXi 5.1) and while I haven't finished yet things do look promising.

I flashed 3 M1015s to P15 and installed them. The M1015 closest to the CPU (SLOT7 PCI-E 2.0 X8) has two cables going to the bottom two backplanes (I'll call them 1 and 2 counting up from the bottom) of the Norco 4224, the next M1015 along (SLOT6 PCI-E 2.0 X8) has two cables going to backplane 3 and 4 and the third M1015 (SLOT5 PCI-E 2.0 X4 ON X8) has one cable going to backplane 5.

While I only have 9 disks total to test with, and I haven't quite finished setting up the array, it seems that Unraid is seeing all the drives.

The drives I have are: 4TB Hitachi parity (installed in a 4224 drive bay), 512GB SSD datastore (not passed through to Unraid), 256GB SSD cache (physical RDM from onboard SATA), 2 x 2TB WD RED (installed in a 4224 drive bays) and 4 x 3TB WD RED (installed in a 4224 drive bays).

The only drives that I have connected to M1015s atm are the parity drive and the 6 x WD REDs. As such, I currently have 3 drives installed in drive bays on the bottom (backplane 1) row and 1 drive installed in each of rows/backplanes 2, 3, 4 and 5 (if that makes sense).

I have some screen shots below and it looks good to me but as I said, I haven't actually built the array yet. I've pre-cleared all of the drives 3 times each and will start to build the array and migrate my media tomorrow. I've attached my syslog too and don't seem to be getting any doorbell errors or whatever RockDawg seemed to be getting.

I'm not sure if drives are supposed to be listed twice as above. If those more knowledgeable than myself in this area see anything wrong with this please let me know.

Thanks

syslog_20130323.txt

There is an issue using Ivy with 3 M1015 passed to unRaid.

There is a post about it...

I think its related to the PCI-E 3.0

Check it before you strat moving data around.

Yah that's different than the problem that I had.  I couldn't get the unRAID VM to recognize all the drives I had.  It would boot up and ignore the drives connected to one or two of the cards depending upon my configuration.  To make sure you are not having my problem, I would make sure your 9 disks are spread out across all three cards and still recognized.

Sorry I can't help with the duplicate drive problem as I've never seen that before.

It's a theory only, but I don't think RockDawg's issue will surface until you get beyond a certain number of disks. 

It's a theory only, but I don't think RockDawg's issue will surface until you get beyond a certain number of disks.

Could be.  I don't remember every configuration I tried as I tried a lot, but IIRC, I was usually using all my disks.  That would suck for the OP if he got it working as is and then encountered my problem later after adding more disks.  OP, I would add every spare disk you may have lying around.  Even if it's not formatted for unRAID.  Again, my problem appeared while the unRAID VM was booting so you should be able to tell without risking the data on those drives or even having to format them..

Thanks all. I've got all the disks I have spare in there so I guess I'll cross that bridge if and when I come to it. With a 4TB disk limit I should be good for a while.