skynet + hal & wopr (esxi head + 2 unraid das)

[lboregard]

so, this is what i'm building ... it's a work in progress right now, as i'm waiting on the ram (last piece missing)

skynet (ESXi vSphere 5U1)
Case:  Cooler Master CM 690 II Advanced
Drive bays:  Cooler Master 4 in 3
CPU: Intel Xeon E3-1230 v2
Motherboard: Supermicro X9SCM-F-O
Storage Adapter: AOC-SASLP-MV8
Storage Adapters: 2x IBM RaidServe M1015 flashed to IT mode (from ebay)
RAM: KVR1600D3D4R11SK4/32G ECC kit (32G total)
PSU: SeaSonic X Series X650 Gold
Flash drives: 2x Kingston Digital DataTraveler 101 Generation 2 - 4 GB
SSD: OCZ 120GB Vertex 3
Hard Disks: 1.5Tb WD15EADS + 500Gb WD5000AAKS (cache + staging for usenet downloader)
Hard Disks: 2Tb WD20EADS + 2Tb Samsung HD204UI (backup storage)
Hard Disks: 2x 1Tb WD10EADS + 1Tb WD10EACS + 1Tb Seagate 7000.12 + 1Tb Samsung HD103UJ (datastore via nfs)
NIC: Intel Gigabit CT PCI-E Network Adapter EXPI9301CTBLK
Expansion Card: 2x Chenbro SAS Expander CK23601

hal (unRaid DAS)
Case:  Norco 4020 with stock fan plate and silent fans
Hard Disks: Parity + 20x 2Tb
Total Capacity: 40Tb

wopr (unRaid DAS)
Case:  Norco 4224 with 120mm Norco plate and ball bearing rails
Hard Disks: Parity + 24x 3Tb (planned)
Total Capacity: 72Tb (planned)

Primary Use
i'm planning to run the following VMs initially

unRaid (2x: hal & wopr) / 4Gb each
For movies and tv shows storage (mostly blu-ray or blu-ray rips)

Solaris 11 (as yet unnamed) / 12Gb
will hold two separate zfs pools
2 disk encrypted mirror to backup my main workstation and other machine's data
5 disk raidz to serve as a datastore

FreeBSD (hermes) / 4Gb
nzbget usenet downloader and sickbeard "pvr"

Ubuntu Server 12.04 (kepler) / 2Gb
gitlab, postgresql and other general purpose linux duties

Additional notes
i have a lot of work ahead of me, i have to build skynet, disassemble/reassemble my current norco unraid server, mount both norcos into my 4 post rack, make sure unRaid (hal at first) runs fine ... and stuff
needless to say, i got my inspiration from Johnm, BetaQuasi and other guy's awesome builds in this section

[Johnm]

OK, so your building what I have designed for my next project...
I will defiantly follow this thread.

Few quick  random thoughts oozing from my head before I have had coffee...
You have the wrong RAM. that is registered RAM.
The good news though... the correct RAM has dropped about 50% in price in the last month or so http://www.superbiiz.com/detail.php?name=W16GE1333K

I have already switched my datastores to ZFS RAIDz array shared VIA NFS and it was worth it. not just are they snappier, but they are now on redundant arrays and easier to backup. my benchmarks on a VM using the array are in the 450MB/s range. (M1015 w/ 4x samsung 2TB F4's)

You might consider using an SSD with advanced Garbage collection. my Vertex 3 didn't last a year. it might have had nothing to do with it being a datastore drive or not. but it will be taking a beating.

As long as you are using the  Chenbro expanders, I would try and find the Chenbro EUK for the DAS boxes. it will make life so much easier for the build. Provantage used to have them listed, now they seem to have stopped selling them.. hmm
http://usa.chenbro.com/corporatesite/products_detail.php?sku=76

I would also order one of these for skynet (your head). http://www.pc-pitstop.com/sas_cables_adapters/AD8788-4.asp
You might be able to get away with the 2 port version since the SAS Expanders only have a single input port. (see below why 4 ports would be used.)
Then get your 8088 cables from MONOPRICE or somewhere.

The way i would build the Head/DAS combo..

Code: [Select]

HEAD (I would stick the M1015 in the head)
M1015
| |
| |-- {channel 1 (SAS1)} for the Parity and cache (and 2 faster array drives if you want) and leave it in the head.
|
|DAS
| --{Channel 2 (SAS2)} Expander with 20-24 DATA drives
The other option is to run a second 8088 to the DAS and use one of the PC PITSTOP adapters and put the Parity into the DAS on its own Channel.


I think naming the servers after computers that went nuts and declared war on their owners is asking for trouble.. Especially since at least 2 of them were clearly homicidal
PS, I think HAL's full name is HAL 9000.


EDIT:.. for the cache drives... you could use virtual drives from your NFS share. I have a 250MB virtual drive now. the nice part is, it is a protected drive since it is on a raidz.

It looks like you're trying to use what you already own.. I would consider moving the head to a Norco..  just to keep it all symmetrical. eventually that is...

[lboregard]

Few quick  random thoughts oozing from my head before I have had coffee...
You have the wrong RAM. that is registered RAM.
The good news though... the correct RAM has dropped about 50% in price in the last month or so http://www.superbiiz.com/detail.php?name=W16GE1333K

f&$k !! you're right, i have to get the correct ram 

You might consider using an SSD with advanced Garbage collection. my Vertex 3 didn't last a year. it might have had nothing to do with it being a datastore drive or not. but it will be taking a beating.

eventually will

As long as you are using the  Chenbro expanders, I would try and find the Chenbro EUK for the DAS boxes. it will make life so much easier for the build. Provantage used to have them listed, now they seem to have stopped selling them.. hmm
http://usa.chenbro.com/corporatesite/products_detail.php?sku=76

i'm going commando at first, but if it turns out to be a problem, then yes, i'll get the kit

The way i would build the Head/DAS combo..

Code: [Select]

HEAD (I would stick the M1015 in the head)
M1015
| |
| |-- {channel 1 (SAS1)} for the Parity and cache (and 2 faster array drives if you want) and leave it in the head.
|
|DAS
| --{Channel 2 (SAS2)} Expander with 20-24 DATA drives
The other option is to run a second 8088 to the DAS and use one of the PC PITSTOP adapters and put the Parity into the DAS on its own Channel.

i'm currently planning to leave parity on the DAS. No cache, as the 1.5Tb will basically serve that purpose for both unRaid DAS.
why are you suggesting to leave the parity on the head ? will it have too big an impact on parity calculation leaving it in the DAS ?

I think naming the servers after computers that went nuts and declared war on their owners is asking for trouble.. Especially since at least 2 of them were clearly homicidal
PS, I think HAL's full name is HAL 9000.

i've had hal for about 2 years now and it's been behaving, although i keep a very close eye on it 
you're right, it's HAL 9000, i took an artistic license there

thanks for your comments Johnm

[Johnm]

why are you suggesting to leave the parity on the head ? will it have too big an impact on parity calculation leaving it in the DAS ?

You could have it in the DAS, You could have it in the head..
MY thought was for performance. you are going to only use one port on the M1015.. I was thinking put the Parity (cache if you used one) on the second port..
If you do that.. leaving it in the head eliminates an expensive cable and 8087 to 8088 adapter..
Leaving it in the das (on its own channel via a second 8088 and its own port) would give you option to power it off.. so that is an advantage..

As I said, there is no real reason other then performance (if it even helps at all? it might not)

I am guessing you are going to toss parity on the expander.. I am sure that will work. I would be very interested in seeing what 24 drives on a single SAS2 channel look like when doing a parity compute..

[lboregard]

I am guessing you are going to toss parity on the expander.. I am sure that will work. I would be very interested in seeing what 24 drives on a single SAS2 channel look like when doing a parity compute..

i think we'll both find out soon (at least for 20 drives) 

[ldasilva]

I am guessing you are going to toss parity on the expander.. I am sure that will work. I would be very interested in seeing what 24 drives on a single SAS2 channel look like when doing a parity compute..

i think we'll both find out soon (at least for 20 drives) 

me to i plan on doing the same thing as i fill up my drives

[Johnm]

I am guessing you are going to toss parity on the expander.. I am sure that will work. I would be very interested in seeing what 24 drives on a single SAS2 channel look like when doing a parity compute..

i think we'll both find out soon (at least for 20 drives) 

me to i plan on doing the same thing as i fill up my drives

Right now I am running 16 + 4 with no impact. 16 on channel1 and 4 on channel 2. my parity is on channel 2.
(I would be willing to move a sas cable to the expander and test a full load of 20 on one channel. I fried my expander and it is mid RMA)
I would go larger myself, unfortunately i ran out of room in my norco. I have 20x 3TB drives for unRAID and 4x 2TB drives for my ZFS. Hence, I need to look into the DAS thing myself to get my additional drives into me system.

I am thinking 3x RPC4224 or 2x RPC4224 + 1x 2212.
I'll probably for with 3x 4224 for flexibility and future expandability.

[ldasilva]

i already have a Norco 4020 which is the head right now i will most likely get another Norco 4020 since its cheaper then the Norco 4424
down the line i will put in a motherboard as im taxing my e3-1230 close to 100% cpu

[lboregard]

i had a glimpse at awesomeness today !!!

i managed to boot unraid as guest inside my esxi host, and was able to access all disks and shares ... woohooo !!!

but then ...

well, there were a couple of hiccups along the way

- the first couple of times i tried, i had a red dot drive .. it was the drive connected to one of the chenbro ports via a sata fanout cable. in the end, i had a norco reverse breakout cable which wouldnt work, but fortunately i also had this forward breakout cable i could use. when i plugged it in, voila ! all drives available, i was able to start the array !!!
- as i was hauling the norco 4220 case around, i managed to unhinge the power connector of my power supply (corsair hx850). Although it worked, i declared it dead. since i had a corsair ax850 for the other das, i went about plugging it in. this is when i found out there aren't enough molex connector to power 5 backplanes, fanplate fan connector AND the chenbro. bad luck man ! i've already ordered this sata to molex adapter to power the chenbro.

some other notes
- i've never seen unraid boot so fast ... it was awesome (i used johnm method, option 2, derived from briangp original method). it was easy to create a 2gb unraid vdmk with

Code: [Select]

vmkfstools -c 2147483648 -a lsilogic -d thin unraid-50rc5.vmdk, copy your chosen unraid version, make_bootable and then offer it to the unraid guest, to boot off of it.
- i found out a supermicro aoc saslp-mv8 doesnt work with solaris 11, so im off to buy another m1015 ... i will end up with 3 of them

a couple of pics of my ghetto rack and some of the gear (sorry for the crap iphone pics)

GHETTO RACK


top row : motorola surfboard / hp procurve 1410-24G switch / Linksys WRT54GL router+wifi with tomato firmware / generic 14" monitor
2nd row : dell pentium III 500mhz pfsense firewall / skynet / harmony one charger / cyberpower CP1350AVRLCD ups
3rd row : set aside for wopr (das #2)
4th row : hal (das #1)

the dell was the first machine i purchased (1999) and has been so resilient, that i'll run it til the very end
i'm not done with cable management, but it was just too much for this round, will do something about it next time

skynet (head)


the sff8088 cables are coming out the watercooler holes in the case 
in that pic i still had the saslp-mv8

hal (das #1)


the chenbro performing like da man !

[Johnm]

nice...

It is coming along quite well.

Are you using the PC that is in the DAS for something else or is it just a source for PCIe power and an easy switch for the DAS?

[Influencer]

Ok, now I'm going to be up all night figuring out a reason to build a esxi box...

Ok, I don't need a reason outside of the geek factor, but my wife will need the Stefan to my Urkel

[lboregard]

nice...

It is coming along quite well.

Are you using the PC that is in the DAS for something else or is it just a source for PCIe power and an easy switch for the DAS?

thank you Johnm.

no other use for the pc/mobo, it doesn't even power the chenbro (since  the chenbro needs a molex to operate).
the uek you mentioned previously is the way to go ... but they seem to have been discontinued (can't find them even on ebay)

[lboregard]

Ok, now I'm going to be up all night figuring out a reason to build a esxi box...

i will only say ... it's worth it

[Johnm]

The uek is discontinued? Ack. I saw them cheap recently. I should have grabbed some.


Sent from my iPhone using Tapatalk

[Influencer]

Yeah if I can find cheap enough hardware, my current build won't cut it. Dual core CPU, 8 gb ram.

I'll probably start buying pieces here and there, just pulled the tranny out of my trooper for a rebuild, hobby money's going there for a while,:)