Author Topic: Minion - my ESXi build! :-)  (Read 8451 times)

Offline BLKMGK

  • Hero Member
  • *****
  • Posts: 829
Minion - my ESXi build! :-)
« on: October 25, 2012, 03:25:22 PM »
Okay, just after buying upgrade parts for my two smaller servers I asked BobPhoenix offline about his ESXi server and what advantages ESX offered. Ooops! I've always been interested in ESX and even have a small "test" ESX machine here running an i7 so once he explained how his setup worked and how vt-D benefited unRAID I was sold and ordered a(nother) pile of parts. D'oh! One thing of sad note, my previous ESX server had little to offer this build. It was budget and hasn't got enough PCIE slots. It was also an i7 build in which I foolishly purchased a 2600K CPU, K CPU don't have vt-D capability - argh!

So, I've had to buy pretty much everything for this. My parts list is as follows (with much credit to Bob and to Johnm :

1x Norco 4224 4u chassis - a coupon for which I posted in the deals section http://www.newegg.com/Product/Product.aspx?Item=N82E16811219038
1x Tyan S5512GM2NR motherboard (2.2 firmware) http://www.newegg.com/Product/Product.aspx?Item=N82E16813151245
1x Intel XEON CPU (note that it's a Sandy CPU and not an Ivy) http://www.newegg.com/Product/Product.aspx?Item=N82E16819115084
1x set of 32Gig ECC memory from Kingston http://www.newegg.com/Product/Product.aspx?Item=N82E16820239371
1x Seasonic 750X PSU http://www.newegg.com/Product/Product.aspx?Item=N82E16817151087
6x Monoprice SFF8087 cables (too long!) http://www.monoprice.com/products/product.asp?c_id=102&cp_id=10254&cs_id=1025410&p_id=8189&seq=1&format=2
3x M1015 SAS adapters off of eBay - flashed them into "it" mode. firmware rev 14
NO HDD yet, I intend to spin it up with spares, move data, and migrate my unRAID over slowly. I expect to max out the unRAID VM with 16 drives on two vt-D adapters and save 8 slots for ESX on a single adapter

Some things to note:
1) I had NO issue flashing the M1015 cards with a DOS boot flash drive - discovered after first trying to flash via UEFI module <sigh>
2) These SAS cables are too long, I need something about a FOOT shorter. They're cheap though!
3) I was told that if you didn't put drives in all slots of the Norco that airflow would be fine, that flaps would cover the open slots. Nope, this isn't true, I expect a full system would flow air best.
4) Noise - the Norco isn't nearly as bad as I thought it would be!
5) I'd prefer an Ivy CPU and it was close in price. However others are having issues with them on ESX and even Ivy firmware with a Sandy CPU seems a problem - I've got Sandy firmware. the issues with a Sandy CPU seem mostly related to the PCI slot which thankfully I won't be using right now.

I'm to the point of installing ESX now and will be running the system on a fresh unRAID license. Much work left to be done and nothing so far is what I'd call "original" but if I run into any quirks or can help others I will. Stay tuned, there's bound to be bumps in the road and things learned as I go. I'll snap some pics when I have some time, currently it's on my coffee table and it looks like a NewEgg bomb went off in there :D
« Last Edit: October 25, 2012, 03:27:04 PM by BLKMGK »

Offline BLKMGK

  • Hero Member
  • *****
  • Posts: 829
Re: Minion - my ESXi build! :-)
« Reply #1 on: October 25, 2012, 10:14:02 PM »
Well, I was able to passthrough the correct M1015s to the ESX server on my first try - yay! However when I attempt to passthrough either of the two USB controllers the system drops the setting upon reboot.

I've tried removing all other USB devices save the boot stick (including cable to front ports) and setting either of the controllers in ESX - no go. I've also tried moving the boot stick to a port on the back of the machine - no go. On the plus side I've not managed to render the system unstable with my mucking around. Trying to figure out how to use an SSD for cache support to the VM drives but that's purely secondary at this point :(

Attached is my list of adapters. If I add either of the top two they disappear on reboot...

Should I install ESX to a disk and see if it will allow access to a USB bus then? :(

Offline BobPhoenix

  • Hero Member
  • *****
  • Posts: 1819
Re: Minion - my ESXi build! :-)
« Reply #2 on: October 26, 2012, 05:56:57 AM »
I have one of mine passed through just fine.  I would set the (2nd one I believe) port to pass through - reboot ESXi and it would be enabled.  I have seen it loose pass through on some/many of the devices if I plug a new one into one of the PCIe/PCI ports before.  I would have to select it/them again and reboot - but I have never failed to get it to pass through.  It is possible that the USB hubs are considered to be PCI rather than PCIe devices and that doesn't work well with ESXi 5.1 so far. But I would have thought you would get pink screens when trying to pass them through rather than just having them dropped.  You could try ESXi 5.0 which is what I'm using currently since I have a PCI card on my Tyan to pass through.  As soon as an update is available (maybe it is now I haven't checked in the past couple of weeks or so) I could try ESXi 5.1 and see if I have any pass through problems on the USB hubs.
1 - 6.0b5a unRAID server: Lian Li V2000, Tyan S5512GM4NR, 16GB, E3-1240, IBM M1015 to Intel RES2SV240, Win7x64 VM 6GB.
2 - 5.0rc16c unRAID server: Lian Li V2000, Tyan S5512GM2NR, 2GB, VM - ESXi 5.0, E3-1230, IBM M1015 to Intel RES2SV240.
3 - 5.0rc16c unRAID server: Lian Li V2000, SM X9SCM-F, 2GB, VM - ESXi 5.0, E3-1230, IBM M1015 to Intel RES2SV240.
4 - 6.0b5a unRAID server: Lian Li V2000, SM X7SBE, 8GB, Q9550, 10x 3TB Toshibas - 1 WHS2011 VM.
5 - 5.0 unRAID server: N54L, 5 - 4TB Seagate ST4000DM000 1 3TB ST3000DM001
6 - 6.0b5a unRAID server: N40L, 9 - 2,3,4TB 7200rpm Hitachi drives 5 in external cage

Offline BLKMGK

  • Hero Member
  • *****
  • Posts: 829
Re: Minion - my ESXi build! :-)
« Reply #3 on: October 26, 2012, 06:07:40 AM »
Ouch, yeah I think the difference between our setups is my usage of 5.1 latest and greatest vs your using the 5.0 release. Looks liek that will be my next step to try!

BTW there's a 2.2b BIOS that was recently released for this board.

Offline BobPhoenix

  • Hero Member
  • *****
  • Posts: 1819
Re: Minion - my ESXi build! :-)
« Reply #4 on: October 26, 2012, 06:16:27 AM »
Ouch, yeah I think the difference between our setups is my usage of 5.1 latest and greatest vs your using the 5.0 release. Looks liek that will be my next step to try!

BTW there's a 2.2b BIOS that was recently released for this board.
I'm not going there.  They don't mention anything in the release notes for 2.02b that will help me out.  I'm hoping to downgrade my S5512GM4NR this weekend to the 1.05a tech support sent me so I can use my tuner cards that are not recognized.
1 - 6.0b5a unRAID server: Lian Li V2000, Tyan S5512GM4NR, 16GB, E3-1240, IBM M1015 to Intel RES2SV240, Win7x64 VM 6GB.
2 - 5.0rc16c unRAID server: Lian Li V2000, Tyan S5512GM2NR, 2GB, VM - ESXi 5.0, E3-1230, IBM M1015 to Intel RES2SV240.
3 - 5.0rc16c unRAID server: Lian Li V2000, SM X9SCM-F, 2GB, VM - ESXi 5.0, E3-1230, IBM M1015 to Intel RES2SV240.
4 - 6.0b5a unRAID server: Lian Li V2000, SM X7SBE, 8GB, Q9550, 10x 3TB Toshibas - 1 WHS2011 VM.
5 - 5.0 unRAID server: N54L, 5 - 4TB Seagate ST4000DM000 1 3TB ST3000DM001
6 - 6.0b5a unRAID server: N40L, 9 - 2,3,4TB 7200rpm Hitachi drives 5 in external cage

Offline BLKMGK

  • Hero Member
  • *****
  • Posts: 829
Re: Minion - my ESXi build! :-)
« Reply #5 on: October 26, 2012, 07:57:21 AM »
Aaaaand - dropping to 5.0u1 has solved this issue! I am now able to see the USB host controller #2 passed-through. I will now pass through the two SAS controller cards, load up the Plop ISO for booting, and see if it recognizes unRAID. After staying up so late last night and being frustrated beyond belief when I went to bed today has now become a full day of "get this thing running" so stay tuned - I hope to have a functioning unRAID\ESX host by COB today :D

Offline BLKMGK

  • Hero Member
  • *****
  • Posts: 829
Re: Minion - my ESXi build! :-)
« Reply #6 on: October 26, 2012, 09:25:31 AM »
Okay, things got weird. I loaded up 5.0U1 and installed my USB passthru - success! I then added the two SAS cards - Failure! I then tried to add my license - Failure! I did a new install and installed the USB, SAS cards, and license all in one go - Success!

For reasons I cannot explain it seems I only get to add things once. Permissions issue? I dunno'! I'm going to now try this with 5.1 and see if I can get it working. :-)

Offline kaiguy

  • Sr. Member
  • ****
  • Posts: 349
Re: Minion - my ESXi build! :-)
« Reply #7 on: October 26, 2012, 09:28:08 AM »
Nice looking build.

I posted about the problem passing through a USB controller with 5.1 here, but you seemed to figure it out fine on your own :)
ESXi 5.5 - unRAID 5.0.5 Pro - Supermicro MBD-X9SCM-F-O - Intel Xeon E3-1230 - 16 GB ECC DDR3 - Seasonic 760W Platinum - M1015 -  Intel RES2SV240 - WD Red/Green drives (21 TB array) - WD Black cache - 120 GB SATA3 SSD - Norco 4224

Offline BLKMGK

  • Hero Member
  • *****
  • Posts: 829
Re: Minion - my ESXi build! :-)
« Reply #8 on: October 26, 2012, 12:49:02 PM »
Nice looking build.

I posted about the problem passing through a USB controller with 5.1 here, but you seemed to figure it out fine on your own :)

Well, that seems to address PCI devices having issues - something Bob has seen and others. In my case I cannot add the USB controller to the config but I get no warnings, crashes, or odd colored screens.

As of right now I have everything I need passed through. I did it by loading 5.0U1 and then configuring it all in one fell swoop. On reboot it's there! Now I can see about accessing the unRAID stick and booting unRAID. Yeesh! Perhaps I should rename this Pandora?

Offline BLKMGK

  • Hero Member
  • *****
  • Posts: 829
Re: Minion - my ESXi build! :-)
« Reply #9 on: October 26, 2012, 01:22:03 PM »
Got as far as creating a VM with the Plop boot ISO when I found no USB device attached to the BUS. I saved the VM and test booted it but with no USB it didn't go far and seemed to have issues drawing the screen. Puzzled about the lack of a USB device I rebooted the system. My VM is now MIA!

I can browse the datastore and see the VM directory so I know I'm not crazy. It really does seem like changes are only ever allowed once. I note that the USB is dark and not flashing like it does usually. I'm going to try adding the OTHER USB bus, maybe this is the system not being able to access the flash to save changes??

Offline BobPhoenix

  • Hero Member
  • *****
  • Posts: 1819
Re: Minion - my ESXi build! :-)
« Reply #10 on: October 26, 2012, 03:49:15 PM »
Got as far as creating a VM with the Plop boot ISO when I found no USB device attached to the BUS. I saved the VM and test booted it but with no USB it didn't go far and seemed to have issues drawing the screen. Puzzled about the lack of a USB device I rebooted the system. My VM is now MIA!

I can browse the datastore and see the VM directory so I know I'm not crazy. It really does seem like changes are only ever allowed once. I note that the USB is dark and not flashing like it does usually. I'm going to try adding the OTHER USB bus, maybe this is the system not being able to access the flash to save changes??
Browse into the directory and select the VMX file and Add to inventory like this
1 - 6.0b5a unRAID server: Lian Li V2000, Tyan S5512GM4NR, 16GB, E3-1240, IBM M1015 to Intel RES2SV240, Win7x64 VM 6GB.
2 - 5.0rc16c unRAID server: Lian Li V2000, Tyan S5512GM2NR, 2GB, VM - ESXi 5.0, E3-1230, IBM M1015 to Intel RES2SV240.
3 - 5.0rc16c unRAID server: Lian Li V2000, SM X9SCM-F, 2GB, VM - ESXi 5.0, E3-1230, IBM M1015 to Intel RES2SV240.
4 - 6.0b5a unRAID server: Lian Li V2000, SM X7SBE, 8GB, Q9550, 10x 3TB Toshibas - 1 WHS2011 VM.
5 - 5.0 unRAID server: N54L, 5 - 4TB Seagate ST4000DM000 1 3TB ST3000DM001
6 - 6.0b5a unRAID server: N40L, 9 - 2,3,4TB 7200rpm Hitachi drives 5 in external cage

Offline BobPhoenix

  • Hero Member
  • *****
  • Posts: 1819
Re: Minion - my ESXi build! :-)
« Reply #11 on: October 26, 2012, 04:02:20 PM »
Just setup pass through on what is labeled USB Enhanced Host Controller #1 (2nd on list in ESXi device display).  Had no problems with it sticking so not sure what to tell you about that.

Did discover which USB ports are connected to which hub:
  • Internal USB ports (2) and front USB MB header is connected to USB Enhanced Host Controller #2 (1st on list in ESXi device display)
  • Back IO shield USB ports (4) and the back USB MB header is connected to USB Enhanced Host Controller #1 (2nd on list in ESXi device display)
1 - 6.0b5a unRAID server: Lian Li V2000, Tyan S5512GM4NR, 16GB, E3-1240, IBM M1015 to Intel RES2SV240, Win7x64 VM 6GB.
2 - 5.0rc16c unRAID server: Lian Li V2000, Tyan S5512GM2NR, 2GB, VM - ESXi 5.0, E3-1230, IBM M1015 to Intel RES2SV240.
3 - 5.0rc16c unRAID server: Lian Li V2000, SM X9SCM-F, 2GB, VM - ESXi 5.0, E3-1230, IBM M1015 to Intel RES2SV240.
4 - 6.0b5a unRAID server: Lian Li V2000, SM X7SBE, 8GB, Q9550, 10x 3TB Toshibas - 1 WHS2011 VM.
5 - 5.0 unRAID server: N54L, 5 - 4TB Seagate ST4000DM000 1 3TB ST3000DM001
6 - 6.0b5a unRAID server: N40L, 9 - 2,3,4TB 7200rpm Hitachi drives 5 in external cage

Offline BLKMGK

  • Hero Member
  • *****
  • Posts: 829
Minion - my ESXi build! :-)
« Reply #12 on: October 26, 2012, 05:02:22 PM »
I'm out to dinner now but that is VERY helpful information! I have been using the front header and the USB A socket for my sticks. I believe some of my issues has been ESX losing access to its stick. This would explain why configs sometimes don't stick, the system has no ability to write to persistent storage when the controller for the stick is passed. Moving the stick allowed me to change things sometimes, no PSOD though :-)

I will reload, put things in proper sockets, and try again. I'd been searching all over for the USB info, you've saved me much frustration I think. I also appreciate the hint on the adding of VMX files, awesome! Hopefully this also helps others down the line....

Offline BetaQuasi

  • Hero Member
  • *****
  • Posts: 720
  • Beer fixes everything
Re: Minion - my ESXi build! :-)
« Reply #13 on: October 26, 2012, 09:31:40 PM »
The length of the Monoprice cables annoyed me too - you might want to consider looking into getting some of these instead:

http://lime-technology.com/forum/index.php?topic=19332.msg186529#msg186529

Have been using them for a few months now - they are the perfect length!

Offline BLKMGK

  • Hero Member
  • *****
  • Posts: 829
Re: Minion - my ESXi build! :-)
« Reply #14 on: October 26, 2012, 11:18:24 PM »
Okay, I think I've got this figured out. Bob's help was invaluable! The answer is interesting IMO and my results are differing from his setup!! We might want to compare board revisions :)

What was apparently happening was I was co-opting the USB bus the ESX stick was sitting on. Changes made to the config after this point couldn't be saved and explains why my VMs were disappearing. Why I wasn't getting PSOD I cannot explain.

The board's USB controller #1 appears to support ONE of the USB A ports - the one closest to the PCI covers and the USB pin header that is closest to the slot covers as well as the ports on the rear I/O shield. The #2 controller supports USB Pin header closest to the drive bays and the USB A port that's also closest to the drive bays. NOT the same as what was seen on Bob's I don't think but once I was sure the boot stick was being taken away it was just a matter of installing a few times and playing port bingo. Grrr! Having the ESX stick on a known good bus I could move a kbrd around and look to see when it worked - it would only work in a bus that ESX was NOT passing thru.

Once I knew the correct mapping of the controllers to ports and got the USB sticks separated properly and mapped things began working. I am using controller #1 for passthru which allows the boot stick to be in an A port that's closest to the drive bays. The unRAID key will be placed in the USB A port closest to the rear of the case (closest to PCI covers). The case header will be attached to the portion of the mobo closest to the drive bays that has a header and I can use this for kbrd access. Doing this will prevent my USB kbrd from being plugged into the back panel and I'll be forced to use the front port or IPKVM full time.

Finally, this appears to be working. Hopefully my experience can help another! (crossposted from HardOCP where I was also asking for ESX help but Bob here saved the day and has proven most patient) ;-)

BetaQuasi, I'll be grabbing some shorter cables ASAP - thank you! I may check the US eBay first just in case but that guy does look to ship worldwide - yay!

P.S. check these auctions: http://www.ebay.com/itm/Mini-SAS-36-Pin-Mini-SAS-36-Pin-SFF-8087-Data-Cable-/220964190145?_trksid=p2047675.m1851&_trkparms=aid%3D222002%26algo%3DSIC.FIT%26ao%3D1%26asc%3D110%26meid%3D3030354693275426027%26pid%3D100005%26prg%3D1063%26rk%3D3%26sd%3D270832045016%26
http://www.ebay.com/itm/New-0-6M-SFF-8087-Mini-SAS-36-Pin-to-Mini-SAS-36Pin-Sever-Raid-Card-Data-Cable-/270832045016?pt=US_Drive_Cables_dapters&hash=item3f0ed90bd8
http://www.ebay.com/itm/New-0-6M-SFF-8087-Mini-SAS-36-Pin-to-Mini-SAS-36Pin-Sever-Raid-Card-Data-Cable-/290618968254?pt=UK_Computing_Drive_Cables_Adapters&hash=item43aa3d88be

The last one is using the same pics as the AU site. However the first one has positive reviews and the cables are $6 w/free shipping.  :D Now I just need custom Molex for my backplanes - I have a modular Seasonic PSU but no joy on eBay. I may have to solder and shrink these myself :(
« Last Edit: October 27, 2012, 12:57:51 AM by BLKMGK »