Minion - my ESXi build! :-)


Recommended Posts

Okay, just after buying upgrade parts for my two smaller servers I asked BobPhoenix offline about his ESXi server and what advantages ESX offered. Ooops! I've always been interested in ESX and even have a small "test" ESX machine here running an i7 so once he explained how his setup worked and how vt-D benefited unRAID I was sold and ordered a(nother) pile of parts. D'oh! One thing of sad note, my previous ESX server had little to offer this build. It was budget and hasn't got enough PCIE slots. It was also an i7 build in which I foolishly purchased a 2600K CPU, K CPU don't have vt-D capability - argh!

 

So, I've had to buy pretty much everything for this. My parts list is as follows (with much credit to Bob and to Johnm :

 

1x Norco 4224 4u chassis - a coupon for which I posted in the deals section http://www.newegg.com/Product/Product.aspx?Item=N82E16811219038

1x Tyan S5512GM2NR motherboard (2.2 firmware) http://www.newegg.com/Product/Product.aspx?Item=N82E16813151245

1x Intel XEON CPU (note that it's a Sandy CPU and not an Ivy) http://www.newegg.com/Product/Product.aspx?Item=N82E16819115084

1x set of 32Gig ECC memory from Kingston http://www.newegg.com/Product/Product.aspx?Item=N82E16820239371

1x Seasonic 750X PSU http://www.newegg.com/Product/Product.aspx?Item=N82E16817151087

6x Monoprice SFF8087 cables (too long!) http://www.monoprice.com/products/product.asp?c_id=102&cp_id=10254&cs_id=1025410&p_id=8189&seq=1&format=2

3x M1015 SAS adapters off of eBay - flashed them into "it" mode. firmware rev 14

NO HDD yet, I intend to spin it up with spares, move data, and migrate my unRAID over slowly. I expect to max out the unRAID VM with 16 drives on two vt-D adapters and save 8 slots for ESX on a single adapter

 

Some things to note:

1) I had NO issue flashing the M1015 cards with a DOS boot flash drive - discovered after first trying to flash via UEFI module <sigh>

2) These SAS cables are too long, I need something about a FOOT shorter. They're cheap though!

3) I was told that if you didn't put drives in all slots of the Norco that airflow would be fine, that flaps would cover the open slots. Nope, this isn't true, I expect a full system would flow air best.

4) Noise - the Norco isn't nearly as bad as I thought it would be!

5) I'd prefer an Ivy CPU and it was close in price. However others are having issues with them on ESX and even Ivy firmware with a Sandy CPU seems a problem - I've got Sandy firmware. the issues with a Sandy CPU seem mostly related to the PCI slot which thankfully I won't be using right now.

 

I'm to the point of installing ESX now and will be running the system on a fresh unRAID license. Much work left to be done and nothing so far is what I'd call "original" but if I run into any quirks or can help others I will. Stay tuned, there's bound to be bumps in the road and things learned as I go. I'll snap some pics when I have some time, currently it's on my coffee table and it looks like a NewEgg bomb went off in there :D

Link to comment

Well, I was able to passthrough the correct M1015s to the ESX server on my first try - yay! However when I attempt to passthrough either of the two USB controllers the system drops the setting upon reboot.

 

I've tried removing all other USB devices save the boot stick (including cable to front ports) and setting either of the controllers in ESX - no go. I've also tried moving the boot stick to a port on the back of the machine - no go. On the plus side I've not managed to render the system unstable with my mucking around. Trying to figure out how to use an SSD for cache support to the VM drives but that's purely secondary at this point :(

 

Attached is my list of adapters. If I add either of the top two they disappear on reboot...

 

Should I install ESX to a disk and see if it will allow access to a USB bus then? :(

ESX-Argh.png.37c397e32f3c9d04204c46df5b631972.png

Link to comment

I have one of mine passed through just fine.  I would set the (2nd one I believe) port to pass through - reboot ESXi and it would be enabled.  I have seen it loose pass through on some/many of the devices if I plug a new one into one of the PCIe/PCI ports before.  I would have to select it/them again and reboot - but I have never failed to get it to pass through.  It is possible that the USB hubs are considered to be PCI rather than PCIe devices and that doesn't work well with ESXi 5.1 so far. But I would have thought you would get pink screens when trying to pass them through rather than just having them dropped.  You could try ESXi 5.0 which is what I'm using currently since I have a PCI card on my Tyan to pass through.  As soon as an update is available (maybe it is now I haven't checked in the past couple of weeks or so) I could try ESXi 5.1 and see if I have any pass through problems on the USB hubs.

Link to comment

Ouch, yeah I think the difference between our setups is my usage of 5.1 latest and greatest vs your using the 5.0 release. Looks liek that will be my next step to try!

 

BTW there's a 2.2b BIOS that was recently released for this board.

I'm not going there.  They don't mention anything in the release notes for 2.02b that will help me out.  I'm hoping to downgrade my S5512GM4NR this weekend to the 1.05a tech support sent me so I can use my tuner cards that are not recognized.
Link to comment

Aaaaand - dropping to 5.0u1 has solved this issue! I am now able to see the USB host controller #2 passed-through. I will now pass through the two SAS controller cards, load up the Plop ISO for booting, and see if it recognizes unRAID. After staying up so late last night and being frustrated beyond belief when I went to bed today has now become a full day of "get this thing running" so stay tuned - I hope to have a functioning unRAID\ESX host by COB today :D

Link to comment

Okay, things got weird. I loaded up 5.0U1 and installed my USB passthru - success! I then added the two SAS cards - Failure! I then tried to add my license - Failure! I did a new install and installed the USB, SAS cards, and license all in one go - Success!

 

For reasons I cannot explain it seems I only get to add things once. Permissions issue? I dunno'! I'm going to now try this with 5.1 and see if I can get it working. :-)

Link to comment

Nice looking build.

 

I posted about the problem passing through a USB controller with 5.1 here, but you seemed to figure it out fine on your own :)

 

Well, that seems to address PCI devices having issues - something Bob has seen and others. In my case I cannot add the USB controller to the config but I get no warnings, crashes, or odd colored screens.

 

As of right now I have everything I need passed through. I did it by loading 5.0U1 and then configuring it all in one fell swoop. On reboot it's there! Now I can see about accessing the unRAID stick and booting unRAID. Yeesh! Perhaps I should rename this Pandora?

ESX-_grr.png.808e9607368cc8d88c9e06695f7b7211.png

Link to comment

Got as far as creating a VM with the Plop boot ISO when I found no USB device attached to the BUS. I saved the VM and test booted it but with no USB it didn't go far and seemed to have issues drawing the screen. Puzzled about the lack of a USB device I rebooted the system. My VM is now MIA!

 

I can browse the datastore and see the VM directory so I know I'm not crazy. It really does seem like changes are only ever allowed once. I note that the USB is dark and not flashing like it does usually. I'm going to try adding the OTHER USB bus, maybe this is the system not being able to access the flash to save changes??

Link to comment

Got as far as creating a VM with the Plop boot ISO when I found no USB device attached to the BUS. I saved the VM and test booted it but with no USB it didn't go far and seemed to have issues drawing the screen. Puzzled about the lack of a USB device I rebooted the system. My VM is now MIA!

 

I can browse the datastore and see the VM directory so I know I'm not crazy. It really does seem like changes are only ever allowed once. I note that the USB is dark and not flashing like it does usually. I'm going to try adding the OTHER USB bus, maybe this is the system not being able to access the flash to save changes??

Browse into the directory and select the VMX file and Add to inventory like this

Import_VM.png.fd6ca5a63bed1de33a0739112c28b684.png

Link to comment

Just setup pass through on what is labeled USB Enhanced Host Controller #1 (2nd on list in ESXi device display).  Had no problems with it sticking so not sure what to tell you about that.

 

Did discover which USB ports are connected to which hub:

[*]Internal USB ports (2) and front USB MB header is connected to USB Enhanced Host Controller #2 (1st on list in ESXi device display)

[*]Back IO shield USB ports (4) and the back USB MB header is connected to USB Enhanced Host Controller #1 (2nd on list in ESXi device display)

S5512GM2NR_-_post_reboot.png.a966a1b3295ec784a97f31d498767e83.png

Link to comment

I'm out to dinner now but that is VERY helpful information! I have been using the front header and the USB A socket for my sticks. I believe some of my issues has been ESX losing access to its stick. This would explain why configs sometimes don't stick, the system has no ability to write to persistent storage when the controller for the stick is passed. Moving the stick allowed me to change things sometimes, no PSOD though :-)

 

I will reload, put things in proper sockets, and try again. I'd been searching all over for the USB info, you've saved me much frustration I think. I also appreciate the hint on the adding of VMX files, awesome! Hopefully this also helps others down the line....

Link to comment

Okay, I think I've got this figured out. Bob's help was invaluable! The answer is interesting IMO and my results are differing from his setup!! We might want to compare board revisions :)

 

What was apparently happening was I was co-opting the USB bus the ESX stick was sitting on. Changes made to the config after this point couldn't be saved and explains why my VMs were disappearing. Why I wasn't getting PSOD I cannot explain.

 

The board's USB controller #1 appears to support ONE of the USB A ports - the one closest to the PCI covers and the USB pin header that is closest to the slot covers as well as the ports on the rear I/O shield. The #2 controller supports USB Pin header closest to the drive bays and the USB A port that's also closest to the drive bays. NOT the same as what was seen on Bob's I don't think but once I was sure the boot stick was being taken away it was just a matter of installing a few times and playing port bingo. Grrr! Having the ESX stick on a known good bus I could move a kbrd around and look to see when it worked - it would only work in a bus that ESX was NOT passing thru.

 

Once I knew the correct mapping of the controllers to ports and got the USB sticks separated properly and mapped things began working. I am using controller #1 for passthru which allows the boot stick to be in an A port that's closest to the drive bays. The unRAID key will be placed in the USB A port closest to the rear of the case (closest to PCI covers). The case header will be attached to the portion of the mobo closest to the drive bays that has a header and I can use this for kbrd access. Doing this will prevent my USB kbrd from being plugged into the back panel and I'll be forced to use the front port or IPKVM full time.

 

Finally, this appears to be working. Hopefully my experience can help another! (crossposted from HardOCP where I was also asking for ESX help but Bob here saved the day and has proven most patient) ;-)

 

BetaQuasi, I'll be grabbing some shorter cables ASAP - thank you! I may check the US eBay first just in case but that guy does look to ship worldwide - yay!

 

P.S. check these auctions: http://www.ebay.com/itm/Mini-SAS-36-Pin-Mini-SAS-36-Pin-SFF-8087-Data-Cable-/220964190145?_trksid=p2047675.m1851&_trkparms=aid%3D222002%26algo%3DSIC.FIT%26ao%3D1%26asc%3D110%26meid%3D3030354693275426027%26pid%3D100005%26prg%3D1063%26rk%3D3%26sd%3D270832045016%26

http://www.ebay.com/itm/New-0-6M-SFF-8087-Mini-SAS-36-Pin-to-Mini-SAS-36Pin-Sever-Raid-Card-Data-Cable-/270832045016?pt=US_Drive_Cables_dapters&hash=item3f0ed90bd8

http://www.ebay.com/itm/New-0-6M-SFF-8087-Mini-SAS-36-Pin-to-Mini-SAS-36Pin-Sever-Raid-Card-Data-Cable-/290618968254?pt=UK_Computing_Drive_Cables_Adapters&hash=item43aa3d88be

 

The last one is using the same pics as the AU site. However the first one has positive reviews and the cables are $6 w/free shipping.  :D Now I just need custom Molex for my backplanes - I have a modular Seasonic PSU but no joy on eBay. I may have to solder and shrink these myself :(

Link to comment

The working S5512GM2NR is my oldest board.  I have a newer one that I'm saving up for another Xeon to put on it and I will check to see if things have changed.  My post was somewhat from memory.  My Lian Li V2000 case front USB connector won't reach the back MB header so I know I pluged it into the front header.  I could believe that the USB A ports on MB could be separated. I didn't test it by switching my unRAID USB boot device and my ESXi USB boot device to test.  The fact that I've been able to save changes make me think they are on the same controller but maybe not.

Link to comment

Well currently system is up and running with a 3tb parity and a 3tb data drive. I have a 1tb drive I stupidly pulled from one of my other servers and upgraded to a 2tb - should have putti straight to this one. I'll do some SMART testing on these drives and then begin moving data over. Still need to buy some licenses, was so focused on getting the software up I forgot to note the GUIDs!

 

Yet to do is figure out how to setup a script to shutdown the VM properly should the system lose lower. Ought to put it on a UPS too :-)

 

Appreciate all the help Bob and from others. The Atlas thread was invaluable and I'm still learning from it.

Link to comment

We're rocking now! Pulled 8 drives from one of my old servers, reset, built parity. Copying data over was just taking too darned long to do! I still have one more server to decommission but it's filled with 1.5tb drives so those I may copy over. I have at least 2x more 3tb drives coming and will try to have everything 2tb or bigger. I've got 8 slots or so open so room to grow for sure.

 

I'm seeing odd issues shutting down in this server that my others didn't display. This setup was copied from one of those sticks too so this is odd. Some process always seems to hang things and I have to kill the process. iStat shows an error concerning accounts for one thing and may be the culprit, I'll reinstall to see if that helps. Was nice to see I could slap drives in bays, restart just unRAID, and see them without restarting the whole box :-)

Link to comment
  • 1 month later...

Okay, so I've made some changes to my system, made some mistakes, and am getting close to making some bigger changes...

 

For starters I want to get all of my VMs off of spinners and onto SSD. I had thought at some point that SSD could act as cache for spinners but have not figured out how to do that. I purchased a 264ish Gig SSD to go along with the 90Gig I already had - figured that would be plenty! This is where I have learned a few things...

 

1) Always copy the VMs from the SSH console, always copy the VMs powered down. The file browser LIES and will not show you all files nor will it properly copy a VM. check the sizes of your VMs from the console - surprise! Yeah, they're taking up a bunch more room than you thought aren't they? My NewzNab VM was built for 170Gigs - mistake! It's not using near the full disk and in the end it was too big for my new spiffy SSD. Since I want to use that SAS controller and backplanes to setup something like NAS4FRee as a secondary high speed storage package I needed to get my VMs off of the drives currently in there. Since I cannot fit my NewzNab VM on the SSD I've been forced to stick a full sized drive into my case in front of the PSU - yuck. I'm hoping the second storage system can be seen by the ESX software when done and I can backup my VMs to it. I have a total of two VMs on the spinner and have removed the larger SSD completely. I had two other mechanical drives designated as storage I'll be removing too that will completely free that adapter for NAS4FREE.

 

2) Copying VMs is painfully slow - they're simply huge. Doing file operations on an SSD is pretty quick though, I like that! the SSH shell is Linux like but not exactly there so Google is your friend. Because of the missteps, retries, and other garbage my system has been down nearly a full day so when doing this know it will take time and maybe plan ahead better than I did for your needs from the beginning. I may be retiring my SAB\Sick\etc. Win7 VM for a dedicated Linux appliance VM too as it will likely be more stable and use less space. I'm starting to become quite stingy as to what I will let a VM use for HDD let me tell you! Linux VMs use far less resources all around too.

 

3) Reinserting the copied VMs is easy. Browse to them in the Datastore Browser, right click on the VMX, and "add to inventory". When you fire up the VM for the first time it will ask if it has been moved or copied. If like me you allow your VMs to do DHCP but nail the address on your DHCP server using MAC address the correct choice here is MOVED. If you say copied then you will get a new MAC address. This can be especially painful in Linux VMs since Linux sees this and adds the new MAC as a different adapter. Windows will also do this but I've found that Linux tends to be more painful to adjust software for the new adapter than the Windows. Obviously you won't want the previous VM and the new VM running at the same time, the MAC collision will wreak havoc, Deleting the old VM AFTER you test the new one can be done from the Datastore Browser.

 

4) If you have an ISO directory that you boot some VMs from you will want to copy this too and you will also want to redirect any VMs that you moved that were using it. Duh simple but easy to overlook IMO.

 

All of this is bringing home to me what a hassle backing up one of these servers is going to be. You cannot just add hardware to the system and have it be recognized by ESX either, in my case I had to power cycle the server for ESX to recognize new drives for the datastore. I will be having to figure out a way to backup VMs occasionally off the box I think. Backing them up on my new NAS software or to unRAID is a possibility but if the ESX drive fails that boots those you get a chicken\egg scenario for recovery that I'm not sure how best to solve. I'm interested in what others have done and are thinking. I'm hoping that the SSD proves solid but I have a ton of work invested in my VMs residing on my spinner now too!

 

Current VMs hosted on Pandora:

Minion my unRAID VM,

NewzIndex VM

an Ubuntu SQL server on 12.04 that takes like NO resources to run

a Win7 32bit junk client for SAB/Sick etc. that I think I want to dump.

 

I still want to setup a VM for OSX and Win8 but not there yet. I have failed with Win8 so far and may have to upgrade ESX to 5.1 to run it, I'm not upgrading ESX this go around just because that's one too many changes and I'm exhausted form all of this and the wiring I've just done. Hope this helps out someone - I learned a bunch and am still continuing to learn more...

Link to comment

Sounds like you setup thin provisioned drives.  When copied they become thick provisioned drives.  When I copy VMs I setup new thin provisioned drives and boot a VM that boots Acronis Disk Director and use it to copy the partitions from old drive to the new one.  For Windows VMs I usually have to make sure the new partition is active and that I have setup a MBR for a boot partition.

Link to comment

Sounds like you setup thin provisioned drives.  When copied they become thick provisioned drives.  When I copy VMs I setup new thin provisioned drives and boot a VM that boots Acronis Disk Director and use it to copy the partitions from old drive to the new one.  For Windows VMs I usually have to make sure the new partition is active and that I have setup a MBR for a boot partition.

 

Yup, that is exactly what I did! I hadn't thought about booting Acronis in a VM, that's a neat idea! Doing that I'd lose any snapshots though right? that's one thing I'm trying to hold onto at least for now. I did dump a few of the ones I knew I wouldn't need but I'm not sure it saved me much.

 

System is humming along now and seems fine after the move. I'm looking into what it will take to dump this Win7 VM and move to a turnkey Linux appliance. So far it's looking REALLY good and uses diddly for resources but I'm stuck on the configuration of the apps as I don't know how to mount my unRAId server as a share in Linux. I won't fill this thread talking about it but I'masking for help over here -> http://aceshome.com/forums/index.php?topic=20.0  The appliance really does look like a sweet deal and the Maraschino interface rocks!

 

Left to do - move the stupid server off my coffee table and into the room where I want it to live. I've yet to build infrastructure for the silly thing in there other than run CAT5 butI'm getting beyond sick of having it in my living room! :o

Link to comment

Sounds like you setup thin provisioned drives.  When copied they become thick provisioned drives.  When I copy VMs I setup new thin provisioned drives and boot a VM that boots Acronis Disk Director and use it to copy the partitions from old drive to the new one.  For Windows VMs I usually have to make sure the new partition is active and that I have setup a MBR for a boot partition.

 

Yup, that is exactly what I did! I hadn't thought about booting Acronis in a VM, that's a neat idea! Doing that I'd lose any snapshots though right? that's one thing I'm trying to hold onto at least for now. I did dump a few of the ones I knew I wouldn't need but I'm not sure it saved me much.

Correct any snapshots shouldn't work.  Since I am passing through PCI/e cards I don't get snapshot capability anyway so it doesn't matter to me.
Link to comment

Veeam Backup Free Edition is pretty fast for migrating and backing up guest.

 

I use it to migrate guests from one datastore to another or from one server to another. I just run it from my management guest on the ESXi box itself.

the only limit is you must have a real email address.

Link to comment

I noticed that last night actually, snapshots are a no-go if you use VT-D. Bummer!

 

I have multiple VMs that are snapshot capable, I think only one is using them though - the biggest of course. I also miss the ability to use templates but exporting and importing VMs should work for that.

 

Just last night I passed through my last M1015 to an install of NAS4Free. It was booting fine prior to my doing this and now it hangs booting even from the install CD. I really hope that this isn't some sign that passing through my 3rd controller is going to be a problem. I had to do a ton of work to free up that controller so if I cannot pass it through I will be a sad panda! I see that others use that controller with the software so not sure what's up yet. This will be my project in the coming week as time allows. I want to have somewhere to backup files that's fast, I want to play with another NAS software, and it has the ability to do ISCSI which sounds pretty interesting!

 

I will have to try your Acronis trick when I've got some time and can back up some files. I need to do my workstation and certainly some of these VMs ought to be backed up. I can store the images on my unRAID server until I get this other package working. A very nice tip and one I hadn't though of - thanks!

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.