GONDOR - My ESXi/unRAID Build

It's amazing how that upgrade bug bites you and just kind of spins you out of control...  At least it does me.    I was just going to add a couple hard drives to my array, honest.  That's all.  Now I am taking that to another extreme.

So, I decided after several happy years on 4.4.2 to upgrade to 5.x.  I also decided that my Norco 4020 was too loud with stock fans.  Recently moved and it's now somewhere that I can hear it.    That led me to build this new smaller and quieter array based on a NZXT H2 w/8 drives.  Then I started thinking what to do with the Norco?  Hence, this post.   

Components:

CPU: Intel Xeon E3-1220 Sandy Bridge $199 

Motherboard: Supermicro X9SCM-IIF-O $193

RAM: 32GB - 4x Super Talent DDR3-1333 8GB ECC Micron $208

Controllers: 2x IBM M1015 ~$100 each from ebay.  Flashed in IT mode and passed through to unRAID VM.

Controllers: 1x IBM M1015 ~$100 each from ebay.  Flashed in IR mode and used for Raid 1 of datastore drives.  Flashed in IT mode and passed through to OI/ZFS/Napp-it.

Cables: 6x .75m Forward Breakout Cable $60

Case: Norco 4020 (older one from 2009)

Power Supply: CORSAIR HX750 $140

Datastore Drives - 2x Seagate Barracuda LP ST32000542AS  in Raid 1 1x 1TB Hitachi, ZFS (4x 1TB WD Green in raidz)

Flash: 4GB Cruzer Micro for unRAID and 8GB Mushkin Mulholand for ESXi

Fans:  3x Noctua NF-P12-1300  2x ENERMAX UC-8EB 80mm Case Fan

Parity Drive: 1x 1 TB WD10EADS Green 2TB Seagate ST2000DL001

Data Drives: 5x 1 TB WD10EADS Green, 2x 2TB Seagate  Barracuda LP ST32000542AS, 1x 2TB Hitachi, 2x 500GB Samsung HD501LJ

Cache Drive: 1x 500GB Samsung HD501LJ

Total Drive Capacity: 20 drives

Hypervisor: ESXi 5.1

Guest OS: unRAID 5.0-rc11 Pro, Windows 7, Open Indiana/ZFS/napp-it

Likes: IPMI is the BOMB!  Love it.  Won't ever go back.  Headless rocks.  No booting from a USB or CD to install software in a VM, just virtually mount the ISO from ESXi and you're good to go.  PRICELESS!

Dislikes: Would like it if the Noctua fans cooled the drives better (running into the 40's during parity sync/check).  HX-750 PS fan isn't spinning, I think this is because it's not hot and automatically controlled;  but I would like to be able to kick it on to see if it would help cool drives.

Add Ons Used: preclear, MySql, unMENU, apcupsd, bwm-ng, socat, unraid_notify, cache_dirs, rsync

Future Plans: Add more Guests (OEL, WHS, Windows 2003 Server).  Add 120mm fans with higher static pressure. Will reflash 3rd M1015 and pass through to unraid if I need more than 16 drives.  Will upgrade drives to Seagate 4TB DM at some point after I am comfortable with running unRAID in a VM.

Primary Use: Data storage, Media streaming (TVs, PCs and Projector), Minecraft server, Terraria server, Backup PCs, Crashplan, Personal Lab Use, Oracle Database Server, MS Sql Server, MySQL

Here's a before picture:

Here's an after picture:

Edit 1

System is built and up and running.  ESXi installed with unRAID and Windows 7 guests.  So far, it all looks good.  IPMI is the BOMB!  So cool running headless.  I'll never go back for a server build ever again.  It works great, I love it. 

I built my own fan bracket out of wood, painted black and moved it further away from the backplane to give me more space.  I attached the bracket with self-adhesive industrial strength velcro.  My temps are higher than I would like to see, high 30's and low 40's during parity sync/check.  So, looks like the Noctua fans are no good for me with my older style 4020 case that has the SATA backplanes vertical instead of the newer cases where it's horizontal for improved air flow.  I need more static pressure.  Since I took the picture I did add foam/tape in the fan wall and any other place that air can get in the case to help pull from front to back.  I'll probably order some 120mmx38mm higher speed fans and add a fan controller of some kind.  It sure is QUIET compared to the old 80mm delta's though!

So far in my unRAID guest I have moved around about 2TB of data, swapped out a couple of disks and am currently rebuilding parity and running a preclear.  Before moving unRAID to ESXi I already backed up all of my data to another array, but I haven't lost anything yet at this point, which is all good. 

Edit 2

So I swapped out my 120mm fans for these:

3x Panaflo 120x38mm Ultra High Speed

1x NZXT Sentry Mesh Fan Controller w/ Five 30 watts Channels

Wow, these fans are amazing.  They move TONS of air and the static pressure is great.  However, they are loud.  I knew they were going to be loud, but even when using the fan controller to slow them down they have an irritating whine that drove me nuts.  So, I pulled them back out and put the Noctua fans back in.  My drives will run a little warmer during parity checks but I think I can live with that, still well within spec. 

Performance using the M1015 flashed in IR mode with two mirrored drives for my datastore was really bad so I flashed it to IT mode and passed it through to OpenIndiana so I could play with ZFS.  I pulled my 6x 1TB WD Green drives out of unRAID to use for ZFS.  I now have OI/ZFS/napp-it running on 4x 1TB WD Green drives in raidz and 2 spare drives on the shelf.  Pretty cool stuff.  Did a bunch of performance tests.  I am getting ~150 MB/s writes and ~250 MB/s reads.  Good for a datastore.  However, when I created a virtual IDE disk on ZFS for my unraid guest I only get ~15 MB/s.  That's HORRIBLE!  I can get ~35 MB/s if I don't have a cache drive at all and take the parity hit or 80 MB/s with my older Samsung 500GB (limit of drive).  I liked the idea of being able to have a parity protected cache drive, but not at 15 MB/s so back to a single disk. 

I REALLY like the Seagate 4TB DM drives A LOT.  I have used them in another build, and just put two into this box and they are preclearing now. 

I just ordered 4 Noctua NF-P12-1300 fans off ebay for $16 each with free shipping.  For that price, I went ahead and ordered a spare for a little less than buying 3 from newegg. 

Just ordered 2 80mm fans to replace those in the rear.  ENERMAX UC-8EB 80mm Case Fan

Why are you using a Sandy Bridge CPU? The Ivy Bridge part is $6 dollars more.

Why are you using a Sandy Bridge CPU? The Ivy Bridge part is $6 dollars more.

That's intentional.  In my research I've read that Sandy Bridge CPU's are more stable on the server platform.  Ivy's are fine for the desktop, but can cause issues with ESXi, Supermicro server motherboards and passthrough of controllers.  I found more than one occurrence where someone was having some weird issue, replaced their Ivy with Sandy, and the problem went away.  Maybe it was just growing pains of the newer CPU and bios related issues, I don't know.  I'll likely be memory constrained in the system anyway, not CPU, and much prefer rock solid stability.  To me it seems like Sandy will give me a greater chance at being rock solid stable. 

Here's an example:

http://lime-technology.com/forum/index.php?topic=23603.msg208733#msg208733

Is there some viable reason that an Ivy Bridge CPU would be preferred over Sandy Bridge for a ESXi build?

Ya know, maybe what I read are old issues, I don't know.  Honestly, I would prefer to use Ivy's newer technology, 22nm and 69W, if I it was as rock solid stable as Sandy Bridge.  Is it?

You are running 3xM1015's - I believe this combo will not show all drives with an ivy bridge and the IIF.  Rockdawg is a member on this board that had that exact issue and I believe several others did too.. http://lime-technology.com/forum/index.php?topic=22327

You are running 3xM1015's - I believe this combo will not show all drives with an ivy bridge and the IIF.  Rockdawg is a member on this board that had that exact issue and I believe several others did too.. http://lime-technology.com/forum/index.php?topic=22327

Yup, thanks.  That is one of the threads that I read through before and actually had me change my HW from the Ivy I originally had on my list to a Sandy.

I went ahead and ordered my CPU, memory and MB yesterday.  Unfortunately Superbiiz went out of stock on the MB just before I went to order. 

So I ordered the MB from PCNation.  Never ordered from them before, hopefully all goes well.

Got my M1015's ordered as well as my cables.  It's all ordered now I just wait for all my parts to arrive. 

I sure wish Superbiiz would have had my MB in stock.  I already got my CPU and Memory delivered via UPS Ground, less than 24 hours after I placed the order.    I am only a couple hours away from them in CA.

As far as the MB, it was ordered at the same time from PCNation and I'm told it will be shipped out today but I probably won't have it until Monday. 

Fans and Cables - Arrived today 

Motherboard - BAH!  Yesterday PCNation told me I would have a tracking number last night.  Nope.  Called again this morning, was told within an hour the system will update and I will get an email.  Nope.  Called this afternoon, was put and hold and then told they shipped it from a warehouse somewhere to their location in Chicago, they should get it tomorrow and then they will re-ship it out to me.  What?!?!?    This was to avoid sales tax on the purchase.  I would have preferred an option and would gladly have paid sales tax to have them ship it from the warehouse directly to me.  Their "Free 3 day shipping" turned into 8 days (6 business days) for me.  No thank you.  I cancelled my order. 

Superbiiz has the MB back in stock, so I just placed an order.  Hopefully they ship today and I get it tomorrow, that would rock! 

Sweet!  Tracking numbers show that my controllers and motherboard should be delivered tomorrow! 

Everything is put together and running great!  I updated the original post. 

Good job!!!  Always nice when a plan comes together.

As Johnm pointed out, the three M1015 problems are probably due to the Ivy's support for PCIe 3.0.  Something in the MB bios is not handling things correctly with multiple PCIe 2.0 cards.  Some specific PCIe 2.0 cards will not even be recognized either.  I have the same MB you do but am using an Ivy.  But I am only running one M1015 right now and don't expect to go to a second one, let alone a third for quite some time.  Everything I have running right now is stable so I am only making incremental changes now, one at a time.

I am using an older Asus P5Q Deluxe MB as a test bed both to learn more about ESXi and try things out before updating my main system.  But the P5Q does not support pass-through (bios non-support) and the on-board nics are useless.  So I had to make some adaptations to make it useful.  I do expect to get another X9SCM-IIF-0 MB, processor and memory down the road.  I do like having backups on-hand for everything running in production mode.

While doing some testing using two Intel nics I noticed that one of them was not being recognized properly by ESXi.  The symptoms were vmnic #s that were out of sequence, and in my case I was also missing vmnic0.  Pulling the second NIC card out would cause vmnic0 to reappear.  I found the below post that explained how to clean this up.

http://www.virten.net/2012/09/esxi-change-vmnic-vmhba-numbering/

Also check VMware's knowledge base article 1971044 which for some reason I cannot pull up tonight.  (I had it bookmarked, but it will not come up even via Google - something is either down or got changed)

This was happening on my P5Q MB test bed.  The same thing can also happen with HBAs.  This is not the cause of the three M1015 incompatibility problem.  But when you start pulling PCIe cards in and out, ESXi does not handle things very gracefully.  It can make a mess of your already installed vm's.  I would recommend taking note of your hardware virtual mappings before making any hardware changes.  VMware's recommended procedure when installing/removing cards is to re-install ESXi.  This is just something we need to be aware of.

Good job!!!  Always nice when a plan comes together.

 

As Johnm pointed out, the three M1015 problems are probably due to the Ivy's support for PCIe 3.0.  Something in the MB bios is not handling things correctly with multiple PCIe 2.0 cards.  Some specific PCIe 2.0 cards will not even be recognized either.  I have the same MB you do but am using an Ivy.  But I am only running one M1015 right now and don't expect to go to a second one, let alone a third for quite some time.  Everything I have running right now is stable so I am only making incremental changes now, one at a time.

Me too!!!  I might have been fine with an Ivy but went with the Sandy just to be safe, since it's more proven and I just wanted stability.  Knock on wood... stable so far!  If I ever run out of CPU (or should I say "when I run out of CPU")... maybe I'll buy an Ivy down the road and see what happens.   

I would recommend taking note of your hardware virtual mappings before making any hardware changes.  VMware's recommended procedure when installing/removing cards is to re-install ESXi.  This is just something we need to be aware of.

Great advice, I didn't know that.  thanks!

I just placed an order to replace my Noctua 120mm fans that aren't cutting it for me:

3x Panaflo 120x38mm Ultra High Speed

1x NZXT Sentry Mesh Fan Controller w/ Five 30 watts Channels

The Panaflo's are 120mm x 38mm so they are fat and rated 114.7 CFM @ 2750 RPM, 45.5 dBA, 6.12w, 510 mA, 12v DC.  They move a lot of air, and should have better static pressure than the Noctua's.  I went with the controller so I can dial them in and hopefully find a good balance between noise and cooling.  We'll see!

Hmmm... so, as I am sitting here watching Windows 7 SP1 be applied to one of my VM's... I am thinking this would run a lot faster if my datastore was on a SSD.  Thought I wasn't going to do that for a while....  Hmmm.... 

Hmmm... so, as I am sitting here watching Windows 7 SP1 be applied to one of my VM's... I am thinking this would run a lot faster if my datastore was on a SSD.  Thought I wasn't going to do that for a while....  Hmmm....

I've had the same thoughts, but have since backed away from them.  I think they are still too expensive for the storage size you get.  And the life expectancy of them is another question mark.  All the SSDs you see on sale at Newegg are MLC (multi-level cell).  The ones that will really have a long life are SLC (single-level cell).  MLC is much higher density than SLC and a whole lot cheaper to build.  But the individual cells are smaller and wear out much quicker.  I hear that SLC will outlast MLC by a factor of ten, but boy are they expensive.

On top of that ESXi does not have TRIM support built into their system according to Johnm (I'm no ESXi expert, but I do read a lot ...) that helps to minimize SSD wear.  Windows 7 & 8 do have support built into them to detect when they are accessing an SSD, but that is lost in a virtual environment.  These devices have a finite number of write cycles they can handle.  Yes they have substantial spare cells built into them and they depend highly on error correction (ECC) as well having some very sophisticated controllers.  But I couldn't tell you which manufacturer has the better controller.  Its all a crap shoot to me.

http://en.wikipedia.org/wiki/Solid-state_drive  (way too much information here)

A spinner on the other hand has no such write characteristics.  However the large capacity drives (3-4TB) are also very dependent on ECC.  Ever see that video of the guy causing disk errors by screaming at his disk?  The densities are just that high.  It doesn't take much to throw them off track.  And yes I am now using a couple of 3TB drives in the array.  But I do keep the array fully backed up onto a complete set of backup disks.

For datastores I prefer smaller drives (300-500GB) depending on the OS and the application needs.  I am not using RDM, as I prefer ease of backup for my vm's so I am using vmdks for everything except for unRAID which has the M1015 passed through to it.  Out of curiosity, how are you backing up your vms?  I am using ghettoVCB with a front-end written in python (I used ghettoUI and heavily modified it for my needs)

I will sit this one out for now, although I think they are great for laptops.  So if you absolutely need that fast bootup speed or to shave a few seconds off here and there, then go for it.

I'm not backing up my VMs yet.    That's on my to do list.  For now my datastore disks are mirrored, so I'm protected from a hard drive failure.  I know that's not a backup, and I am not protected from a failure caused by my own stupidity.    I'll look into ghettoVCB, thanks.

I know SSD's are expensive and can be fairly unreliable.  It's the best single upgrade I have ever done to my primary desktop in terms of "Wow... I felt that performance improvement".  I am using an Intel X25-M with MLC and have been happy for 4+ years with it so far.  Maybe it's overkill for my ESXi server.  I'll probably wait for a while and see how I like the performance of my VMs.

Now I'm reading about using ZFS as a datastore.  So many choices.  I am really enjoying ESXi so far. 

I'm not backing up my VMs yet.    That's on my to do list.  For now my datastore disks are mirrored, so I'm protected from a hard drive failure.  I know that's not a backup, and I am not protected from a failure caused by my own stupidity.    I'll look into ghettoVCB, thanks.

I am not using Lam's most recent update to ghettoVCB.  ghettoVCB is a bash script.  I took his previous version and reformatted it, just so I could read and understand it.  As a programmer, I detest poorly formatted code.  However, in his defense that may be due to the editor he is using.  I prefer to use Notepad++.

I did make a small change to the restore script.  I did not want it to automatically re-register a restored vm.  There are some naming issues that needed to be addressed there.  At some point, I will be updating to ESXi 5.1 and will download his latest version.  Otherwise I think it is a pretty well written script.

VMware made it difficult for "free" users of ESXi to use other free backup software such Veeam.  They made the api to ESXI read-only effectively making it useless.  PowerCLI will not work either.  The only solution available is to purchase a minimal license for ESXi such as Essentials for $500+.  No thanks, I'll roll my own ...

I know SSD's are expensive and can be fairly unreliable.  It's the best single upgrade I have ever done to my primary desktop in terms of "Wow... I felt that performance improvement".  I am using an Intel X25-M with MLC and have been happy for 4+ years with it so far.  Maybe it's overkill for my ESXi server.  I'll probably wait for a while and see how I like the performance of my VMs.

As for desktop performance, the best investment I ever made was to get good defrag software and have sufficent RAM.  I use PerfectDisk and am very happy with the results.

Now I'm reading about using ZFS as a datastore.  So many choices.  I am really enjoying ESXi so far. 

That is the biggest problem with ESXi.  It allows you to do just about anything you want, limited only by your time and money.

I downloaded and started using Notepad++, it's cool, thanks for that. 

Backups - Played around with ghettoVCB and got some backups working to my other array via NFS.    Windows VM's will backup hot, but unRAID fails after 15 min on snapshot creation, it will backup cold.  I'll play with that later.  It's not really a big deal.

Boot - I also setup a vmdk to boot unRAID from instead of plop so that makes it boot much faster. 

Patching - I went ahead and applied the latest ESXi 5.1 patch too.  Pretty painless.

Health Status Issue - I don't see anything in the vSphere client for Health Status.  I want to see CPU/MB temps but everything is listed as "? Unknown".  Anyone have any ideas for this?  Why doesn't anything show?  Any other way to get this information?  Thanks!

Datastore Performance - I posted the below info in the Atlas thread, but I am posting it here too.  There isn't really a good forum to post ESXi related questions in.  Some things are build related, but not specific to any kind of hardware.  Wasn't sure where to put it.  Maybe Mods could start an ESXi sub-forum?

ESXi datastore performance testing - What's a good way to test and measure performance?

I currently have 2 datastores:

M1015 Raid1 - 2x Seagate Barracuda LP 2 TB 5900RPM SATA 3 GB/s 32 MB Cache

NFS over 1GB network to unRAID with a cache disk

I ran a couple of quick tests using dd like this:

dd if=/dev/zero bs=1024 count=1000000 of=<datastore>/testfile

dd if=<datastore>/testfile of=/dev/null bs=1024k

My quick results for a 1GB file:

Datastore    Write MB/s    Read MB/s

Local Raid1      2.9          5.2

NFS share        5.5        12.2

I want to test some other datastore configurations like a local drive connected to the MB, a SSD, and will probably setup ZFS/OI/napp-it with 4 drives.

What is a good way to test datastore performance?  Is dd good or is there something better?  Thanks!

 

Health Status Issue - I don't see anything in the vSphere client for Health Status.  I want to see CPU/MB temps but everything is listed as "? Unknown".  Anyone have any ideas for this?  Why doesn't anything show?  Any other way to get this information?  Thanks!

I am still on ESXi 5.0U1 and my Health Status has been working fine.

This may be a 5.1 issue.  A quick search turned this up.

http://communities.vmware.com/message/2113736

 

Health Status Issue - I don't see anything in the vSphere client for Health Status.  I want to see CPU/MB temps but everything is listed as "? Unknown".  Anyone have any ideas for this?  Why doesn't anything show?  Any other way to get this information?  Thanks!

 

I am still on ESXi 5.0U1 and my Health Status has been working fine.

 

This may be a 5.1 issue.  A quick search turned this up.

 

http://communities.vmware.com/message/2113736

Wow.  Feeling kind of dumb now.  I actually read that earlier, but only made it to the 4th post and thought that was the end of the thread.      Duh! 

Hey Kegler - Where did you put your backup script so that it would persist?  I made a directory on /tools but that didn't survive a reboot.  Right now I have the backup script in one of my datastores.  Also, I noticed that there is no cron configured by default, do you have your backups automatically scheduled?  Thanks!

Nevermind, I think I got the backups all figured out.  Got the scripts stored in a datastore and have added the job to cron.  I think it will persist a reboot, we'll see.

I followed instructions here:

http://blog.waldrondigital.com/2012/09/14/modifying-ghettovcb-to-run-on-vmware-5-1/

UPDATE - Yay, it worked.  Cron entry persisted through a reboot so my backups are now good to go!