Hardware Build and ESXi install.Hardware install notes:
Original Hardware unboxing
The 650Watt Corsair Power Supply pictured was not going to cut it. I used the spare 750watt Seasonic I had from an earlier sale. I just need to swap it out from a workstation and put the 650 into it. In addition, the Seasonic is gold certified, That is a bonus for an always on PC.
The first step I did was assemble everything into the Norco as if i was going to install unRAID
I installed the Motherboard, RAM, CPU, 1 of the MV8's and the Power Supply for testing.
Current Build Photo with 32GB Ram, E-1240, 2x M1015, Expander, Corsair Pro SSD's, and custom power cables
I don't think I need to go into detail here.
I'll assume you can assemble the hardware.
Plug in the power and Ethernet cables to both the IPMI and
[Use LAN1 For ESXi, LAN2 is for baremetal unRAID]
After this step, I stuck the ESXi flashdrive into the internal USB
Yes, it is still blank. this is for the Bios config step.IPMISetup:
At this point go to ftp.supermicro.com/utility/IPMIView/ and download IPMI View.
If you have a monitor and keyboard installed and you dont plan to use IPMI, skip ahead to the BIOS configuration.
Start the IPMIView software and click the Magnifying glass icon to have it auto detect your new server. go ahead and add it to your "IPMI Domain"
Go ahead and login to your new server. the default login is "ADMIN" pass "ADMIN"
Start the server under IMPI Device TAB and open the KVM console in the KVM TAB.Raid Card Bios Settings
As the PC starts to post, watch for the Raid card BIOS.
When it starts detecting drives on the raid card, start pushing "ctrl m" (For mv8 anyways)
Disable INT 13hOptional
Under staggered spin up: set spin up groups to lower the hit on your power at boot.
Exit and save.
If you have more then one HBA card. You should now swap them and do the same thing to the next card.Bios Settings:
hit the "Del" key to enter the bios.
In the advanced tab: Processor and Clock options
Enable "Intel Virtualization Technology"
In the advanced tab: Integrated IO Configuration
In the advanced tab: PCIe/PCI/PnP Configuration
Set PCI ROM Priority to "EFI Compatible ROM"(NOTE: for Ver 2.0a BIOS this is replaced with "Disable OPROM for slots 7&6" set them to "Disabled")
In the advanced tab:IDE/SATA Configuration
SATA Mode = ACHI
Set staggered spin-up and Hot Plug for all drives if you want.
BOOT: Boot Options Priority.
Select your ESXi Flashdrive.
And last (optional
IPMI: BMC Network config
Set a static IP for your IPMI
At this point you should save setting and exit.
Manually power off the server if You were using IPMI and you changed the IP in the Bios.
In IPMIView, modify your your IP to reflect the new one you just changed your IPMI IP to.Basic Pretesting:This step is not really a step. It was something I did to test my hardware.
it is optional, but it made me feel better..
I pulled my unRAID flash drive from my second unRAID server.
I placed the unRAID Flash drive into the ESXi box.
I booted with the unRAID flash drive and ran several cycles of memtest (Note With this hardware you will need to upgrade the memtest that comes with unraid. SEE HERE
After that passed, I felt all warm and fuzzy...Installing ESXi:NOTE* These instructions are for 4.1.0.
Since this thread was created, ESXi5.0 has been released.
the instructions are ALMOST the same. these instructions should get you through the ESXi 5.0 setup also.
If there is a major change or a part that is confusing, let me know and I'll update this thread.
(Get a screen shot if you can)
I wont pretend to be an expert at ESXi.
Infact, even though I use it at work, all I know is from google and trial and error..
For this build, we will be installing ESXi 4.1.0 to a flashdrive.
When you download ESXi from VMware, it is an ISO image.
I decided for ease of install, I would just burn it to a CD.
You can create a flash drive install to install from flash drive if you wish.
After basic google-fu and realizing i didn't have another flash drive laying about, the CD install won. Besides, my RPC-4224 came with a free SATA DVD?! It is karma.[Edit: You could also use the "Virtual Media" option in the IMPI and mount the ESXi ISO for the install if you don't have a SATA DVD] Prep for ESXi Intall:
At this point, if you have not already, Download your free copy of ESXi and register it to get a free Serial number.http://www.vmware.com/products/vsphere-hypervisor/overview.html?ClickID=bledqduu6egnnqg6nl6vsdzelzklyvkfzgne
Burn the ISO to CD (no need to waste a DVD)[or use the USB Install Method]REMOVE ALL DRIVES!Remove/Unplug all Hard Disks and Flash Drives from the server!
During install, ESXi will erase ALL drives it sees!!Don't say I didn't warn you.
Install your ESX flash drive into the internal USB header. You can use an external header if you like. It makes more sense to put it inside your case so you have access to your unRAID drive.
Go ahead and plug a DVD drive into one of your internal Sata ports (USB CD should work also).
You should have no drives in the drive bays so it is OK to leave the top off for now.ESXi Install:
Power on your server.
Start hitting the F11 key once you get the supermicro splash screen. This is to bring up a boot menu.
Select your CD Drive.
Welcome Screen: > (Enter) Install
EULA Screen: > (F11) Accept
Select A Disk: > Select Your Flash Drive (IT SHOULD BE YOUR ONLY DRIVE. IF NOT, STOP! SEE ABOVE!!)
Confirm Install: > (F11) Install
Wait for the install. It should take 10-15 minutes.
Complete: > (Enter) Reboot!
Assuming you configured the ESXi Flash drive as your first boot device, you should now boot into ESXiConfiguring ESXi Console:
On our first boot into ESXi, We Should be welcomed with this screen.
If you see a grey screen with red text flash past and you are now sitting at an error code, Chances are you have an incompatible NIC. (We wont see that with this build.)
However, the issue I did have, I did not get a DHCP IP Address.
I had to move the Cable from LAN2 to LAN1.
This is after i told You to place the Ethernet cable into LAN2.
Apparently I have a newer revision of the motherboard on this build.
My last build was with a Ver 1.0.
This Board is a 1.0b. I wonder what else has changed?
Assuming you have a DHCP server (who doesn't?), you should have an IP and it should say http://HTTP://IP_Addy/
This takes a few minutes sometimes.
This would be the web address of the server. this is where you would go to get ESXi tools (more on this later).
Lets go and set up a static IP and set the Root Password. We could do this from inside the vSphere client later, but lets see what options are in the console.
We will need to hit F2 to Customize the system.If you just hit F2 in your IPMI window, you just found the exit hotkey...
If you are using IPMI, in the top toolbar, on the left, Select "Virtual Media" and then "Virtual Keyboard".
You should now have an on-screen keyboard.
Hit the F2 on the Virtual Keyboard.
You should now have a Login ScreenYou can now close the Virtual Keyboard, We are done with it
The default Login is "root" with no Password.
Set up a password now while we are here. (optional)
Select Configure Password > Enter the new password. (you have to use a complex password)
Set up a Static IP. (optional but recommended)
Select Configuration Management Network.
Select IP Configuration.
Select Set static IP Address
Fill in your IP Address, Subnet and Gateway.You can setup IPV6 While you are here if you use it.
I do not run IPV6 at home so I skipped that.
You can modify your DNS configuration now.
It should have locked in what your DHCP server assigned when we set a static address.you can change the hostname if you wish also.
After you are finished with your changes, hit "esc" until you are greeted with a save changes page and a warning your VM Hosts will be kicked off the network.We have no hosts yet so this is OK.
Select <Y> Yes
This should bring us back to the "System Customization" Menu.One last step to do while we are here.
We are going to turn on SSH.
This allows us to telnet and use WinSCP into the server.
Select "Troubleshooting Options"
Select "Enable Remote Tech Support (SSH)" (This enables SSH on the server)Double check your settings. This screen is a bit confusing to some.
After you Enable SSH, You can <ESC> all the way back to the main screen.
You should see the static IP now.We are done with this portion of the install.You can close the IPMI window if you want.Configuring ESXi from vSphere:This is where most people get lost, VMware vSphere client is not very intuitive.
The first step is to get the vSpere Client.
Open up a web browser and put in the IP address of your ESXi box and hit enter/go/whatever makes it start..Stop!
You will now have a warning message in your browser!This is OK!
You are connecting over a secure connection with a private security certificate.
Go ahead and connect and save the certificate if it asks you (IE wont save it).
Once you get to the ESXi webpage, Download/Install the "vSphere Client"I wont hold you hand here.
Install the client.
Once you have the client installed.
Enter the IP address of your ESXi box, Admin ID, Password and hit "Login"STOP!
We we are greeted a certificate error once again.
Check "Install Certificate..."
Then Select "Ignore"
vSphere Client will now start up and give you a nag box about your license.
It will also remind us that we have no persistent storage.
OK, Lets fix the License Issue first.
Configuration > Licensed Features > Edit
Check "Assign New License Key to this Host"
Click "Enter Key" Button
Enter your License Key
You will have a Licensed ESXi server now.
Now we need to add the Datastore drives. These are the drives where we store the virtual disks and hosts along with other data for the ESXi server.
You can hotswap the drives into the server while it is on.
But for the sake of safe practice, we will shut the server down.
Right Click on the server in the top left pane > Shut Down.
Summary > Reboot
The server will nag that it is not in "Maintenance Mode"
That is OK.
It will then confirm why you are shutting down.
OK and shut down.Install your Datastore disks at this pointWe could have done this sooner, we just didn't get to it.I am going to install 1 SSD off of one of the White SATA600 (Sata3) ports and one Large mechanical Drive off of one of the Black Sata300 (Sata2) ports.
I'll eventually add the second SSD. For now, I'll hold off.
(Honestly I have some test VM's on my second one in my other ESX box I need to reclaim)
You can do what you feel is best for your need.ANY DRIVE WE ADD AND ASSIGN AS A DATASTORE DRIVE WILL BE FORMATTED!!
FOREVER LOST! THERE IS NO GOING BACK!
That is, unless it was already contains a Datastore. You can move those from ESXi to ESXi box.
Once you are done adding the drives, power up your server and restart vSphere.
Adding Datastore Drives:
In vSphere Client,
Configuration > Drives > Datastore > "Add storage"
We are adding a Disk/LUN
Select the disk you want Added to the Datastore.
NextAll Partitions / Data will be Wiped!
Name your Datastore.At Work we call them Datastore1, Datastore2, etc. At home, I name them a bit more descriptive. I like to keep the name simple for scripting later. SSD1, SSD2, 2TB1 for example
Enter a name > NextSTOP!Format "Set Block Size".... this part is critical and most people screw this up and loose all their data after they figure this out.
you have 4 Block size settings!
what you select determines the maximum size of your Virtual Drives!!
Block Size Vs. Maximum Virtual Drive size
1Meg = 256MB vDrive
2Meg = 512MB vDrive
4Meg = 1TB vDrive
8Meg = 2TB vDrive
Supposedly, there is no performance hit or loss of drive space for choosing a larger block size.
Choose wisely based on your needs.
You can not undo this without reformatting the drive.
In this case, I'm going to choose 1 meg blocks.Edit: I now think it is best to format all drives the same block size.
My SSD is small and most of my clients will be 30gigs or Smaller.
I formatted my Mechanical drives with 8Meg blocks, I am going to format my SSD's 8Meg blocks.
Choose a Block Size > Next
Confirm > Finish
Repeat if needed for each drive.
We now have our DatastoreYes, there is data on the 2tb drive (it is borrowed from another ESXi box, more on that later)Updating ESXi to the latest version.
For 4.1, See this thread > http://lime-technology.com/forum/index.php?topic=14695.msg152540#msg152540
For Version 5.0, See this thread > http://lime-technology.com/forum/index.php?topic=14695.msg169119#msg169119This Pretty much concludes the Basic ESXi setup.
We will get into more tips and tricks like pass-through as we install the VM's
If anyone sees any changes I should implement, let me know.
the new ESXi box sitting with my 2 unRAID servers. On the Floor of a spare bedroom temporarily.
Here is a crappy cellphone picture of the Servers in a Lack Rack.