Andromeda: My Virtualized unRAID server

This UCD post, or at least the start of it, has been a long time coming.  Between building the machine, testing it, moving it, and wiring everything so that it is in my laundry room it has taken me near on 2 months total.  I probably could have gotten it done faster but frankly I wanted to get everything working correctly before I put the server in its final resting place.  It is quite heavy when loaded full so I don't want to be moving it if I don't have to.

OS at time of building: 5.0b6a

OS currently being run: 5.0rc16c

CPU: Intel Core 2 Quad Q9300

Motherboard: SUPERMICRO MBD-C2SBC-Q-O

RAM: 10GB total, 2x4GB of Patriot Signature 4GB and 2x1GB of OCZ Platinum Revision 2

RAM: 12GB total, 2x4GB of Patriot Signature 4GB and 2x2GB of generic RAM

Case: Norco 4220

Drive Cage(s): NA

Power Supply: CORSAIR Enthusiast Series TX650 V2

Power Supply: SeaSonic SS-750KM3 750W

SATA Expansion Card(s): SUPERMICRO AOC-SASLP-MV8 x2

SATA Expansion Card(s): IBM ServerRAID M1015

SAS Expander Card:Chenbro CK12803 Expander

RAID5 Card:Areca ARC-1222

Cables: 4x SFF-8087 to SFF-8087 cables for the SASLP cards, 1x Reverse Breakout cable

Cables: 1x SFF-8087 to SFF-8087 from the M1015 to the Expander -> 4x SFF-8087 to SFF-8087 from the Expander to the backplanes, 1x SFF-8087 to SFF-8087 from the ARC-1222 to the backplane

Fans: The standard fans for the midwall plate and I replaced the back fans with Antec TriCool

Fans: I replaced the midwall and put in 120mm fans. I replaced the back fans with Antec TriCool

Parity Drive: Hitachi Deskstar 7K2000

Parity Drive: Seagate ST4000DM000 4TB

Data Drives: 2x2TB Hitachi 7K2000, 1x2TB Seagate, 2x1.5TB Seagate, 2x1TB Seagate, 3x1TB Samsung, 1x1TB Western Digital, 1x750GB Seagate 7200RPM, 1x500GB Western Digital

Data Drives: 1x3TB Seagate, 2x2TB Hitachi, 4x2TB Seagate, 2x1.5TB Seagate, 2x1TB Seagate, 2x1TB Samsung, 1x1TB Western Digital

Cache Drive: 1x1TB Western Digital

Total Drive Capacity: 23TB

Primary Use: Media storeage (Blu-Ray, DVD, TV, Music), Crashplan Backup, Random file storage

Likes: No longer in the same room as my TV

Dislikes: None so far really

Add Ons Used: SABnzbd, SickBeard, Transmisson, Crashplan, Powerdown, Shell-in-a-box

Future Plans: APC Shutdown working since moving to Virtualization, Printer working again

Boot (peak): TBD

Idle (avg): TBD

Active (avg): TBD

Light use (avg): TBD

PHOTOS AVAILABLE HERE

I won't reiterate what Johnm put it in his post about setting up ESXi.  I am using all standard parts for unRAID and since I am using SASLP cards I had to use the hack to get them to work.

I switched to using the M1015 card and the Expander instead of using the the SASLP cards.  The benefit of this was not having to use the ESXi hack and using the x8 PCI-e of the card to feed the Chenbro Expander.

I do have an XP VM up and running on this machine. I now have a Windows 7 VM up and running on this box.  I have a USB PCI card passed through to the VM.  I have attached a USB BluRay drive (still have the internal one) to it for now.  I use the VM for my BluRay and DVD backup for the most part.  I have a 60GB OZC SSD for the XP VM The ESXi datastore is now a RAID5 of 4x Seagate ST1000DM003 1TB drives off the ARC-122 (with BBU) and a Slim BluRay drive in the slot of the 4220.  The one I linked to is an external but I pulled it out, added an adapter, and it worked perfectly in 4220.  AnyDVD+Clone DVD work great for my DVD backups and MakeMKV works a treat for the BluRay's.

I also have a CentOS based VM running right now to interface with my CyberPower UPS I hooked up.  CyperPower has a PowerPanel Business Edition VM that is for managing there UPS's.  After a little tweaking to get the .ovf install on my ESXi 4.1 box everything is working fine.  I am still playing with the settings for the PowerPanel VM but it is looking like this is going to work a treat.

I added an ubunbtu server VM to the box and have it running xbmc minified.  The xbmc instance is based on the frodo branch as that is what I run on the HTPC's around the house.  The server acts as a web interface when needed and also scans for any new content added via sickbeard.

If there is any information in particular that anyone would like to know about the build feel free to ask.

mother board didnt care you went over 8gb of ram? i see its listed at max 8

mother board didnt care you went over 8gb of ram? i see its listed at max 8

Nope, not in the slightest.  I installed the 8GB (2x4GB sticks) and ran memtest for 24 hours on that.  It all passed with flying colors, so I figured what the hell and through the other 2GB in.  I ran memtest for another 24 hours to make sure everything was playing nice together and everything was good.

I can 6GB passed to unRAID and 2 for my XP VM.  The reason so much for unRAID is that I run SABnzbd, SickBeard, and Transmission on the unRAID server itself.

mother board didnt care you went over 8gb of ram? i see its listed at max 8

Nope, not in the slightest.  I installed the 8GB (2x4GB sticks) and ran memtest for 24 hours on that.  It all passed with flying colors, so I figured what the hell and through the other 2GB in.  I ran memtest for another 24 hours to make sure everything was playing nice together and everything was good.

 

I can 6GB passed to unRAID and 2 for my XP VM.  The reason so much for unRAID is that I run SABnzbd, SickBeard, and Transmission on the unRAID server itself.

Any particular reason for running them on the unRaid VM instead of the XP VM in order to offload to work from unRaid?

Any particular reason for running them on the unRaid VM instead of the XP VM in order to offload to work from unRaid?

No particular reason at this point.

I had unRAID running by itself originally and did not want another computer on, so I installed the applications on the server.  Installing them on the server also allows my room mates to get into those apps via the web interface, so they can download what they want to.

The XP VM is really only used for DVD and BluRay backup at this point.  I might eventually make it do more things, but for right now SABnzbd, Sickbeard, and Transmission are running perfectly fine on the unRAID server.

I use VMware workstation on my computer. To load a couple other legacy operating systems. I'm having issues knowing what this ESXi VM thing is? Do you need to run a full OS first, then run this program? Once you have an OS loaded up and running, how do you start another one? I'm still thinking VMware workstation, where you need to load your OS and run the VMware program to access your VM's. What is the difference between that and this ESXi? I'm very curious and since VMware has a ton of products on their site, it does get difficult trying to figure it all out.

ESXi is an operating system of itself... kind of.  ESXi is not like VMWare Fusion, VirtualBox, Parallels Desktop, or VMWare Workstation as all those require an OS (Linux, Windows, Mac OS X) to be installed as the base.  In the case of ESXi it is the base that you build up from.  It is very small (I run it from a USB drive just like unRAID).  You do need a windows machine to install the vSphere client on so you can manage and admin the ESXi box.

So one more thing can be knocked off the list.

I order this from newegg to get the old parallel printer working.  I shut the unRAID server down, attached everything up to the server, edited the configuration to add a USB device to unRAID, start the unRAID VM up.

I only had to really change one thing once unRAID was started up.  I had to change the p910nd printer port variable from lp0 (what it was originally when running from a parallel connection) to usblp0 for the printing to work.  After that small change everything is once again good to go.

Only thing left on the list is to get the APC unit working with all the VM's.  I could do a USB pass through with it also, but I would prefer to find a better way.

mother board didnt care you went over 8gb of ram? i see its listed at max 8

Nope, not in the slightest.  I installed the 8GB (2x4GB sticks) and ran memtest for 24 hours on that.  It all passed with flying colors, so I figured what the hell and through the other 2GB in.  I ran memtest for another 24 hours to make sure everything was playing nice together and everything was good.

 

I can 6GB passed to unRAID and 2 for my XP VM.  The reason so much for unRAID is that I run SABnzbd, SickBeard, and Transmission on the unRAID server itself.

thats great havnt seen to many motherboards like that

i noticed that you have a slim bluray drive on your norco 4220. what do you use the drive for?

thanks.

i noticed that you have a slim bluray drive on your norco 4220. what do you use the drive for?

 

thanks.

I have it set up in my XP VM so I can rip my DVD and Blu-Ray collection.

my system spec is

case - norco 4220

mb - supermicro x9scm

raid card - 2 supermicro saslp

parity - 3tb wd green 

cache - 3tb wd green 

data - 3tb wd green

2*4 kingston unregistered ecc ram

sony slim bluray drive. my objective with the bluray drive is to rip 3d bluray movies.

if i want to go exsi route,

1. how much data drive can i install? my target is maximum 20 hdd. your spec doesnt mention how many raid card you have. what is your recommended optimal/cheap solution?

2. do i need to have os hdd? is it possible to put everything os and exsi on usb drive? how big usb drive you recommend? where do you put your os?

3. im thingking exsi with unraid and ubuntu(maybe other smaller linux distro). maybe not linux(i havent found a way to backup 3d bluray using linux). maybe slipstream win7 or windows embedded. what is your thoughts.

Im interested to know the full spec of your system.

thanks.

my system spec is

 

case - norco 4220

mb - supermicro x9scm

raid card - 2 supermicro saslp

parity - 3tb wd green 

cache - 3tb wd green 

data - 3tb wd green

2*4 kingston unregistered ecc ram

sony slim bluray drive. my objective with the bluray drive is to rip 3d bluray movies.

 

if i want to go exsi route,

1. how much data drive can i install? my target is maximum 20 hdd. your spec doesnt mention how many raid card you have. what is your recommended optimal/cheap solution?

I have 2 of the SASLP cards in my system and therefore have 16HDD available for us in unRAID.  The SASLP cards will work perfectly fine once you apply a little hack to make then work with ESXi. Your board has more PCIe slots available so you should be able to get 20 drives into unRAID without issue, just need 3 SASLP cards. Use the 3 SASLP cards for the 20 hot swap bays and then run any other drives you might want to add to the system (SSD VM drive) from the SATA III motherboard ports.

 

2. do i need to have os hdd? is it possible to put everything os and exsi on usb drive? how big usb drive you recommend? where do you put your os?

ESXi and unRAID can both run from a flash drive but will need to run from a different flash drive each.  I highly suggest getting different brand flash drives for each OS.  I have ESXi and unRAID on a 2GB and a 4GB flash drive.

 

3. im thingking exsi with unraid and ubuntu(maybe other smaller linux distro). maybe not linux(i havent found a way to backup 3d bluray using linux). maybe slipstream win7 or windows embedded. what is your thoughts.

I highly suggest an SSD of some sort for the VM's in your system.  The random access times of an SSD are much better and will make the VM's seem more "snappy."

 

Im interested to know the full spec of your system.

 

thanks.

My full list of specs are in the first post.  The one thing I did not see listed above was what processor you have with your system.  That will make a difference.  You should also looks through Johnm's Atlas post on running ESXi and read everything you can from there.

Thanks for the reply.

My cpu is elcheapo i2100. since the only thing im planning on using is ripping bluray, maybe the cpu is enough.

Thanks for the reply.

 

My cpu is elcheapo i2100. since the only thing im planning on using is ripping bluray, maybe the cpu is enough.

the i3-2100 will not work for an ESXi server.  The board supports VT-d but your processor does not.  VT-d will allow you to passthrough the SASLP cards for us in unRAID.

*smack in the head*

unless i change the processor to xeon(vt-d) esxi with unraid is impossible right?

Can i conclude that, as of right now, with my current config, the bluray drive is unseable?

*smack in the head*

Thanks.

unless i change the processor to xeon(vt-d) esxi with unraid is impossible right?

correct

Can i conclude that, as of right now, with my current config, the bluray drive is unseable?

correct

Andromeda will be getting an update/upgrade this weekend.  It has been a while coming and I have had most of the parts for a while now... just not the time.

I will take before and after pictures but the general breakdown of the update is:

1. 120mm fan wall

2. 750W Seasonic fully modular PSU

3. Moving from the 2 SASLP cards that are in passthrough to an M1015 flashed to IT mode

4. One Chenbro expander to run 4 rows of the backplane on my case

5. One ARC-1222 with BBU running a RAID5 in the top backplane row for my datastore

6. 4 of the Seagate ST1000DM003 1TB drives (these drives are some of the fastest single drives I have tested).

7. Likely going to try and push the board to 12GB of RAM from the 10GB it has in it already. The 10GB is over the specified max but i have had no problems with it this way.

After the main hardware upgrade is complete I will be updating some of my disks in my unRAID array.  I have 2 Seagate 4TB drives to put into service, a 3TB seagate to put in place and some other drives to move around.

I thought about getting a new motherboard (X9SCM-IIF, processor and RAM) but did not want to outlay for that right now and did not really need to upgrade those parts at this time.

I will attempt to take pictures of the before and after so I can put them up here so everyone can see the transformation.

Andromeda will be getting an update/upgrade this weekend.  It has been a while coming and I have had most of the parts for a while now... just not the time.

 

I will take before and after pictures but the general breakdown of the update is:

1. 120mm fan wall

2. 750W Seasonic fully modular PSU

3. Moving from the 2 SASLP cards that are in passthrough to an M1015 flashed to IT mode

4. One Chenbro expander to run 4 rows of the backplane on my case

5. One ARC-1222 with BBU running a RAID5 in the top backplane row for my datastore

6. 4 of the Seagate ST1000DM003 1TB drives (these drives are some of the fastest single drives I have tested).

7. Likely going to try and push the board to 12GB of RAM from the 10GB it has in it already. The 10GB is over the specified max but i have had no problems with it this way.

 

After the main hardware upgrade is complete I will be updating some of my disks in my unRAID array.  I have 2 Seagate 4TB drives to put into service, a 3TB seagate to put in place and some other drives to move around.

 

I thought about getting a new motherboard (X9SCM-IIF, processor and RAM) but did not want to outlay for that right now and did not really need to upgrade those parts at this time.

 

I will attempt to take pictures of the before and after so I can put them up here so everyone can see the transformation.

So most everything went as expected... except one thing.

1. The fan wall went in without issue and is working great

2. The Seasonic PSU is GREAT and the fact that it is fully modular is very nice

3. The removal of the SASLP cards to the M1015 without issue

4. The chenbro expander is hooked to the M1015 and that is working perfectly

5&6. The setup on this was a little interesting.  I had the RAID5 created but the foreground init of the RAID was going to take 24+ hours.  I did some reading and found that with the BBU attached this can sometimes happen.  I powered down, disconnected the BBU and the init went much faster.  The init of the RAID volume finished in about 2 hours (still longer than I would have liked but fast enough).  I ended up with 2.75(ish) TB from the 4 Seagate 1TB drive in the RAID5

7. I did put 12 GB of RAM in the box. there are 2 4GB sticks and 2 2GB sticks.  They are running at DDR-667 speed and the memtest passed without issue.

The one issue I ran into... and that is really annoying is that I could not get the ESXi updated to 5.1.  The standard ESXi image would boot to a solid green screen.  I tried to create a customized 5.1 install for my board and the areca arc-1222 card and that also failed.  I could not even get to the install part.  I ended up just using my current ESXi 4.1 install.  This is working fine for me and I don't really care so long as my new components are working.  The downside of not being able to update to ESXi 5.1 is that I can not actually use the entire RAID5 volume for datastore.  Datastores in 4.1 seem to be limited to 2TB in size.  I went just shy of that mark when creating the datastore and everything is working as expected.  I will just deal with not being able to use all the RAID5 for now and when I finally feel the need/need to use the extra space I will just update the motherboard/RAM/CPU to an X9SCM-IIF and be done with it.

I did not get before pics but I will for sure have the after pics now that everything is back up and running.  I need to do a few last things and make sure everything is ready to go before putting it back on its shelf.

So how fast is the RAID5 datastore with VMs?

I'm using a 256GB Samsung PM 840 PRO and it's lightning fast. 

XP PRO boots up to desktop in less then 10 seconds.

Centos 6.4 in less then 7 seconds (no gui).

Since I'm always concerned of a single drive failure. I'll be rsyncing it to a 3TB spinner.

So how fast is the RAID5 datastore with VMs?

 

I'm using a 256GB Samsung PM 840 PRO and it's lightning fast. 

XP PRO boots up to desktop in less then 10 seconds.

Centos 6.4 in less then 7 seconds (no gui).

 

Since I'm always concerned of a single drive failure. I'll be rsyncing it to a 3TB spinner.

I will have to do some testing but boot up of a Windows 7 VM that is running on the box is quick.  I had the previous ESXi box running off an SSD and there is little to no difference that I can tell between the 2.  The RAID5 vs SSD seems to be pretty much comparible at this point.

If you have any specific test that you want me to run from within the Windows 7 VM please let me know and I will try to get those done and posted here.

So how fast is the RAID5 datastore with VMs?

 

I'm using a 256GB Samsung PM 840 PRO and it's lightning fast. 

XP PRO boots up to desktop in less then 10 seconds.

Centos 6.4 in less then 7 seconds (no gui).

 

Since I'm always concerned of a single drive failure. I'll be rsyncing it to a 3TB spinner.

I will have to do some testing but boot up of a Windows 7 VM that is running on the box is quick.  I had the previous ESXi box running off an SSD and there is little to no difference that I can tell between the 2.  The RAID5 vs SSD seems to be pretty much comparible at this point.

 

If you have any specific test that you want me to run from within the Windows 7 VM please let me know and I will try to get those done and posted here.

It's more of a usable perception. 

I ran my XP PRO under vmware workstation for years on a pair of mirroed 10K SAS drives. It was OK.

Over the past years, I've swapped out all my laptop drives to SSDs (except for my DJ latops)

I once moved the XP PRO vmware instance to an SSD and it felt like a native machine. Unfortunately, there was some issues in that Centos Kernel that would have problems with the SSD so I had to move back.

The instant access of SSD's has spoiled me.  The only time I use spinners is for archiving or if I'm doing constant writing and building ISO or VMware Images.

I've been curious how well the Areca RAID5 performs and if it's worth my effort.

I don't have any specific tests at the moment.

I suppose if you build a decently sized .vmdk, boot unraid and do a DD read and write it could provide some answers.

Probably not worth your time unless you have it running like this already.

I have 250GB assigned to the Windows 7 VM so I can do some file copy tests across from unRAID.  I can grab some of the BluRay rips.

If you want to send me a PM about stuff you want me to try let me know.  I have nothing else set up on the machine right now so  creating another VM to use 1TB of the RAID5 would not be a huge issue. I just don't really know what tests to run.  I can delete the testing VM when testing is done so no issues there.

Have you done any speed tests yet with the Areca array?

VM Boot times would be more interesting to me than copy speeds from UnRAID (which I'd think would be limited by the single disk that was "feeding" the copy than by the Areca write speeds).

By the way, FWIW I've read that the C2SBC-Q will support 16GB of RAM (4 4GB modules), although I do NOT have this board and thus can't confirm it.  I do know that many SuperMicro boards will support higher density modules than their very conservative specifications indicate.  [i have 8GB in an X7SPA-H-D525-O that claims a 4GB max]

Since you've already got a couple 4GB modules installed, I'd say it's very likely it will work with 4 of them