Running ESXi? How do you handle data protection on your VM Server?

So I spent the last two weeks building and creating an ESXi server.  I got my unraid VM's up and running and things seem to be good.

 

I had originally purchased two 120GB SSD's for my VM's and two 1TB drives for my data stores for those VM's.  (Non-critical data).  Originally I had planned to mirror the drives so I would have some data protection.

 

Unfortunately as I got started I realized ESXi does not offer any sort of raid support. 

 

Well now one of my Datastores has failed - and as "non-critical" as I had thought that data would be - it is not important to me.  As I try to restore and save this data.....

 

How or others handling data protection on the Server side?  Is there any way to create a mirror? 

 

Anxious to hear....

You can use RAID for your datastores, you just need a hardware raid card.  I have 3 M1015's, and I am currently using 1 of them (flashed with IR bios) to mirror 2 drives.  I also have another datastore on NFS on an external unRAID array.  So I am currently protected from a single disk failure for my datastores.  I also backup my VMs.

 

My 2 datastores:

M1015 Raid1 - 2x Seagate Barracuda LP 2 TB 5900RPM SATA 3 GB/s 32 MB Cache

NFS over 1GB network to unRAID

 

I am planning on playing with ZFS/Open Indiana/nappit to create a ZFS datastore on my M1015 that I am currently using for the mirror.  The current mirror is too slow for me. 

 

My quick results for a 1GB file using dd:

Datastore    Write MB/s    Read MB/s

Local Raid1      2.9          5.2

NFS share        5.5        12.2

I only have unRAID VMs and Windows VMs so I have a script copy the USB "UNRAID" drive every day.  For Windows VMs I use Acronis True Image from inside the VMs once a day.  Then just re-setup the VM if I have datastore problems.  It only takes about 15 minutes to setup ESXi including the VMs if you write the configuration of each VM somewhere so that you can put it back.  Restoring of the image with a bootable CD of True Image is dependant on the size but is usually just minutes as well.

 

So in otherwords I don't backup the VMs themselves.

You can use RAID for your datastores, you just need a hardware raid card.  I have 3 M1015's, and I am currently using 1 of them (flashed with IR bios) to mirror 2 drives.  I also have another datastore on NFS on an external unRAID array.  So I am currently protected from a single disk failure for my datastores.  I also backup my VMs.

 

My 2 datastores:

M1015 Raid1 - 2x Seagate Barracuda LP 2 TB 5900RPM SATA 3 GB/s 32 MB Cache

NFS over 1GB network to unRAID

 

I am planning on playing with ZFS/Open Indiana/nappit to create a ZFS datastore on my M1015 that I am currently using for the mirror.  The current mirror is too slow for me. 

 

My quick results for a 1GB file using dd:

Datastore    Write MB/s    Read MB/s

Local Raid1      2.9          5.2

NFS share        5.5        12.2

 

That local raid1 data store does seem a bit slow.

you could probably improve that quite a bit with switching to the 3TB 7200 RPM Seagates that are out.

They can get up to 195MB/s on the outer tracks.

 

 

I'm planning to install a used areca arc-1222 controller for the raid1.

 

Either that or using an external raidbox such as the SANS DIGITAL TR4UTBPN 4Bay USB 3.0 / eSATA Hardware RAID 5 Tower RAID Enclosure

http://www.newegg.com/Product/Product.aspx?Item=N82E16816111149

That local raid1 data store does seem a bit slow.

you could probably improve that quite a bit with switching to the 3TB 7200 RPM Seagates that are out.

They can get up to 195MB/s on the outer tracks.

Yup, haven't gotten around to trying out some different options yet.  It's only been setup for a few days.  I just grabbed a couple drives that I had on hand and threw them in the box.

What a great idea.....

 

A quick bit of research - and I picked up this external array - http://www.newegg.com/Product/Product.aspx?Item=N82E16816856039

 

Add in this controller - http://www.newegg.com/Product/Product.aspx?Item=N82E16816124044

 

I threw in 4 1TB drives I had laying around (two were inside my machine as VM Data stores) - set the dip switches for Raid 10, plugged the array into my laptop via USB to confirm the setup - unplugged and hooked up the esata cable - powered up ESXi - and there it was.  Just added it to my datastores.

 

So far seems like a great solution. 

 

Now I just need to figure out what to do with my two SSD drives....

 

What a great idea.....

 

A quick bit of research - and I picked up this external array - http://www.newegg.com/Product/Product.aspx?Item=N82E16816856039

 

Add in this controller - http://www.newegg.com/Product/Product.aspx?Item=N82E16816124044

 

I threw in 4 1TB drives I had laying around (two were inside my machine as VM Data stores) - set the dip switches for Raid 10, plugged the array into my laptop via USB to confirm the setup - unplugged and hooked up the esata cable - powered up ESXi - and there it was.  Just added it to my datastores.

 

So far seems like a great solution. 

 

Now I just need to figure out what to do with my two SSD drives....

 

Is that controller supported by esxi native or do you need hacks?

 

I would probably use the ssd's for all but the busiest vm's. Then put in place some sort of automated backup from SSD to the external array.

 

 

What drives are you using and What speeds are you getting to the external RAID 10 array?

 

 

 

The card is supported by ESXi 5.1 without any hacks or tricks.  Just plugged it in - and ESXi saw the array as it was created.  Added it to the data store and away I went.

 

Not sure how to test the speed.  Let me know what app or what process is used to report some of these write speeds.  I would be happy to report back.  So far I don't notice anything different than before when I just had 1TB drives hooked up directly. 

 

 

The card has two ports - I see that Cineraid also make a two drive box.  Maybe I should mirror my SSD drives in that box.  But they are only 120GB. 

 

Right now I am running 3 VM's - unRaid,  Windows Server 2008, and Windows 8.    I don't anticipate needing much more. 

 

I'm not virtualizing my unRAID on ESXi, but since you asked...

 

I'm running an OpenFiler server with an Areca 1680 card running a four drive RAID10 array for my datastore.  My ESXi hosts connect to it via fiber channel, with a brocade switch.

 

Yes, this is at home!  Kinda crazy. It was an experiment to teach myself how to use fiber channel. It worked so well, I decided to keep it.  I have an extra fiber card if I do decide to virtualize my unRAID

 

And of course, nightly backups.

Using Teracopy -

 

Within the my Windows 8 VM.

Reading the Array to my SSD drive - I am getting 95-99MB/Sec.   

Writing to the Array is slower - I am getting 25-30MB/sec.

 

Update - found that moving data from unRaid to the Windows 8 drive that is on the external array  -  I am getting 48-56MB/S

ESXi is very dependent on write caching when using a RAID array for storing your datastores. Without the existence of it, performance is severely hampered. You want a card that has a BBU that will be utilized for the caching operations not to mention the "write-back" feature turned on for the array rather than "write-through". I learned all of this the hard way and still haven't ponied up for the extra cash for a proper hardware card.

Anyone aware of a good external array that supports 2.5 inch laptop drives? I've got tons of them lying around and it would be great to use some of them as an external RAID 5 datastore. I realize I could get adapters and put them in 3.5 inch bays but the small size of a 2.5 inch only external array would be nice.

 

This is the only one I can find on Newegg and it has very mixed reviews. http://www.newegg.com/Product/Product.aspx?Item=N82E16816216004

Anyone aware of a good external array that supports 2.5 inch laptop drives? I've got tons of them lying around and it would be great to use some of them as an external RAID 5 datastore. I realize I could get adapters and put them in 3.5 inch bays but the small size of a 2.5 inch only external array would be nice.

 

This is the only one I can find on Newegg and it has very mixed reviews. http://www.newegg.com/Product/Product.aspx?Item=N82E16816216004

This is for 8 drives and much more expensive and only one review but it is 5 stars.

 

 

 

SANS DIGITAL MobileSTOR MS28X JBOD Mini-SAS x 2 (2400 Gb/s mini-SAS interface) 8 Bay 2.5 tower SAS / SSD / SATA to 2 x mini-SAS (SFF8088) JBOD Storage

http://www.newegg.com/Product/Product.aspx?Item=N82E16816111186

Anyone aware of a good external array that supports 2.5 inch laptop drives? I've got tons of them lying around and it would be great to use some of them as an external RAID 5 datastore. I realize I could get adapters and put them in 3.5 inch bays but the small size of a 2.5 inch only external array would be nice.

 

This is the only one I can find on Newegg and it has very mixed reviews. http://www.newegg.com/Product/Product.aspx?Item=N82E16816216004

This is for 8 drives and much more expensive and only one review but it is 5 stars.

 

 

And it's SAS JBOD, doesn't look like it has built in RAID functionality so I'd have to buy a controller card as well. Maybe I'll just give that other one a shot.

I have this one, In fact I have two. I'm about to buy my 3rd.

SANS DIGITAL TR4UTBPN 4Bay USB 3.0 / eSATA Hardware RAID 5 Tower RAID Enclosure (no eSATA card bundled)

 

http://www.newegg.com/Product/Product.aspx?Item=N82E16816111149

 

I'm not sure using 2.5" drives is worth it at that price.

I would sell them for what I can get for them, get 3.5"s or get SSD's and possibly those 2.5" to 3.5" adapters.

Anyone aware of a good external array that supports 2.5 inch laptop drives? I've got tons of them lying around and it would be great to use some of them as an external RAID 5 datastore. I realize I could get adapters and put them in 3.5 inch bays but the small size of a 2.5 inch only external array would be nice.

 

This is the only one I can find on Newegg and it has very mixed reviews. http://www.newegg.com/Product/Product.aspx?Item=N82E16816216004

 

Addonics seems to have the same product rebadged.

I suppose if they think it's good enough to rebadge, this one might be worth it for the low cost.

http://www.addonics.com/products/cpr5sa.php

 

I had a venus AMS raid enclosure once. The cable slowly slipped out of the PSU over time. I'm not fond of the wall wart PSU's.

This is the reason I went with the SANS DIGITAL Unit. No trays either.

 

But if I had a ton of 2.5" drives laying around, I might try this unit out.  Also the rosewill 5 drive PM units on newegg support 3.5 and 2.5. Only thing is they probably are not hardware raid, which is what you want.

 

What I like about the AMS VENUS/ADDONICS unit is you could load it up with SSD's or 2.5" hybrid drives and get some decent performance out of it.

Anyone aware of a good external array that supports 2.5 inch laptop drives? I've got tons of them lying around and it would be great to use some of them as an external RAID 5 datastore. I realize I could get adapters and put them in 3.5 inch bays but the small size of a 2.5 inch only external array would be nice.

 

This is the only one I can find on Newegg and it has very mixed reviews. http://www.newegg.com/Product/Product.aspx?Item=N82E16816216004

 

Addonics seems to have the same product rebadged.

I suppose if they think it's good enough to rebadge, this one might be worth it for the low cost.

http://www.addonics.com/products/cpr5sa.php

 

I had a venus AMS raid enclosure once. The cable slowly slipped out of the PSU over time. I'm not fond of the wall wart PSU's.

This is the reason I went with the SANS DIGITAL Unit. No trays either.

 

But if I had a ton of 2.5" drives laying around, I might try this unit out.  Also the rosewill 5 drive PM units on newegg support 3.5 and 2.5. Only thing is they probably are not hardware raid, which is what you want.

 

What I like about the AMS VENUS/ADDONICS unit is you could load it up with SSD's or 2.5" hybrid drives and get some decent performance out of it.

 

 

Maybe I'll give it a shot then. I have like 15 320GB WD Black drives and 15 or so 250GB WD Blue drives. I figure a 5 disk array of the Blacks should give me some decent performance since they're pretty on par with the desktop Black drive performance.

Anyone aware of a good external array that supports 2.5 inch laptop drives? I've got tons of them lying around and it would be great to use some of them as an external RAID 5 datastore. I realize I could get adapters and put them in 3.5 inch bays but the small size of a 2.5 inch only external array would be nice.

 

This is the only one I can find on Newegg and it has very mixed reviews. http://www.newegg.com/Product/Product.aspx?Item=N82E16816216004

 

Addonics seems to have the same product rebadged.

I suppose if they think it's good enough to rebadge, this one might be worth it for the low cost.

http://www.addonics.com/products/cpr5sa.php

 

I had a venus AMS raid enclosure once. The cable slowly slipped out of the PSU over time. I'm not fond of the wall wart PSU's.

This is the reason I went with the SANS DIGITAL Unit. No trays either.

 

But if I had a ton of 2.5" drives laying around, I might try this unit out.  Also the rosewill 5 drive PM units on newegg support 3.5 and 2.5. Only thing is they probably are not hardware raid, which is what you want.

 

What I like about the AMS VENUS/ADDONICS unit is you could load it up with SSD's or 2.5" hybrid drives and get some decent performance out of it.

 

 

Maybe I'll give it a shot then. I have like 15 320GB WD Black drives and 15 or so 250GB WD Blue drives. I figure a 5 disk array of the Blacks should give me some decent performance since they're pretty on par with the desktop Black drive performance.

 

 

Let us know how you make out.

Can anyone think of a reason this wouldn't work with an external eSATA array? I don't see why it wouldn't.  http://www.amazon.com/gp/product/B001GV5WBE/

 

Using this will prevent me from having to use a PCIe slot, I can just drop it an empty bay and plug it directly in to one of my SATA ports on the board, only one of them is currently in use.

Can anyone think of a reason this wouldn't work with an external eSATA array? I don't see why it wouldn't.  http://www.amazon.com/gp/product/B001GV5WBE/

 

Using this will prevent me from having to use a PCIe slot, I can just drop it an empty bay and plug it directly in to one of my SATA ports on the board, only one of them is currently in use.

 

 

It should work. I had a few in my arsenal.  The only issue you may come up with is if the motherboard and drivers support PMP.

If you are using a hardware raid array, it should look like 1 large drive and work without a problem. Has for me in the past.

Can anyone think of a reason this wouldn't work with an external eSATA array? I don't see why it wouldn't.  http://www.amazon.com/gp/product/B001GV5WBE/

 

Using this will prevent me from having to use a PCIe slot, I can just drop it an empty bay and plug it directly in to one of my SATA ports on the board, only one of them is currently in use.

 

 

It should work. I had a few in my arsenal.  The only issue you may come up with is if the motherboard and drivers support PMP.

If you are using a hardware raid array, it should look like 1 large drive and work without a problem. Has for me in the past.

 

 

That is what I figured.

 

This project may be getting put on hold for now though. Newegg sent me a coupon code for Norco cases and I've been waiting on one forever so I think I'm going to pull the trigger on a Norco 4224, which means I've also got to buy a 120mm fan bracket/fans and mini SAS cables for my controller cards so I'm looking at $425+ even with the coupon. Hopefully I will be able to offset some of the cost selling my Supermicro 5 in 3 cages and possibly my Antec 900 case. Either way my budget for server parts will be used up for the next couple months. My birthday is coming up soon though so maybe I'll ask the girlfriend to pick this array enclosure up for me.