Blackhole 2.0 - 60TB v.Now with 100% more ESXi

Update 2/05/13

So after a year of heavy usage on the server, I finally decided it was time for an upgrade to visualizing unRAID on ESXi.  To be honest I should have gone this route from the start but it was fairly painless upgrade and I want to give a big thanks to Johnm and all his work and information on his Atlas guide as it has been a huge help.

Anyways the server upgrade was as follows:

Blackhole 2.0 - EXSi 5.1 - unRAID 5.0RC11

Case: Lian Li PC-A77FB

CPU: Intel I3 2120 3.3Ghz – Intel I3-1230 V2

Motherboard: Supermicro MBD-X9SCM-F-O  BIOS v1.1 -> BIOS 2.00b

RAM: Kingston KVR1333D3E9SK2/8G 8GB 24GB DDR3 ECC Unbuffered Ram

Drive Cage(s): 4x Supermicro CSE-M35T-1B Hot Swap Bay

Power Supply: Corsair TX750M

SATA Extender: 2x 3x Supermicro AOC-SASLP-MV8 SATA Extender

Fan Replacements: 4x Gelid 92MM Fans

Data Store Drives: 128GB Corsair Performance Pro SSD  / 2GB Seagate 2TB ST2000DM001

Parity Drive: 1x Seagate 3TB ST3000DM001 7200RPM

Data Drives: 10x Seagate 3TB ST3000DM001 7200RPM / 5x WD RED  WD30EFRX 3TB

Cache Drive: 1x WD WD15EARS

Decided to try out the Ivy Bridge despite reporting some problems (mainly due to stock and pricing). I know there was some reported bugs with IVY Bridge but so far so good. Upgrade for X9SCM-F from BIOS v1.1 to 2.00b went without a problem as well. If anyone needs help here I’d be happy to post a guide but there’s plenty on Google. Also upgraded to beefier 24GB of ram and possibly 32GB later depending on what I’m using it for. Had a spare SSD lying around so threw it into data store and hopefully it doesn’t die on me.

Random Notes:

- I hate that the Seagate and WD drives blink differently on the CSE-M35T Swap bays. i.e Seagate drives activity lights are always on and WD activity lights off, but blink on activity. Being kind of OCD in this I find it annoying!

- Had to pick up another MV8 recently as I had to pass both the cache and the parity through and didn’t want to RDM into the unRAID guest.

- For some reason, after passing through the parity drive through MV8 to the unRAID guest (Instead of directly from motherboard), my parity checks is FASTER? Averaging 120 instead of the usual 80? I find this odd.

- …..

Anyways just wanted to share my upgrade process for upgrading an EXISTING unRAID server to ESXi in the safest possible way. I was a little paranoid as I had a lot of data and was scared to lose it and was pretty much testing data integrity at every step. Anyways here is my process. This is just a detail of the steps I took. For a better guide on ESXi I strongly recommend reading John’s Atlas thread …

• Upgraded BIOS to 2.00b
  
• Testing
  
• Installed new CPU/RAM/DATASTORE drives/MV8
  
• Moved Parity & Cache to new MV8
  
• More Test to ensure unRAID recognizes drives on MV8
  
• Unplugged all drives from ESXi (ESXi will format all drives it sees)
  
• Install ESXi through your choice of methods (I preferred USB stick method)
  
• Configure ESXi
  
• Install unRAID Guest using PLOP method
  
• Passthrough hack for MV8 and configuration
  
• Testing with passing through MV8 and 1 Drive
  
• Plug everything back in and unRAID should recognize all drives
  

And that’s it! Took me a bit longer than a few hours but I was meticulously testing and making sure my data was intact at most of the steps. However unRAID recognized the drives perfect and had no problems upgrading.

==========================================================

Original Post

===========================================================

Hi Everyone,

Just sharing a build I'm finishing with. Extremely happy with results so far. Spent weeks deciding on parts/which OS etc to go with but finally stuck with Unraid and very happy with it. Big s/o to all the helpful post on here on selecting my parts. I'm also sharing a few of my notes for parts I selected just in case it helps anyone else out as well.

Goals:

I currently have an HTPC computer that previously stored all my movies/tv shows and it was filling up fast! It had a total of 11TB of storage and was about 90% filled. As a result, I sought out to build a computer to store all my media. I know that 60TB was the goal and nothing less as I went through 11TB fairly quickly. Furthermore, the way my condo is wired up, having the server in a different room was NOT an option (I went through all the options and it wasnt going to happen). Thus the server would need to sit in the living room, be extremely quiet, and look good. So my priorities as follows:

1) 60TB Storage

2) Silent

3) Looks good in the living room

Case: Lian Li PC-A77FB

CPU: Intel I3 2120 3.3Ghz

Motherboard: Supermicro MBD-X9SCM-F-O

RAM: Kingston KVR1333D3E9SK2/8G 8GB DDR3 ECC Unbuffered Ram

Drive Cage(s): 4x Supermicro CSE-M35T-1B Hot Swap Bay

Power Supply: Corsair TX750M

SATA Extender: 2x Supermicro AOC-SASLP-MV8 SATA Extender

Fan Replacements: 4x Gelid 92MM Fans

Parity Drive: 1x Seagate 3TB ST3000DM001 7200RPM

Data Drives: 10x Seagate 3TB ST3000DM001 7200RPM (So far)

Cache Drive: 1x WD WD15EARS

Random Build Notes:

- I couldn't be happier with the case. I've owned a number of Lian-Li cases in past and the quality is top notch. The build was a bit on the expensive side especially on just the case + cages... But again was not an option to go with a cheap Norco case and store it in a closet.

- I should of read a bit more on the difference between forward and reverse breakout because like a noob I bought the reverse breakout and had to exchange  :'(

- CPU/Motherboard is a bit overkill... but hopefully i'm transitioning all downloading duties over to the server from the HTPC... which makes the HTPC overkill.. alwell

- Read a bunch of reviews on the supermicro cages being tight... my god I dont know how they are going to get out if I ever need to replace a cage lol... they are in there for good

- Completely silent... actually my HTPC is noticably louder (and its pretty quiet in general) than my server so im very happy with outcome

- Case looks beautiful in living room.

- IPMI on the supermicro board is absolutely AMAZING....

- Was scared the SASLP did not support 3TB drives.... but actually it was just because i bought the wrong cables.... lol everything worked out

- Drive temperatures max out at 35 for all drives during parity or preclears

- I went a bit creative and attached the cache drive and the parity drive on a seperate rack on top of the motherboard so it is not taking up any of the cages (23 drive capacity)

- More to come i mess with things...

Pictures:

Opened up

Dont mind the dirty HTPC 

Full Setup picture

Details setup for anyone who was asking

Llano HTPC Media Center - Running XBMC

Antec 300

AMD A6-3650 APU

Asus F1A75 FM1 Motherboard

G.Skill 8GB DDR3 Ram

1x 500GB WD Green Hard Drive

3x 3TB WD Green hard Drive

1x 1.5TB WD Green Hard Drive

Very nice living room setting.

I see you are missing the WII

Looks nice however I would highly suggest putting your USB stick inside your machine opposed to the front of your machine. There are several adapters you can use to plug it in. Once kick or somebody wanting to annoy you by pulling it out could be a real pain in the butt later.

Looks nice however I would highly suggest putting your USB stick inside your machine opposed to the front of your machine. There are several adapters you can use to plug it in. Once kick or somebody wanting to annoy you by pulling it out could be a real pain in the butt later.

Like this one:

http://www.amazon.com/gp/product/B000IV6S9S

My other server uses one that I made work. It's a USB header with bracket that I simply removed the bracket from and plugged in. I like them both.

Looks nice however I would highly suggest putting your USB stick inside your machine opposed to the front of your machine. There are several adapters you can use to plug it in. Once kick or somebody wanting to annoy you by pulling it out could be a real pain in the butt later.

Where do you see a front mounted USB on the UnRaid server?  I see two on what I guess are his HTPC, but not the server with the cages.

I have a similar build with the Lian-Li A77 and Super Micro cages. I replaced the Super Micro cage fans with Noctua NF-B9.

It is a nice setup and quiet enough to not intrude.

:-)

Alex

I have a similar build with the Lian-Li A77 and Super Micro cages. I replaced the Super Micro cage fans with Noctua NF-B9.

 

It is a nice setup and quiet enough to not intrude.

 

:-)

 

Alex

i am running these in my My supermicro cages, Did you just pinch the cable, or did you mod something?

 

- I went a bit creative and attached the cache drive and the parity drive on a separate rack on top of the motherboard so it is not taking up any of the cages (23 drive capacity)

 

Can you please post a picture(s) of the "separate rack" and/or elaborate a bit of detail on how you mounted this inside the case?  I am in-process of building a very similar setup, (same MB, case and SM cages) and would like to find a creative way to mount the SSD cache drive inside the case without taking up one of the 20 slots in the 4x Super Micro 5-in-3 cages.

Very nice living room setting.

 

I see you are missing the WII

Never was a fan of the Wii... to be honest I dont even use the PS3 or Xbox much nowadays I just watch movies and TV Shows all day.

Looks nice however I would highly suggest putting your USB stick inside your machine opposed to the front of your machine. There are several adapters you can use to plug it in. Once kick or somebody wanting to annoy you by pulling it out could be a real pain in the butt later.

The USB Stick that is protruding is the mouse/keyboard on the HTPC. I only leave them in front because I move around the keyboard and mouse  a lot around the house so its easier that way. If you are refering to the USB drive for the UnraidOS, that is inside the case directly attached to the motherboard so i'm not to worried about accidentally kicking it.

I have a similar build with the Lian-Li A77 and Super Micro cages. I replaced the Super Micro cage fans with Noctua NF-B9.

 

It is a nice setup and quiet enough to not intrude.

 

:-)

 

Alex

Thanks The A77 was way overkill and unnecessary but it looks beautiful and with the quiet fans my HTPC ends up being louder so im pretty satisfied thus far.

 

- I went a bit creative and attached the cache drive and the parity drive on a separate rack on top of the motherboard so it is not taking up any of the cages (23 drive capacity)

 

 

Can you please post a picture(s) of the "separate rack" and/or elaborate a bit of detail on how you mounted this inside the case?  I am in-process of building a very similar setup, (same MB, case and SM cages) and would like to find a creative way to mount the SSD cache drive inside the case without taking up one of the 20 slots in the 4x Super Micro 5-in-3 cages.

Sure thing... it wasn't anything fancy to be honest... actually its quite crude and I'm not sure how you would do it with another case due to one of the unique features of this case.

The A77 case comes with a Power Supply strap that mounts on the powersupply without the use of screws. Removing the hard drive cages in the A77F I ended up with 3x 4 harddrive cages. I just attached one the hard drive cages with the strap with a layer of foam to dampen it.

The strap I'm referring too

Wiring could be cleaner but alwell

Clearer picture of the attached cage. As pictured the Cache and the Parity drive in there with room for 1 more as its' a 4x3 cage

Thanks for the detail kevinsyn.  Very inventive solution!  I like the idea, the only problem I would have is that my power supply (TX-850 V2) has an intake fan on top and I would be afraid to mount the HDD cage (or foam) atop that intake fan.  I wonder how much clearance the PSU intake fan needs for airflow.

Since the PC-A77F case has so many PCI brackets, I'm going to see if there is a internal case HDD bracket that might use 1-2 spare PCI brackets but allow suitable airflow for the power supply.  This might work since the motherboard (X9SCM-F) is a microATX.

Thanks for the detail kevinsyn.  Very inventive solution!  I like the idea, the only problem I would have is that my power supply (TX-850 V2) has an intake fan on top and I would be afraid to mount the HDD cage (or foam) atop that intake fan.  I wonder how much clearance the PSU intake fan needs for airflow.

 

Since the PC-A77F case has so many PCI brackets, I'm going to see if there is a internal case HDD bracket that might use 1-2 spare PCI brackets but allow suitable airflow for the power supply.  This might work since the motherboard (X9SCM-F) is a microATX.

I have the TX-750 (same model almost)... you can flip the power supply so it draws air from the bottom (since there is spacing at the bottom) and ur good to go.

Awesome set up.  Once my wife and I buy our own house, I want to build something similar ...hoping to get the house by the end of the year.

I have the TX-750 (same model almost)... you can flip the power supply so it draws air from the bottom (since there is spacing at the bottom) and ur good to go.

Yes that would work.  I checked and there is a large screened opening on the bottom of the case to provide ample outside airflow into the PSU fan, so that would seem to work quite well.  I suppose there might be some negligible loss of airflow from within the case through the PSU, however with 4x Super Micro 5-in-3 cages and the 2 top case fans and 1 back case fan, there is probably considerable air movement (throughout the case) to negate any air flow loss by flipping the power supply over.

Thank you for the ideas! 

Thanks for sharing this. I plan to follow your lead. Already ordered the same case. Already have 3 of those cages. I like the use of that rack for placing the parity and cache drive.

Did you flatten any 5.25" tabs inside the upper 3 drive bays? Is there a tighter fit for the cse-m35t in the upper bays than in the lower ones?

Did you use IPMI to power-cycle the server? Reason for asking is the fact that my motherboard wouldn't do that with a TX750M PSU.

Thanks for sharing this. I plan to follow your lead. Already ordered the same case. Already have 3 of those cages. I like the use of that rack for placing the parity and cache drive.

 

Did you flatten any 5.25" tabs inside the upper 3 drive bays? Is there a tighter fit for the cse-m35t in the upper bays than in the lower ones?

 

Did you use IPMI to power-cycle the server? Reason for asking is the fact that my motherboard wouldn't do that with a TX750M PSU.

Glad to hear it.

Yes the top 3 drive bays need to be flatten. But since it is an aluminum case it was very easy to mold and work with so i didn't have much difficulties.

Man it is a very very tight fit for the CSE. I spent a good 3 hours getting all 4 of them inside. I dont even know how i did it really just pushed really hard. To be honest i don't think they are coming out as it seems to be a one way trip 

Yes i usually use IPMI to powercycle. I've had no problems with the PSU compatibility with the motherboard.

Thanks for sharing this. I plan to follow your lead. Already ordered the same case. Already have 3 of those cages. I like the use of that rack for placing the parity and cache drive.

 

Did you flatten any 5.25" tabs inside the upper 3 drive bays? Is there a tighter fit for the cse-m35t in the upper bays than in the lower ones?

 

Did you use IPMI to power-cycle the server? Reason for asking is the fact that my motherboard wouldn't do that with a TX750M PSU.

 

Glad to hear it.

 

Yes the top 3 drive bays need to be flatten. But since it is an aluminum case it was very easy to mold and work with so i didn't have much difficulties.

 

Man it is a very very tight fit for the CSE. I spent a good 3 hours getting all 4 of them inside. I dont even know how i did it really just pushed really hard. To be honest i don't think they are coming out as it seems to be a one way trip 

 

Yes i usually use IPMI to powercycle. I've had no problems with the PSU compatibility with the motherboard.

I used a big pair of needle nose pliers to start a very slight downward bend on the tab and then used a multiple passes with a 6" c-clamp to flatten each tab.  I torqued down the c-clamp to get all the tabs essentially flat. The flattening process with the 6" c-clamp did leave a slight imprint of the clamp ends in the case but that's OK.  Took a while to do this and was a bit of a pain but the good news is its a "one-time" step.  If I remember correctly there were a few tabs towards the back of case where I had to get a bit creative with the c-clamp placement but it's definitely doable.

I also removed all of the horizontal aluminum dividers before installing each Supermicro CSE-M35T-1B 5-in-3 drive cage.  This gives more room to work with to install each cage.  (I had problems inserting the cages (too tight) until I removed the horizontal dividers.)  You can remove the screws holding each horizontal aluminum divider with a small screw driver.  The top horizontal divider was a bit trickier (for me) since the access holes were slightly misaligned with the screws, but if you have an assortment of small flathead and phillips screwdrivers you can get all the screws out and then remove all the horizontal aluminum dividers.

Adding the Supermicro CSE-M35T-1B 5-in-3 drive cages in the PC-A77FB case reinforces the vertical structures without the horizontal aluminum dividers, so your case will be solid even without the horizontal dividers.

I didn't have any trouble installing (3) Supermicro CSE-M35T-1B 5-in-3 drive cages.  I still have a 4th drive cage to install from my other unRAID server.  I'll install the 4th drive cage in the bottom slot once version 5.0 final is released and I port everything over from my existing 4.7 pro server to the new server (2nd unRAID server in the Lian Li PC-A77FB case).

I found solution to mounting SSDs in the Lian Li PC-A77FB case.  With a microATX motherboard in this case, there are 4 unused PCI slots towards the bottom of the case.  The PCI slot covers that come with the case have diagonal slots that line up perfectly with the side mounting holes on the SSD.  I mounted the SSD to the PCI slot cover and installed it in the case with the power and SATA connectors facing upward/outward.  (See attached pictures.)

I don't really like that you have to force those cages in but I guess that will happen with all cases. Are other cages less prone to this?

Thanks for all advise guys.  I shall see if C-clamp are available here in Holland. Do I need a specific type or size?

Thanks for all advise guys.  I shall see if C-clamp are available here in Holland. Do I need a specific type or size?

If you flatten the ~ 9 tabs in the top section with a c-clamp (see attached picture) and remove the horizontal dividers between each section, you won't have any trouble inserting the Supermicro CSE-M35T-1B 5-in-3 drive cages, no forcing required.

I had both 4" and 6" c-clamps.  I think I used only the 6" clamp since the clamping surface area was greater (compared to the 4" clamp).  I would think these c-clamps would be readily available in Holland at a hardware store.  They cost ~ $10 here in the USA.

This 4" and 6" does that refer to the width or depth of the clamp? Do you need a certain depth or can you use a standard c-clamp.

This 4" and 6" does that refer to the width or depth of the clamp? Do you need a certain depth or can you use a standard c-clamp.

See the dimensions in this picture.  As far as I know they are regular c-clamps.  I believe I only used the 6" c-clamp.

I found solution to mounting SSDs in the Lian Li PC-A77FB case.  With a microATX motherboard in this case, there are 4 unused PCI slots towards the bottom of the case.  The PCI slot covers that come with the case have diagonal slots that line up perfectly with the side mounting holes on the SSD.  I mounted the SSD to the PCI slot cover and installed it in the case with the power and SATA connectors facing upward/outward.  (See attached pictures.)

Very ingenuitive   Just in case you weren't aware, there are options like this as well - I have 2 in my case and they're great.

http://www.amazon.com/StarTech-com-2-5-Inch-Removable-Expansion-S25SLOTR/dp/B002MWDRD6/ref=sr_1_4?s=electronics&ie=UTF8&qid=1338713364&sr=1-4

What speakers are in that picture in the OP? Can you share some info on your AV-setup?

Very nice build!

I pretty much did the same thing, although i used the norco 5 in 3's and am running windows 2k8r2

I spent a lot of time deciding on the OS, because i have always wanted to play with raid, I ended up going down the windows route and used a areca 1880ix-24-4gb raid controller.

The way i got rid of the tabs was to get a hacksaw and cut them out. As you can see below

For drives I am using Hitachi 4TB Coolspins, have 6 in there ATM.

To mount the SSD i got some strong velcro with adhesive backing and stuck it to the back of the motherboard tray

Here are some more pics of the beast. I went a little over the top and added water cooling.