Sign in to follow this  
nightanole

Taming a 12th gen Dell PowerEdge

5 posts in this topic Last Reply

Recommended Posts

Gotta love decommissioned servers...

 

Anyway these are starting to hit the used market and are getting affordable.

The 2U series such as the R720 would make a great medium sized unraid box.

They can use V1 and V2 XEON 2XXX series socket 2011 chips.

They can handle ALOT of ram, last head count was 786 gig of ddr3 1600-1866mhz depending on the cpu.

They need 2 cpus to take advantage of all the expansion and ram slots.  You are down to 4 8x pci-e slots(and one will be a sas controler) if yo only use one cpu.

you can get them with a front plane of 8 3.5" sas/sata trays.

 

 

Now before you just haul out and get 2 $85 2670v1 like half the planet.  If you are installing video cards that require external power, you are limited to 95watt cpus, and you need 1100watt power supplies.  However they do make a poor mans 2670, a 2650.  Its only 2.0ghz before turbo (vs 2.6ghz for the 115 watt 2670) but other than that, its got the same amount of cores and features of the 2670, and i have scored them for $30 shipped on auction style on ebay.  SR0KQ is the latest.

 

Now on to taming this bastard.

Gee i have a perc310 already onboard, lets flash the sucker to IT mode.  DO NOT DO THIS!

If anything other than what is approved in the bios is in the special dell slot, the server will halt on post.  Yes i figured that out the hard way, $45 door stop now.

You might get away with flashing the perc directly from a h310 to a sas6gb hba dell, but remember if the server reboots with a blank firmware in the lsi, you are hosed, and i dont know if you can directly cross flash dell to dell in one boot.

 

Ok lets keep going.  You have your 2 2650s (you bough that dell heatsink for the 2nd cpu that cost as much as said cpu right?), you also bought a normal perc h310 and cross flashed it. 

GG ready to go right?  Wrong...

You need to win some 35" sas right angle cables from a HP server in order to route the back plane to the standard perc h310.  I have not tried the Chinese knock offs.

You also need a SLIM sata to sata adapter so you can replace your dvd rom with your cache drive.  Dont be like me and spend $8 on a MICRO sata to sata adapter, i put that next to my mini mono perc door stop.

 

Ok so we got everything setup, we have the bios set to boot from the unraid stick, and we are ready to do right?  WRONG.

That sucker is going to idle at 200-250 watts with a ups depending on bios settings.

In bios under system profile, set it to performance per watt.  This will engage all the C states and also allow turbo.

Go into the idrac under power, and set the power supply to redundant, hot spare, and active pfc.  This shuts off one power supply (20% idle power reduction) and puts the main power supply in the meat and potatoes section of its efficiency band, you go from 85% to 95% in most cases.

Now go to back into idrac and set thermal to "performance per watt controlled by idrac" and set fan profile to "exhaust temp" and its either going to be adjustable, or if you have the new commie bios it will be set for 50c.  This will let the fans idle down to 2000rpm (out of 10krpm).  If you have the older bios they may even idle down to 1000rpm, but the new bios has a floor of 2000rpm if there is ANY card in the pci-e section, or more than 1 hard drive in the SAS plane.

Now plug power supply 1 into the "surge only" section of your ups, and the 2nd power supply into the battery backup section.  This will again decrease your idle power usage by another 10%.  When the power goes out the server will kick over to the ups on the 2nd power supply.

 

With 2 xeon 2650v1's, 4 16 gig 1.35v dimms, a IT flashed per h310, and six 7200rpm HGST nas drives, my system idles at 84 watts from the wall, and 120-130 watts with all the drives spun up.

 

On a final note, some bios revs have some horrible thermal profiles.  Its called the 3rd party card "bug".  So while the current 2.X firmware will do a thermal test to set the min fan speeds for pci-e cards  (at least bus powered only anyway) others will set the min fan speed to 4krpm.  And if you want to really live in hell, some firmwares will set the min fan speed to 75% if a non bus powered card is installed, even if it is a gtx 1050 that idles at 5-10 watts.

To get around this(only if you have this problem) to need to set the thermal profile to ignore 3rd party card default values. 

You need to get into the idrac settings and enable a 2nd network port to run IPMI over LAN and set an ip address.

Download dells open management tool.

http://www.dell.com/support/home/us/en/19/Drivers/DriversDetails?driverId=J49YD

 

and run:

ipmitool -I lanplus -H (ipaddress)-U (username, default is root) -P (password, default is calvin) raw 0x30 0xce 0x00 0x16 0x05 0x00 0x00 0x00 0x05 0x00 0x01 0x00 0x00

and if you want  to set it back to default, you just run 0x30 0xce 0x00 0x16 0x05 0x00 0x00 0x00 0x05 0x00 0x00 0x00 0x00

 

to check your handy work, run:

ipmitool -I lanplus -H -U -P raw 0x30 0xce 0x01 0x16 0x05 0x00 0x00 0x00

 

The response data is:

16 05 00 00 00 05 00 01 00 00 (Disabled)

?16 05 00 00 00 05 00 00 00 00 (Enabled)

 

 

So parts list:

Dell perc h310 normal pci-e (not a mini mono).  $35-45 In unraid 6.3 the new perc H330 mini mono will be supported, and can be set to HBA with stock dell bios, but will it run in a 12th gen server?

Slim Sata to Sata adapter $8

2 Xeon 2670v1 or (or 2650v1 if running video cards) $60-$200 shipped

2 HP server 35" right angle sas cables $15  (HP part SFF-8087 90 Degree Cable 35" 685183-001 672241-001)

1 dell heatsink for the 2nd cpu 05JW7M $40

 

 

 

Share this post


Link to post

The main server at work is a R515 with a pair of 6-core Opterons.  It's quiet, and really quite power efficient (mine has 8x3.5" drives and 2x2.5" 15k drives and only uses 120W). It'd make a great unRAID box.

Share this post


Link to post

Thanks for this thread. I've gotten a Dell R720xd - the 12 bay model. I have a LSI SAS 9200-8e as the only PCI card and I have applied all of your recommended settings above. My unit has 2x E5-2650.

 

I'm going nuts trying to get the fans to run less than 8000 RPM. 

 

I did the ipmitools trick, I have used your settings as well as trying the dense configuration settings, etc. I've also tried manual system profiles, disabling the C states and C1E states and turbo modes - saw this recommended elsewhere.

 

I updated all of the firmware to the latest editions using the Lifecycle controller.

 

I've also tried pulling out RAM to see if a large volume of RAM was causing the issues - runs the same whether I have 8x4gb sticks or 22x4gb sticks.

 

I've shut down dockers and VM's, no change. Seems like they're dead set on running between 8000 and 8400 RPM.

 

Any tips? I'm thinking about selling this on ebay, where I got it from.

Share this post


Link to post

So here's a follow up in case anyone finds this in the future.

I've found that having ANY drives in the server causes massive fan ramp-up.

I have now moved all of my drives to a Lenovo SA120 and consolidated my array (shrunk it a little)

Next up I'm going to re-do the thermal grease on the CPUs, but I'm now down to idle fan speeds of 5k or so.

Also, I may also pull the secondary PSU to reduce noise.

Share this post


Link to post

Probably due to your choice of drives.  Some drives report temps differently to what Dell's iDRAC expects.  Dell really want you to use their super-expensive drives.

 

Generally, if you use WD RE4/Gold or Seagate Constellation drives, you're OK.  Anything else will be "uncertified", and may run the fans in OMG loud mode.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  


Copyright © 2005-2018 Lime Technology, Inc.
unRAID® is a registered trademark of Lime Technology, Inc.