Improving unRAID Performance

From unRAID

Jump to: navigation, search

Although the unRAID system is not designed for high-performance requirements, like database servers and high demand media servers, there is no reason we cannot try to achieve the maximum possible performance with the money and hardware we have invested into it.

Here are a number of collected ideas to help improve your performance. Of course, these assume that you have your unRAID system running correctly. If you have a bad drive or cable, or incompatible hardware, or mis-configured CMOS settings, no amount of tips are going to help. Checking the User Benchmarks tables may be helpful, for comparing your system with others.


Contents

Hardware Tips

Networking

Use Gigabit Networking

  • You aren't using gigabit networking? Why not? See this post.
  • Gigabit networking requires good quality Cat5e or Cat6 cables. A bad or cheap cable, or poorly wired connectors, can severely affect your networking speeds.
  • In general, Intel provides better networking chipsets and the best performance. Realtek chipsets are the least recommended (but the most common!).
  • In networks with more active network devices, better quality routers and switches provide better performance and lower latency.

Reduce Latency

  • You can use ping to report latency. You should have latency of 2 ms or less on your local network. If it is higher than that, you should consider swapping out your network switch/hub. Test your system with a crossover cable, and no switch, to ensure that the latency is not being introduced by the NIC.

Jumbo Frames

  • Jumbo Frames are Ethernet frames with more than 1500 bytes of payload (1500 bytes being the default). Jumbo Frames could potentially speed up your network performance and decrease CPU use, but care must be taken when using them due to variations in network hardware.
  • Many gigabit NICs support jumbo frames of 9000 bytes or more, but it's advisable to find the datasheet for your specific controller and determine exactly what MTU is supported. Also, make sure your network hardware and other endpoints support the same size MTU.
  • You can set the MTU size from the command line by logging in as root and using the ifconfig command. Settings changed via ifconfig will be reset when the machine is rebooted, which makes this method ideal to experiment with different MTU settings.
# ifconfig {interface-name} mtu {size}
# ifconfig eth0 mtu 9000
  • To persist Jumbo Frames configuration, add the MTU property to /boot/config/network.cfg with the size, in bytes, of the payload.

An example of /boot/config/network.cfg:

# Generated network settings
USE_DHCP=yes
IPADDR=
NETMASK=
GATEWAY=
MTU=9000


Hard Drives

PCI and PCIE

In the following notes SATA is preferred over PCI BUT you should note that PCIe has a bus speed in excess of 1GHz and may have multi-lane capability that would offer significant speed increases especially for SSD drives.

Use Cache Drive

Add a Cache drive (with the Plus or Pro license only) to your User Shares for a major improvement in write speed. The Cache drive improves performance by postponing the parity processing, so writes to it are at full speed. The files are later moved to the parity-protected data drives during off-hours. A side effect of this is that file fragmentation is also reduced, if there is simultaneous streaming from the User Shares. Although the simultaneous writes produce fragmented files, the transfers later to their permanent locations on the parity protected drives are done one file at a time, which should normally create files without fragmentation.

The cache drive is described in the release notes here

Move Parity Drive Off PCI Bus

If possible, ensure that your Parity drive is not sharing an IDE channel or the PCI bus or a Port Multiplier channel. For best write speed, move it to the fastest unshared connection. For best overall performance, it should be connected to the fastest SATA II port. Normally, onboard SATA ports are the fastest, then SATA II ports on the PCI Express bus. It does not have to be the fastest drive, but for best performance, it should be at least a SATA II drive. For best configuration, see the Improving unRAID Performance#Experiment with Hard Drive Config section below.

Assign Disk slots for Performance

There is some evidence that you can improve parity check performance, reducing the impact of having several drives on the PCI bus, by alternating the disk slots across the various controllers. Assigning drives in a round robin style (parity on controller1, disk1 on controller2, disk2 on controller3, disk4 on controller1, etc.) may have a positive performance advantage.

Move Largest and Fastest Data Drives Off PCI Bus

Your largest drives will most likely store the majority of your data, and therefore will be read more often when you retrieve that data. In addition to moving your Parity drive off the PCI bus, also move your largest and fastest Data drives off the PCI bus. If you must have drives on the PCI bus, realize that they will be slower, and organize your data accordingly.

Consider this as you are building your unRAID server and position your drives and cables to optimize performance. If you have already built your unRAID server and populated it with data, you can still rearrange your drives and cables to optimize performance without ever losing parity protection. Here's how:

  1. On a different computer (not the unRAID server), open a web browser and go to the unRAID main page, http://tower/main.html for UnRAID v4.7 or below, http://tower/Main for UnRAID v5 and above (replace tower with your server's name).
  2. Click the 'Devices' tab at the top of the page.
  3. Take a screenshot of this page and save it to a safe place. Do not save it on the unRAID server, since you will need to be able to access it with the array offline. You can also copy down the information by hand, all you need are the serial number (for example, WCAU45951367) and the drive slot (for example, parity, or Disk7) for each drive in the array. Make sure that you can read all the disk slots and serial numbers for each drive, and that you are clear about any potentially ambiguous characters (letter 'S' versus number '5', for example).
  4. Click the 'Main' tab, then click the 'Stop' button, then the 'Power Down' button to initiate a clean power down.
  5. After server has shut down, unplug your unRAID server (or flip the power switch on the power supply).
  6. Open the case and rearrange the drives, cables, and most importantly the ports on the motherboard, PCI-e, or PCI card to which they are attached. The rule of thumb is that you want the largest and fastest drives connected to the fastest ports, which are generally your onboard ports (directly on the motherboard), then your PCI-e ports (on a SATA card connected to a PCI-express slot), then your PCI ports (on a SATA card connected to a PCI slot) in that order. SATA II ports are much faster than SATA I, and both are much faster than IDE. As mentioned above, your Parity drive should always be connected to an onboard port (or your fastest port if you don't have onboard SATA ports) whenever possible.
  7. Check all SATA cables are tight, close the case, and power on your unRAID server.
  8. On a different computer (not the unRAID server), open a web browser and go to the unRAID main page. The array may be stopped if it doesn't recognize your new configuration. If for some reason the array is not stopped, stop it immediately. You may also see an error like 'Drive in Parity slot is not the biggest'. Don't worry, this is normal.
  9. Click the 'Devices' tab at the top of the page.
  10. Using the drop down menus, unassign all your drives. You only have to unassign drives that unRAID puts in the wrong slot, but I find it easier just to unassign them all, so there's no confusion.
  11. Using the screenshot/notes you took earlier as a guide, reassign all your drives into their correct slots. Start with the Parity drive, then work your way down the list. Be very careful, double and triple check that you do in fact have the correct drive in the correct slot. If you have several drives from the same manufacturer, the drive serial numbers may be very similar and may only differ by one or two characters.
  12. Once you are sure that all your drives are in the correct slots, especially the Parity drive, you can click the 'Main' tab and click the 'Start' button to start the array. unRAID should report a healthy system status with no errors.

For more discussion of this method, see this thread: Physically rearranging drives cables, swapping position on mobo/pci card

For a complementary guide to the same process, see the FAQ: What is the safe way to rearrange disk numbers, assignments, slots, etc?

Parity Check Speed

  • In general, concerns over Parity Check speeds are misplaced, but it can be useful as a way of determining whether your UnRAID system is performing satisfactorily.
  • There is much confusion over this, even the term itself, as to whether we are referring to the overall parity speed (found in the syslog, measured in seconds), or referring to the reported speed at a particular moment, at a particular percentage point of the process. The overall speed found in the syslog is reliable, but you have to wait for it! The current speed displayed in the UnRAID Webgui is notoriously unreliable. It is usually very slow and inconsistent for the first couple of percent, then rises to a near maximum rate, then slowly falls quite a ways, then may rise and fall several more times before completion. Unless you understand the reasons for this, you cannot draw any conclusions from these low or high speeds!
  • To understand why there is so much variance in your parity speed, this post has a good explanation, and there is additional explanation in this post.
  • The more drives you add to your array, and the larger the drives you add, the slower the parity check will be. There is little you can do to speed it up, assuming you have followed the guidelines above, about putting your parity drive and your fastest drives on the fastest ports. You cannot improve the speed by adding a faster drive, and in fact it may slow it down a little more. The only way to improve the speed is to remove the slowest drive, or replace it with a faster. It is always the slowest drive that limits the speed.
  • Regarding the use of PCI and PCI-express hosted controller boards... The use of PCI controllers with multiple drives will almost always result in slower parity checks. The 32-bit PCI bus lacks the bandwidth to keep up with modern hard disk drives. Even the rarer PCI-X 64-bit bus operates at 532 MB/s or 1064 MB/s only if used with a matching 64-bit PCI-X adapter card. In contrast, the more modern PCI-express bus (PCI-e) which is implemented on most motherboards and often in different bus widths (sometimes known as lanes) is much faster and is recommended when there are insufficient SATA ports on the motherboard to connect to all of the drives in the server. PCI-e x1 is generally OK for one or two drives; PCI-e x4 for up to eight drives; PCI-e x8 will not present problems; PCI-e x16 can also be used except on those motherboards that only allow a graphics card to be used in a x16 slot. As a general rule, for optimum performance the SATA ports on the motherboard should be filled before using those on adapter cards.

Remove SATA150 Jumper

Note: This tip only applies to the early Seagate SATA drives with the SATA I jumper installed.

Seagate (and some other SATA II drive manufacturers) install a small and almost hidden jumper, close to the drive connectors, that limits it to SATA150 speeds and the SATA I feature set. Remove the jumper to enable SATA300 speed, and the full SATA II features available with that drive. A tweezer or needle-nose is usually necessary to remove it.

The improvement here is usually very modest. You may not be able to see the improvement since there are other bottlenecks, and SATA at 1.5 Gbps is still a little faster than most individual drives can communicate, but that is no reason not to take them off. Who needs unnecessary handicaps? The fastest drives are now approaching the limits of SATA150 speeds.


Software Tips

Update nameserver & hosts file

Note: This tip may no longer be necessary (or only partially necessary) with recent versions of UnRAID.
Note: The following pair of lines are just an example. Although you are welcome to Cut & Paste them, they MUST be edited, customized for your situation.

echo nameserver 192.168.0.1 >/etc/resolv.conf
echo 192.168.0.100 tower >>/etc/hosts

Using the example above, change 192.168.0.100 to the local IP of your unRAID server. If necessary, change tower to your unRAID server's name, and change 192.168.0.1 to be the same as your gateway (normally, the location of your nameserver). And note that the first line has 1 right-angle (greater than symbol), the second line has 2.

Append this pair of lines to either the end of your go file or somewhere in the middle. In Linux, it is at /boot/config/go. In Windows, it will be at something like //Tower/flash/config/go.

For more information, see http://lime-technology.com/forum/index.php?topic=1247. --- (tip from BubbaQ)

To fix the issue of not being able to browse to http://tower, see http://lime-technology.com/forum/index.php?topic=9581.0.

Increase Read-Ahead Buffer

Note: the following tip is no longer needed as of unRAID v4.4-beta2, as it is now built in, with a value of 1024. See this post for more information. If you would like to use another value, such as 2048, or are still using an older version of unRAID, read and implement the following tip. See also this post. It mentions the need to loop through the drives, but see the command as given below, which does the same thing, in one line.

For an often large improvement in read performance, add the following line towards the end of your go script:

sleep 30; blockdev --setra 2048 /dev/md*

Why the sleep 30? You need to add a sleep command to ensure the array has been started by the emhttp program. It waits for 30 seconds before setting the read-ahead buffer.

In recent versions of unRAID, it is necessary to set the read-ahead for each device. Using /dev/md* as the argument to the blockdev command will do this automatically, as the shell expands the "*" to all of the existing devices.

For more information, see http://lime-technology.com/forum/index.php?topic=965.0. --- (tip from JoeL)

A simple way to add the above, from a console or Telnet prompt:

echo "sleep 30; blockdev --setra 2048 /dev/md*" >>/boot/config/go

For older versions of unRAID, older than v4.4-beta2, add the following, from a console or Telnet prompt:

echo "sleep 30; blockdev --setra 2048 /dev/md1" >>/boot/config/go

Vista SP1

If you are transferring files between your unRAID server and a Microsoft Vista machine, make sure you install Vista Service Pack 1 for a huge boost in network speed.

Other Vista tips

See the Best of the Forums, Solutions to Common Problems section for more Vista-only tips.

Experiment with Hard Drive Config

  • Don't assume that after plugging in a drive, it will automatically be optimally configured. Also, do not assume that recommended tweaks for others are ideal for your setup. You may have to experiment to find the best config for your hardware.
  • On many motherboards, the BIOS settings for SATA drives default to IDE Emulation mode to provide backward compatibility. But for full SATA support and and best performance, this should be changed to a true SATA support mode. Selecting AHCI is best, but if unavailable, then select a native SATA mode, possibly Enhanced SATA. Least desirable is an IDE or IDE Emulation or compatibility mode, but it may be necessary to gain access to the drive. Any IDE mode also carries an additional danger, in that in certain situations where one drive fails, a second drive may also be disabled, even though nothing is wrong with it. In most RAID setups (like UnRAID), having 2 drives disabled is a much worse problem than one disabled!
  • To understand which of your drives are the fastest and slowest, use the instructions on Check_Harddrive_Speed to benchmark each drive, as a base for analysis and to compare before and after performance.

Access from Another Linux box

You may need to disable DFS in your unRAID box to get the same speed of access from another Linux or Unix box as from a Windows box. See this post. See this thread for more information.

User Tunables

For reference, this was first introduced and explained here.

User testing (starting with this post plus subsequent discussion and testing) has found that significantly higher numbers can provide significantly better performance, though results may vary greatly depending on your particular hardware.

Here are the current defaults:

  • md_num_stripes=1280
  • md_write_limit=768
  • md_sync_window=384

For users with at least 1GB of RAM, the following (conservative) settings are suggested:

  • md_num_stripes=2048
  • md_write_limit=768
  • md_sync_window=1024

For UnRAID v5 releases, you can view and change these values on the Disk Settings page. For UnRAID v4 releases, please see the bottom of this post.

If you read the additional discussion in the thread mentioned above (here), you will find other recommendations, and how to further tweak them for best performance with your hardware. Just remember that if you begin to get OOM (Out Of Memory) errors or CPU stall errors, then decrease the first and third numbers. It has been suggested to change them by increments of 128.

Disable Windows Network Throttling

This tweak is not applied to UnRAID servers, but to Windows 7 stations. Unsure, but it may also apply to Windows Vista and Windows 8.

Tom reports here that after this 'fix', "this is actually the best performance I've ever measured". I can say the same. My write performance to parity protected drives (very subjectively estimated) is about 10% better, between 30MB/s and 35MB/s. My read and copy speed from UnRAID array drives (also very subjectively estimated) is about 50% to 70% better, generally between 60MB/s and 65MB/s. Unfortunately, this involves editing the Windows registry... If you are not confident in your registry editing skills, then you shouldn't. At least make a complete registry backup first, or a restore point.

Please see this page for explanation and instructions.

Tips for Better Operation

The tips above are for making your server perform faster, the following tips are for making your server perform better. For example, the first tip helps to keep your drives spun down.

Keep directory entries cached

The Linux kernel keeps the most recently accessed disk buffers and directories in memory, and flushes out the least recently accessed entries. If we can 'trick' the kernel into keeping our directory entries in that memory cache, then any directory scan will find what it is looking for already in memory, and not need to access the physical disk to get it. As a result, since the physical disk is not accessed, Linux will (after the defined time-out delay) let the physical drives spin down, and allow us to save on power costs, plus remove the drives as heat sources. This is especially useful when media files are spread across multiple drives, and a media player begins to scan for a particular media file to play. You want the scan to look at all of the relevant directories, but only spin up the one drive containing the desired media file.

There are two things that have to be done, in order to keep the directory entries cached: (may change ...)

  1. Inform the Linux kernel that you want it to put the highest priority on keeping directory entries in the cache, and avoid flushing them out. This is done by executing a statement similar to sysctl vm.vfs_cache_pressure=10. This really does help, but if very large files are transferred, the directory entries will still be dumped. For more information, see this post, and also search the vm.vfs_cache_pressure setting.
  2. Since the cache management decision process tries to keep the most recently accessed disk buffers and directory entries, we need to 'trick' it by constantly accessing the directories of the folders we want to keep in the cache, so that they will always appear to be the most recently accessed. An expert user has developed an easily customizable script called cache_dirs to do this.

The latest cache_dirs script is found here, with instructions as to how to customize it. By default, it handles all top level folders on all unRAID data drives and the Cache drive. And now the cache_dirs script automatically handles the setting of vm.vfs_cache_pressure=10.

Simple instructions: (using conservative values)

  • Remove any previous directory caching lines from your startup scripts, such as ls-r and sysctl vm.vfs_cache_pressure=0
  • Unzip the latest cache_dirs to the root of your flash drive
  • Edit your go script to include the following line, near the bottom:
/boot/cache_dirs  -w
  • This will go into effect on your next boot, and will cache all files and folders, with a scanning interval of between 1 and 10 seconds

The cache_dirs command is commonly customized, for example like this:

/boot/cache_dirs  -d  5  -m  3  -M  5  -w

This will cache all files and folders down to 5 levels deep, and will use a scanning interval of 3 to 5 seconds. You probably do not need to go that deep. You probably have folders that do not need to be cached, and happily, the cache_dirs command above can be easily customized to meet your needs. For example, you can add excludes to eliminate certain folders from being cached. Or you can add includes, that specify precisely which top level folders to be cached. But remember that if you specify any includes, then all other folders are excluded except for those you include. In other words, by default all top level folders are included, but if you use any includes, then by default all folders are excluded, except those you specify to be included. The primary guide to customizing it is here, but there are also hints on how to use some of its options here, here, here,and here.

To better understand the script itself and its parameters and how to customize it to your satisfaction, please see both the original post, the v1.2 post, the v1.3 post, and the v1.4.1 post.

My personal experience is that, unless you have more than 512MB, you can run out of RAM for processes, as it is hoarded for the disk buffer cache (and I have no idea how much is enough).

Cache_dirs already sets cache_pressure to 10. Even that is too low a value on my server with 512MB of RAM if I'm trying to perform certain tasks. Even a "find" of my /mnt/data hierarchy can run out of memory. Compiling a large program does not stand a chance of completing. It is not fun running out of memory and trying to regain control of your server to stop it cleanly. Kids, do not try this at home... Before running memory intensive programs, set cache_pressure to 200.

If you are running addons with a limited amount of RAM, set it higher (eg. 200), at least to 100.

For older methods, using the user-developed ls-r script that depends on cron, see this and this. This method is deprecated, in favor of cache_dirs, but still useful for non-User Shares. See this for instructions on setting this up, and what to put into your go script.

Related threads: the first about directory caching, intro to vfs_cache_pressure, an intriguing tree walking solution, a reason for numerous drives spinning up after one large file copy, the ls-r thread, ask Linux gods why vfs_cache_pressure=0 does not guarantee staying cached, post about 45 second pruning timer on dentry cache, another tree walker tease!, instructions for both methods in go script, more ls-r discussion, more on cached directories, the latest on directory caching

Avoid filling drives >99%

All filesystems can run into fragmentation and resulting performance problems when drives fill all the way up. With ReiserFS in unRAID, it can result in extremely slow write speeds and/or 10 to 30 second pauses where the drive and network share becomes unavailable (and can cause a windows machine trying to write a file to become somewhat unresponsive during that period). These pauses are believed to be the kernel searching the filesystem tree for free blocks or perhaps it is the filesystem adding area to the superblock (Reference). Depending on sizes of the files stored, it it proposed to stay below about 95 to 98% on each drive.

Transferring Files Within the unRAID Server

See Transferring Files Within the unRAID Server

Personal tools