Jumbo Packets

Someone was trying to tell me how amazing they are and that they really help. So I decided to enabled them but I was a little confused and want to make sure this is right. There seems to be many different numbers you can put and I don't know if using a wrong value would cause some type of packet loss.

 

I have it set to 9014 MTU on my PC NIC (I only have 2 options, that was the larger number).

I have "ifconfig eth0 mtu 9014" set in my go file. I have no idea about this one, I just matched it with my PC NIC. Motherboard below.

 

Is there anything I should know, or can test, to make sure these settings are stable? Thanks.

I set mine in my \unraid\flash\config\network.cfg

_____________________

# Generated network settings

USE_DHCP=yes

IPADDR=

NETMASK=

GATEWAY=

MTU=9000

___________________

The "MTU=9000" sets the jumbo frames every time it boots up.  Just make sure your  network switch supports jumbo frames or you will not get anything if the packet is over 1500.  It is pretty straight forward and it really helps when I read from the unraid box.  I get 70-80MBytes/sec  reads and 30-40MBytes/sec writes.  Without jumbo frames my reads top out at 40 and writes top out at 30

 

Someone was trying to tell me how amazing they are and that they really help. So I decided to enabled them but I was a little confused and want to make sure this is right. There seems to be many different numbers you can put and I don't know if using a wrong value would cause some type of packet loss.

 

I have it set to 9014 MTU on my PC NIC (I only have 2 options, that was the larger number).

I have "ifconfig eth0 mtu 9014" set in my go file. I have no idea about this one, I just matched it with my PC NIC. Motherboard below.

 

Is there anything I should know, or can test, to make sure these settings are stable? Thanks.

 

If you use a switch or router it will need to support the jumbo frame size you are using.  After adjusting the MTU on both ends, query the adapters to verify the MTU has been changed:

 

unRAID, telnet in:

ip link show | grep mtu

 

Windows 7:

netsh interface ipv4 show subinterfaces

Windows XP:

netsh interface ip show interface

 

The MTU sizes should match, if not correct them.

 

To check if everything works:

unRAID (telnet):

ping -M do -s {MTU - 28} [iP address] -c5

If you chose an MTU of 9014, as you stated above, then 9014-28=8986.  The 28 bytes is for the IP and ICMP headers.

 

Windows:

ping -f -l {MTU - 28} [iP address]

(-l = lower case L)

Thanks for this i'm showing the MTU on both as 9014.

 

Do I need to change anything in my router or does it just need to "support" them? I know it does but I don't see any settings related to it.

Thanks for this i'm showing the MTU on both as 9014.

 

Do I need to change anything in my router or does it just need to "support" them? I know it does but I don't see any settings related to it.

 

Do the ping tests as shown in the previous post, if it works then you're gold.  There should be a setting in the router to specify the frame size for jumbo frames.  It will be on the LAN interface.

 

I've did some experimenting on this topic few weeks ago. My first unRAID setup (details in the sig) was giving me ~75-80MBps for reading and 35-40MBps for writing on disk shares from Windows 8 client. There was also a drop in the reading speed to ~50MBps after around 5GB of transfer. I had always wondered what was the limiting factor (disks, network, Windows setting). I did some research and found out that the Realtek based NIC on my motherboard did  not support jumbo frames properly. However I was doing some benchmarking on a new unRAID SFF-build and decided to test the over-the-network performance with it since it had all the new things: 5.0rc16c unRAID, 6Gbps sata-III ports etc.

 

The new setup has two WD Green 3TB as parity/data disks and 256GB Samsung SSD as cache disk. They are all connected to the 6Gbps ports of the ASRock FM2A85X-ITX. As preclear pre-read speed I was getting around 140MBps for the WD Greens and 400MBps for the SSD. The parity check speed is ~150MBps (slows down a bit to the end side). The Win8 client machine has several older SATA-II disks (Samsung 256GB) and one 80GB OCZ Vertex 4 SSD as the system disk.

 

I started from the optimal combination reading from the unRAID SSD Cache drive and writing to the Win8 SSD system disk. Read speed ~110MBps. I then did the same but writing to the 256GB SATA-II drives -> read speed 70-80MBps. Looking at the Win8 resource monitor the local SSD was not too loaded during the write but the SATA-II were maxed out. Then I switch to reading from the WD Green drive through disk share. The result was ~100MBps. Since the read speed is so close to the theoretical maximum of a 1Gbps network, I would not bother setting up jumbo frames. You just have to make sure you are using fast enough hard drives on the client side too.

 

The write speeds for the old and new system on the disk shares are fully limited by the unRAID and not by network. I was expecting to get much higher write speeds (~50-60MBps) from the new system but I was a bit disappointed when I saw disk share write speed in the same 35-40MBps range as the old system. I don't fully understand what unRAID actually does when writing (besides reading and writing to both the data and parity disks) but it somehow manages to level out faster drives' performance. The write speed to the cache disk is 110-115MBps which is once again so close theoretical maximum that jumbo frames just aint't worth the hassle.

Have a look at ethtool which I now believe is included with unRAID. Here's the some info from the wiki page:

http://lime-technology.com/wiki/index.php/Console#Console_Commands_for_Networking and a discussion of it in the forum:http://lime-technology.com/forum/index.php?topic=2109.msg15568#msg15568

 

It can help you isolate testing to just the networking portion of your system.

I don't fully understand what unRAID actually does when writing (besides reading and writing to both the data and parity disks) but it somehow manages to level out faster drives' performance. The write speed to the cache disk is 110-115MBps which is once again so close theoretical maximum that jumbo frames just aint't worth the hassle.

when writing to the array, it must first READ both drives involved (the data disk and the parity disk) to get the current contents of the sectors involved, and then it can write those same sectors with the new data and the calculated parity.  Before it can write those sectors, the disk platters MUST both spin one complete revolution to get the read/write head back to the same sector so it can be written.

 

Your write speed is completely limited by the rotational speed of the platters involved.   If 7200RPM disks are involved, the speed writing will be 1/3 faster than if 5400 RPM disks are involved, simply because they can rotate the disk faster.

 

For best performance writing you want all 7200 RPM disks.  about 35/40 MB/s is about the best you'll get for sustained writing.  Write speed is limited by the rotational speed of the lowest disk involved if disk RPM speeds are different.

 

Joe L.