802.3ad Link Aggregation (LACP, a/k/a teaming or bonding)

SMB2 in the latest 5.0 betas can saturate GigE, so Link Aggregation is now much more desirable.  So I wanted to start a thread on LACP.

I'm getting a PROCURVE 1810G-24 to start playing with it.  It is fanless (so silent) and will provide POE to downstream devices.  I've used other higher-end Procurve boxen in the past with good results.  Reviews of this one seem good.

There is an 8-port version that is cheaper (about $150) but will not supply POE, and I really think I will regret only having 8 bondable ports in the long run.

Comments?

I got it working, and it turns out 802.3ad will not be a solution.  Based on some good info at:

http://www.cyberciti.biz/howto/question/static/linux-ethernet-bonding-driver-howto.php

I will quote:

The 802.3ad mode does have some drawbacks: the standard mandates that all devices in the aggregate operate at the same speed and duplex.  Also, as with all bonding load balance modes other than balance-rr, no single connection will be able to utilize more than a single interface's worth of bandwidth.

This was exactly what I saw.... even with a 3-NIC team, a single workstation copy only used on NIC for inbound packets and another for outbound.

Balance-rr is the only teaming protocol that stripes the data across multiple NICs.  That's not hard to do on Linux, but I'm still looking for a way to do in on Windoze.

balance-rr: This mode is the only mode that will permit a single TCP/IP connection to stripe traffic across multiple interfaces. It is therefore the only mode that will allow a single TCP/IP stream to utilize more than one interface's worth of throughput.  This comes at a cost, however: the striping often results in peer systems receiving packets out of order, causing TCP/IP's congestion control system to kick in, often by retransmitting segments.

I found one way to do balance-rr under windows... by routing everything through a Linux VM guest.  :'(

http://bora.bilg.in/blog/04/multi-wan-load-balancing-under-windows-with-pfsense

I found one way to do balance-rr under windows... by routing everything through a Linux VM guest.   :'(

 

http://bora.bilg.in/blog/04/multi-wan-load-balancing-under-windows-with-pfsense

Freaking brilliant  

I can't do samba mounts from one unRAID box to another with B14... something is missing.... so I can't test SMB directly.

I was able to set up balance-rr on 2 unRAID boxes, and using iperf, I got 150MB/sec  of throughput.

When I added a simultaneous SMB2 file xfer from a Windows box, I got a combined 215MB/sec between the SMB2 xfer and iperf.

This was pounding the CPU on the smaller server pretty hard.

I need to get smbmounting working with B14 (it appears to be broken at present).... after that, I can tell more.

Since balance-rr is the only LAG mode that stripes data so you can get more than 1 NIC pumping data on the same connection, I concentrated on that.

I set up large (5GB) ramdisks to handle 4GB file copies on both workstation and unRAID.

Getting balance-rr working on unRAID was fairly simple.  2Gb/sec was definitely possible.

I could not get high speed transfers going from one unRAID box to another, since unRIAD only has smb2 support as a server, not as an smb client.

On the workstation side, I easily got 802.3ad working using the HP Procurve switch, but that does not stripe data across 2 NICs, so it is limited to the throughput of 1 NIC for copying to/from unRAID.  But it did work, with 802.3ad on the workstation side of the Procurve, and the unRAID side doing balance-rr.

I had a lot of trouble getting balance-rr going on the workstation side.  The Intel teaming interface does not support it, so I had to abandon the Ihtel NICs.  The Realtek teaming utility does support balance-rr, but I could not get the *#$&#^ Realtek teaming working under Win7/64.  >

So I did a clean install of Win7/32, and did get the Realtek teaming working.

Reading from unRAID, I could burst close to 2Gb/sec.  ;D

I reliably did 170MB/sec sustained reading from unRAID with this configuration. ;D

But when I checked CPU utilization on unRAID, smbd had pegged one core of the dual core CPU at 100%, while the other core was doing nuttin.  I would venture that without that CPU bottleneck, I would have been able to saturate the 2Gb/sec link and read at well over 200MB/sec.

Writing to unRAID, however, did not produce such good results..... about 65MB/sec max.  I was able to do better (100MB/sec) writing to unRAID with a single NIC, not teamed.

Writing, however, did not show the CPU bottleneck on unRAID.  Both cores were comfortably humming along at 30 to 40% utilization.

So more questions than answers.  Is smbd not multithreaded?  It sure seems to balance work between both CPUs when writing, but not when reading from unRAID.  Even going back to a single NOC in the workstation, reading at 100MB/sec is pegging one core of the CPU on unRAID, while writing at the same datarate does not.  Once again, all reading and writing is on RAMdisk.

Since Tom is compiling Samba himself, does the configure script or make file need tweaking?  I'm not up on Samba to know if the Samba 4.x tree addresses this issue or not.

A little more research reveals that, indeed, smbd is not multithreaded, and due to its nature, there really is no way for it to be so and not take a huge performance hit.

So back to the drawing board.

Interesting! Have you tried to write to unRAID server using NFS for Windows?

Waiting for NFS to be fixed in the next release of unRAID.

Any news on this topic?

No, still waiting for 5.x final.

I've confirmed that Intel NICs on Windows do NOT do mode 0 (balance-rr) Ethernet bonding.

Since that is the only mode where a single workstation can benefit from the aggregated bandwidth, you won't break 1gbps in file xfers involving a Windows box (unless some NIC vendor has some special utility for their NICs under Windows.)

So regardless of the fact that unRAID supports it, and I've been using a 2GB mode 0 bonded link for some time, it won't help if your clients are Windows, unless you have multiple workstations access unRAID at the same time.

Yeah... it sucks.

Sad news... Thanks for your work anyway.

... it won't help if your clients are Windows, unless you have multiple workstations access unRAID at the same time.

But, if you happen to use a sensible O/S, or you have a 10Gb link on your client, then you can benefit.  In any case, I would think that there are a significant number of users who would be happy to run multiple 1Gb clients at full speed.

Bubbaq and others,

Can you provide some details on how you got this working?  I'm looking at section 3.3 of the reference. How do I determine the correct modprobe command?

Thanks,

David

Got it balanced.

root@Micro:~# ifconfig
bond0     Link encap:Ethernet  HWaddr 00:50:43:01:14:c7  
          inet addr:192.168.11.49  Bcast:192.168.11.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:879 errors:0 dropped:0 overruns:0 frame:0
          TX packets:264 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:86644 (84.6 KiB)  TX bytes:46462 (45.3 KiB)

eth0      Link encap:Ethernet  HWaddr 00:50:43:01:14:c7  
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:450 errors:0 dropped:0 overruns:0 frame:0
          TX packets:132 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:46498 (45.4 KiB)  TX bytes:21257 (20.7 KiB)
          Interrupt:18 

eth1      Link encap:Ethernet  HWaddr 00:50:43:01:14:c7  
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:429 errors:0 dropped:0 overruns:0 frame:0
          TX packets:132 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:40146 (39.2 KiB)  TX bytes:25205 (24.6 KiB)
          Interrupt:18 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:560 (560.0 B)  TX bytes:560 (560.0 B)

root@Micro:~# 

But, if you happen to use a sensible O/S, or you have a 10Gb link on your client, then you can benefit.  In any case, I would think that there are a significant number of users who would be happy to run multiple 1Gb clients at full speed.

I'd like to know the unRAID configuration that supports multiple 1Gbe clients at full speed. Multiple does mean 250+MB/sec right?

Yes, 250+. You'll need a switch that supports LACP to bond the links on the switch.  Under Network Settings choose No but leave or add a DNS server address. This is usually your router. Add this to the go file:

ifconfig eth0 down
modprobe bonding mode=balance-rr miimon=100
modprobe sky2
modprobe tg3
ifconfig bond0 192.168.11.49 netmask 255.255.255.0 up
ifenslave bond0 eth0
ifenslave bond0 eth1
route add default gw 192.168.11.1 bond0

You'll need to customize this for your NICs and addressing. Version 5 should eventually have a GUI for this.

But, if you happen to use a sensible O/S, or you have a 10Gb link on your client, then you can benefit.  In any case, I would think that there are a significant number of users who would be happy to run multiple 1Gb clients at full speed.

 

I'd like to know the unRAID configuration that supports multiple 1Gbe clients at full speed. Multiple does mean 250+MB/sec right?

I meant the disks and controller side, without cache unRAID does about 60MB/sec, with cache it seems to be limited to single disk performance.

It is useful for concurrent client access or SSD or HW RAID under unRAID.

 

ifconfig eth0 down
modprobe bonding mode=balance-rr miimon=100
modprobe sky2
modprobe tg3
ifconfig bond0 192.168.11.49 netmask 255.255.255.0 up
ifenslave bond0 eth0
ifenslave bond0 eth1
route add default gw 192.168.11.1 bond0

 

Can you explain the need for including the marvel (sky2) and broadcom (tg3) drivers? Do they bring something to bonding?

 

ifconfig eth0 down
modprobe bonding mode=balance-rr miimon=100
modprobe sky2
modprobe tg3
ifconfig bond0 192.168.11.49 netmask 255.255.255.0 up
ifenslave bond0 eth0
ifenslave bond0 eth1
route add default gw 192.168.11.1 bond0

 

 

Can you explain the need for including the marvel (sky2) and broadcom (tg3) drivers? Do they bring something to bonding?

They are required to designate the NICS to be configured. Two NICs using a common driver require only a single entry.

 

ifconfig eth0 down
modprobe bonding mode=balance-rr miimon=100
modprobe sky2
modprobe tg3
ifconfig bond0 192.168.11.49 netmask 255.255.255.0 up
ifenslave bond0 eth0
ifenslave bond0 eth1
route add default gw 192.168.11.1 bond0

 

 

Can you explain the need for including the marvel (sky2) and broadcom (tg3) drivers? Do they bring something to bonding?

 

They are required to designate the NICS to be configured. Two NICs using a common driver require only a single entry.

After further testing I've determined that the following lines are not required:

modprobe sky2
modprobe tg3

Is this info still relevant?  Will be trying to do this with an Intel Pro/1000 on a system running the latest 5.x release in the very near future.

I'd like to know too.

I have a HP DL380 G6 running Server 2012R2/HyperV, and it has 4 NICs onboard.  I was looking at teaming them to the HP 2500 series switch.  2012R2 has a few different teaming features built into the OS these days, including LACP and some others.

For the Unraid box I have a 1st-gen i3 CPU and a single NIC, but I have a spare NC-series (I think) HP PCI ethernet card with 4 outlets that I was looking at installing.

In theory this should give me 4Gbit/sec between the Hyper-V box (and its VMs) and the Unraid box.  The Unraid box has an SSD for its cache drive.

Running Unraid 5 with Plex.  Haven't seen a need to upgrade to Unraid 6 yet.

Just wondering if this is feasible, or if we are still limited to single-NIC speeds between unraid and windows?