NFS: copying from unRaid, if the transfer rate is high enough, the client process hangs

I experience two problems with nfs.

1) When copying from unRAID, if the data transfer rate is high, the client process hangs.  This does not happen if I slow the network speed down to 100Mb/s.  I think that it is possible that you cannot transfer data fast enough between your sites.  I can overcome this problem by switching from tcp to udp.

2) When writing to unRAID, nfs can report a 'stale nfs handle'.  I think that rsync does not generate the conditions which provoke this error.

1) When copying from unRAID, if the data transfer rate is high, the client process hangs.  This does not happen if I slow the network speed down to 100kb/s.  I think that it is possible that you cannot transfer data fast enough between your sites.

Actually I have gbit between the servers as this rsyncing is done when the servers are local.  It happens blindingly fast for the amount of data being verified. (10 min per full 2TB drive provided its mostly in sync already).  Its in the order of 10X faster than the CWRSYNC we use to back the Windows workstations up to the main server, Server1 over the same gbit network. 

(6 windows computers back up to Server1.  Server1 is mirrored onto Tower1 and Tower2 which live offsite but are local when the mirroring takes place)

1) When copying from unRAID, if the data transfer rate is high, the client process hangs.  This does not happen if I slow the network speed down to 100kb/s.  I think that it is possible that you cannot transfer data fast enough between your sites.

 

Actually I have gbit between the servers as this rsyncing is done when the servers are local.  It happens blindingly fast for the amount of data being verified. (10 min per full 2TB drive provided its mostly in sync already).  Its in the order of 10X faster than the CWRSYNC we use to back the Windows workstations up to the main server.  (6 windows computers back up to Server1.  Server1 is mirrored onto Tower1 and Tower2 which live offsite but are local when the mirroring takes place)

That sounds like an awesome setup, have you got a link to a build thread for any of these systems? Would love to see some pics but this thread probably isn't the place for that. I'm working towards running two unRAID servers long term I think, along with a dedicated VM machine.

That sounds like an awesome setup, have you got a link to a build thread for any of these systems? Would love to see some pics but this thread probably isn't the place for that. I'm working towards running two unRAID servers long term I think, along with a dedicated VM machine.

They really aren't worth documenting yet.  The main server is housed in a Norco 4224, but the other 2 backup servers need chassis that work better.  I am looking for something with handles on top, but can handle 10-15 drives.  Here is what I am using now.  Handle is nice, for picking them up and taking them offsite.  But, not the best for loading up with unraid drives.

http://www.newegg.ca/Product/Product.aspx?Item=N82E16811154094

I wanted to try replicating nfs problems with another distro on my client, other than Ubuntu.  I wanted to use Slackware, but couldn't find an easily bootable image, using a 3.x kernel.  So, searching for a distro which would be easy to boot and uses a 3.x kernel, I came across Mageia RC2, which is using 3.3.4 kernel.  I booted Mageia, mounted my unRAID share manually (not autofs, as someone suggested that autofs can cause problems of its own) and then tried to copy a 16GB movie (.mkv) file.  This hung at around 85MB exactly as it does in Ubuntu.

Mageia is based on Mandriva, not Debian, as Ubuntu is.  So, if the fault is not in unRAID/3.0.31 kernel, it must be something else which is common between the two environments.  If it was a filesystem corruption on my desktop client why would a copy using udp protocol succeed?

I've also tried performing the copy from the command line in Ubuntu, eliminating Nautilus.  This also hangs in the same way.

I will try to reproduce the 'stale nfs file handle' problem too, but this is not as easily repeatable as the hang.

Copied from the RC1 thread:

Yes, it occurs with both disk and user shares.  I have two nautilus windows open, one on an unRAID share, and one on a local device.  I drag a file from the unRAID share to the local device.  It starts copying (the test I've just done with a user share reports 116.7MB copied) but then hangs up.  I will try doing the same thing from command line.

As far as I can tell, this happens with any version of unRAID using a 3+ kernel.  As I've stated before, there is no obvious clue in the syslog.

Edit:

A little more information:

I tried copying to a laptop (also running Ubuntu 12.04), but using a wireless connection.  This doesn't hang.

I then tried throttling my desktop back to 100Mb, and this also copies to completion ... slowly.

It is also clear that the server is not affected by the problem ... I just need to reboot the client (or use another client).  Perhaps this is a fault in the Ubuntu ethernet drivers and it's just that the 3.x kernels in unRAID are able to push data down the cable more quickly than the 2.x kernels do?

I do know that the raw bandwidth between my unRAID server and my main Ubuntu desktop achieves more than 110 MB/s according to iperf.

Also, copies to the server complete without problem ... at 80+MB/sec to cache drive.

Copied from RC2 thread:

Okay, I believe that I have a work-around for the hanging file transfers.

If I mount the network shares as udp, rather than the default nfs, then file transfers from unRAID do not hang.

Testing with transferring a single 16GB file from unRAID to my Ubuntu desktop, I get an average transfer rate well in excess of 90MB/s (I saw a peak of over 97MB/s).

I have tried other suggested work-arounds (async/sync etc), but only the udp option seems to offer a solution (other than artificially slowing down the transfers).

This means that I can now move on to current unRAID releases, but I do still regard this as a work-around solution.

Questions:

1) Why does tcp not work?

2) Why is there no problem with unRAID 5b11, and earlier (in other words, v2.x kernels)?

3) Does the blame lie with unRAID or with Ubuntu?

All current test are performed with Ubuntu 12.04 (3.2.0-24 kernel) and running AMD (64bit) mode.

Also, it seems that file transfers to unRAID are a little slower using udp than they were with tcp, I'm getting around 75MB/sec compared with 95+ previously - I'm not sure whether this might be expected or not.