get_filesystem_status: statfs: /mnt/user/ Transport endpoint is not connected

Hello,

I'm getting I was getting these "emhttp: get_filesystem_status: statfs: /mnt/user/${sharename} Transport endpoint is not connected" in my syslog.  Running RC11.  When it happens, the same error repeats for all shares on the system.  I also get a "Feb 12 22:14:57 tower kernel: nfsd: non-standard errno: -107" error also. 

• Windows Client:  The flashdrive is available via cifs, but all other shares get a "Windows cannot access \\tower\${sharename}
  
• Linux Cifs Client:  The shares are mounted but show as empty on ls
  
• VMWare NFS Client: The share shows as inactive.
  

The web page thinks all is happy (green).  I saw other users had the same problem, and a note in the rc10 release notes that says "- linux: use fuse version 2.9.2 in effort to solve "transport endpoint not connected" problem".  The problem is very reproducible, in that anything that touches a directory with a large number of files in it triggers this.

So, some oddities:

On the command line trying to just list /mnt/user:

 ls /mnt/user
/bin/ls: cannot access /mnt/user: Transport endpoint is not connected

I've got enough memory I think (4GB)

root@tower:~# free
             total       used       free     shared    buffers     cached
Mem:       4147044     252528    3894516          0       1056     219712
-/+ buffers/cache:      31760    4115284
Swap:            0          0          0

I was only running open_vm_tools for RC11 and simplefeatures.  First I removed openvmtools, no change.

Then I removed simple features, and stopped getting the "emhttp: get_filesystem_status: statfs: /mnt/user/${sharename} Transport endpoint is not connected" errors.  (As I was typing this I got one for each share.  Before they were spamming the syslog.)  Now I only get the nfsd errors:

Feb 12 22:26:57 tower kernel: nfsd: non-standard errno: -107
Feb 12 22:27:33 tower last message repeated 3 times
Feb 12 22:28:45 tower last message repeated 6 times

The problem still occurs though and is reproducible.

I'm not sure what to test next.  And I'll just stop trying to get to that directory, but what's the point in that...  Any advice? 

Syslog attached.

syslog.txt

Hoping it's a good thing that no one has responded, means hopefully no one else is currently having this issue. 

I tried to access the data on the /mnt/disk#/dir_with_lotsa_files/...  And that works. 

So it's only on the /mnt/user/dir_with_lotsa_files/... that breaks.  And when that breaks, it basically knocks unRaid on it's butt.

I get the "Transport endpoint is not connected" message whenever I have Plex running and try to do anything too adventurous at the same time. I'm hoping this is something that will get ironed out in a future build!

Send a message to Lime. He may have some tests for you since you can reproduce at will.

Getting exactly the same error here with RC11 with Plex installed.

Back to RC8 for me.

I get it also when I extracted newznab's nzb torrent archive to a directory on my cache drive - too many files I assume?

Myk

It does have to do with too many files, although I don't have any idea how many too many are .

So, if I drop back to RC8 as I saw mentioned, would that resolve this issue?

And on sending a PM to Tom, doesn't he read these?  If so then why bother him with a PM?

Thanks,

Al

It does have to do with too many files, although I don't have any idea how many too many are .

 

So, if I drop back to RC8 as I saw mentioned, would that resolve this issue?

 

And on sending a PM to Tom, doesn't he read these?  If so then why bother him with a PM?

 

Thanks,

 

Al

Did you include a link to this thread?

Did you include a link to this thread?

I haven't PM'ed him, that was my point.  I don't want to bother him with a PM if he reads the messages in the "unRAID OS 5.0-rc" section of the forum.

Still think it's worth a PM?

i am getting the same issue. no plex however.

I have the same issue on 5.0-rc8a (with Plex running). So not sure downgrading will help you...

Documented with logs on this thread: http://lime-technology.com/forum/index.php?topic=22971.45

Strange that I see nothing about memory in my syslog...

I am also getting the Transport endpoint not connected whenever I try an rsync to an external drive or try to run Crashplan. It does indeed seem that the number of open files in unraid 5.x have something to do with this. My initial thought was that I didn't have enough RAM (4GB), so I upgraded to 8GB and checked that I never got close to using all of it. But the problem persist. The web interface also seems to crash at that point. I am following a suggestion in another related post to see if

sed -i "sX^/usr/local/sbin/emhttpXulimit -n 20000;/usr/local/sbin/emhttpX" /boot/config/go

changes the behavior. I never had any of these issues in unraid 4.x. A fix would be greatly appreciated. Ok, tried the above, which had been suggested by Joe L., but it didn't help.

I am also getting the Transport endpoint not connected whenever I try an rsync to an external drive or try to run Crashplan. It does indeed seem that the number of open files in unraid 5.x have something to do with this. My initial thought was that I didn't have enough RAM (4GB), so I upgraded to 8GB and checked that I never got close to using all of it. But the problem persist. The web interface also seems to crash at that point. I am following a suggestion in another related post to see if

 

sed -i "sX^/usr/local/sbin/emhttpXulimit -n 20000;/usr/local/sbin/emhttpX" /boot/config/go

 

changes the behavior. I never had any of these issues in unraid 4.x. A fix would be greatly appreciated. Ok, tried the above, which had been suggested by Joe L., but it didn't help.

"Transport endpoint not connected" in this context means that fuse/shfs has crashed.

Some questions:

a) what version unRaid?  -rc11 contained an important fix for one possible cause of this problem.

b) is NFS enabled on Settings/Network Services/NFS?  If so, please disable and retry test.

c) I'm interested in that you can use rsync to reproduce this problem.  Please post the details.

thank you

Here's another voice in this crowd.  I get this error whenever I have Plex running.  Updates of the plex database seem to trigger the problem.  I've looked at both the system log & the plex log and can't find any error messages.

This is on rv11 of unraid and pms version  0.9.7.22.511

I have no other addons running.

I'd be happy to test if you can give some guidance.

See the other post about transport errors ..... FYI Disabling NFS does not make any difference in my case.

 

"Transport endpoint not connected" in this context means that fuse/shfs has crashed.

 

Some questions:

a) what version unRaid?  -rc11 contained an important fix for one possible cause of this problem.

b) is NFS enabled on Settings/Network Services/NFS?  If so, please disable and retry test.

c) I'm interested in that you can use rsync to reproduce this problem.  Please post the details.

 

thank you

Just got back from vacation. Sorry for not responding earlier. The version I was running back when I posted the issue was rc11. This morning I upgraded to rc12a, but nothing changed. I now also get the same problem when mounting an AFP share that stores my music files and iTunes scans the library. All of a sudden, the drive disappears and when I log into the server via SSH I can see that the /mnt/user share has been 'disconnected', i.e. is no longer a pointer to a directory. This is getting to a point where the server no longer performs the most basic tasks that it did well under 4.x. Thinking about going back to that version somehow unless we can find the reason for this behavior. This time, no other services were running and the server had been rebooted. Anything I can do to diagnose this issue, please let me know. NFS is not enabled, but AFP is. I found that iTunes doesn't like SMB shares, which is the only reason for me to use AFP. Let me know if you think that this might be the cause for the problem.

Update: I tried mounting the Music share via SMB and was able to add additional tracks to the iTunes library w/o experiencing the disconnection problems that I can reproduce when mounting the share via AFP. Still, in the past, AFP hadn't caused any problems for me until recent upgrades to unraid. This shouldn't explain why rsync called from the command line when logged into the server via SSH also causes the 'endpoint disconnect' problem, or why running a headless Crashplan server now causes problems. I think the answer will be somehow related to the number of opened files.

Update 2: Spoke too soon about the problem not occurring when using SMB shares. I disabled AFP globally and mounted the music share via SMB. That worked for a while but in the end also caused the same 'endpoint disconnected' issue which requires a restart of the server. It's getting to a point where I can no longer use the server even for the most basic things. Does anybody have a recommendation for what to try or how to gracefully downgrade to a more stable version? I am also open to switching to a commercial NAS from Synology or QNAP, if these work.

Update 3: Ran Memtest all day long to make sure that I didn't have a bad stick in my system, but no errors showed up after 2 passes.

Having same issues on rc-12.

When browsing my FLAC directory via AFP, the shares will become unavailable, producing similar output in the syslog.

Haven't tried this over SMB, but while watching media over the network via SMB, it becomes unavailable when this happens (makes sense -- the shares as a whole disappear).

Apr 21 13:51:46 Tower afpd[8319]: acl = acl_get_file(path, ACL_TYPE_ACCESS) failed: Transport endpoint is not connected
Apr 21 13:51:46 Tower afpd[8319]: posix_acls_to_uaperms(path, st, ma) failed: Transport endpoint is not connected
Apr 21 13:56:05 Tower afpd[9365]: volume "Backup" does not support Extended Attributes, using ea:ad instead
Apr 21 13:56:05 Tower afpd[9365]: volume "Dropbox" does not support Extended Attributes, using ea:ad instead
Apr 21 13:56:05 Tower afpd[9365]: volume "Media" does not support Extended Attributes, using ea:ad instead
Apr 21 13:56:05 Tower afpd[9365]: volume "TimeMachine" does not support Extended Attributes, using ea:ad instead
Apr 21 13:56:15 Tower emhttp: get_filesystem_status: statfs: /mnt/user/Backup Transport endpoint is not connected (Other emhttp)
Apr 21 13:56:15 Tower emhttp: get_filesystem_status: statfs: /mnt/user/Dropbox Transport endpoint is not connected (Other emhttp)
Apr 21 13:56:15 Tower emhttp: get_filesystem_status: statfs: /mnt/user/Media Transport endpoint is not connected (Other emhttp)
Apr 21 13:56:15 Tower emhttp: get_filesystem_status: statfs: /mnt/user/Network Trash Folder Transport endpoint is not connected (Other emhttp)
Apr 21 13:56:15 Tower emhttp: get_filesystem_status: statfs: /mnt/user/Temporary Items Transport endpoint is not connected (Other emhttp)
Apr 21 13:56:15 Tower emhttp: get_filesystem_status: statfs: /mnt/user/TimeMachine Transport endpoint is not connected (Other emhttp)
Apr 21 13:56:32 Tower emhttp: shcmd (72): /usr/local/sbin/emhttp_event stopping_svcs (Other emhttp)
Apr 21 13:56:32 Tower emhttp_event: stopping_svcs (Other emhttp)
Apr 21 13:56:32 Tower emhttp: Stop AVAHI... (Other emhttp)

And it continues with emhttp attempting to shut down the array. Restarting the array tends to bring the Shares back -- not always necessary for a reboot.

Running Rc-8 and have the same problem, running plex, sickbeard, couchpotato, sabnzbd, crashplan, plexupdater and DenyHosts.

Running Rc-8 and have the same problem, running plex, sickbeard, couchpotato, sabnzbd, crashplan, plexupdater and DenyHosts.

lsof | wc -l

ulimit -a

cat /proc/sys/fs/inotify/max_user_watches

Running Rc-8 and have the same problem, running plex, sickbeard, couchpotato, sabnzbd, crashplan, plexupdater and DenyHosts.

Just for kicks, what do you see for:

lsof | wc -l

ulimit -a

cat /proc/sys/fs/inotify/max_user_watches

In case this is also applicable to me...

lsof | wc -l 
453

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15681
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 15681
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

cat /proc/sys/fs/inotify/max_user_watches
100000

lsof |wc -l

lsof: WARNING: can't stat() fuse.shfs file system /mnt/user

      Output information may be incomplete.

1717

------------------------------------------------------------------------------------------------------

ulimit -a

core file size          (blocks, -c) 0

data seg size          (kbytes, -d) unlimited

scheduling priority            (-e) 0

file size              (blocks, -f) unlimited

pending signals                (-i) 62251

max locked memory      (kbytes, -l) 64

max memory size        (kbytes, -m) unlimited

open files                      (-n) 1024

pipe size            (512 bytes, -p) 8

POSIX message queues    (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time              (seconds, -t) unlimited

max user processes              (-u) 62251

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

------------------------------------------------------------------------------------------------------

cat /proc/sys/fs/inotify/max_user_watches

8192

Did u get anything out of this information?

Did a reboot, and did the commands again.

lsof |wc -l

1872

----------------------------------

ulimit -a

core file size          (blocks, -c) 0

data seg size          (kbytes, -d) unlimited

scheduling priority            (-e) 0

file size              (blocks, -f) unlimited

pending signals                (-i) 62251

max locked memory      (kbytes, -l) 64

max memory size        (kbytes, -m) unlimited

open files                      (-n) 1024

pipe size            (512 bytes, -p) 8

POSIX message queues    (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time              (seconds, -t) unlimited

max user processes              (-u) 62251

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

----------------------------------

cat /proc/sys/fs/inotify/max_user_watches

8192

/M

Did u get anything out of this information?

(and most likely, so is the max_user_watches for people with marge media and file collections)

And to fix that?