kontu Posted June 6, 2013 Share Posted June 6, 2013 Hey All, In an effort to resolve some transport endpoint failures resulting in all my shares being unavailable, I've been trying to manipulate my ulimit settings to see if that will resolve it. While I can set them manually during runtime and restart emhttp afterwards, after a short while they get set back to what must be defaults somewhere. Additionally, trying to set them in the go file as below does not seem to have any effect after a reboot. It was a solution for 5.x I found in a couple threads (with a couple other ulimit settings modified). I've modified the settings that Plex exports for ulimit in /boot/config/plugins/plexmediaserver/settings.ini as well - still no luck anywhere. Even disabling Plex entirely does not affect the ulimit settings, so I think I've ruled that out at least. Does anyone have any ideas or suggestions on what to try? Any additional information you want on why manually setting ulimit seems not to be sticking for long? From my /boot/config/go: ulimit -n 30000;ulimit -s 10000;ulimit -l 3000;/usr/local/sbin/emhttp & ulimit -a that it keeps resetting to core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 129458 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 20000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 129458 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Link to comment
kontu Posted June 7, 2013 Author Share Posted June 7, 2013 Just documenting more steps / data as I work through this. Reboots are performed after each modification unless otherwise noted Have had this on both 5.0rc12a and 5.0rc13 Plex packages tested: 0.9.6.3 and 0.9.7.22.511 (latest) NFS is disabled period (for testing. I would like to use it later but we'll see once I get things working) SMB is the only sharing protocol I have enabled and am exporting with Plex Library is stored on: /mnt/user/plex/library Plex Temp is store on: /mnt/user/plex/temp I have all my drives configured to never spin down so I just keep it all on a user share stored on the array 8 drives in array with one parity - all 3TB WD reds Below is my entire go script with the settings I've been trying to incorporate getting Plex to not trigger a Transport Endpoint not connected error It includes oom killer fixes (though memory has not been an issue - 16GB in the system, meminfo generally showing >14-15GB Free at all times, even during copies) #!/bin/bash # Start the Management Utility sysctl fs.inotify.max_user_watches=100000 ulimit -n 30000;ulimit -s 10000;ulimit -l 3000;/usr/local/sbin/emhttp & /boot/unmenu/uu cd /boot/packages && find . -name '*.auto_install' -type f -print | sort | xargs -n1 sh -c pgrep -f "/usr/local/sbin/emhttp" | while read PID; do echo -1000 > /proc/$PID/oom_score_adj; done pgrep -f "/usr/sbin/smbd" | while read PID; do echo -1000 > /proc/$PID/oom_score_adj; done pgrep -f "in.telnetd" | while read PID; do echo -1000 > /proc/$PID/oom_score_adj; done pgrep -f "/usr/local/sabnzbd/SABnzbd.py" | while read PID; do echo 1000 > /proc/$PID/oom_score_adj; done Reboot with original line in my /boot/config/go script just to verify its still being ignored: ulimit -n 30000;ulimit -s 10000;ulimit -l 3000;/usr/local/sbin/emhttp & core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 129458 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 20000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 129458 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Dropped my ulimit valutes to either below defaults or super close, just to verify it wasn't the values I was using causing a problem: ulimit -n 15000;ulimit -s 8000;ulimit -l 66;/usr/local/sbin/emhttp & core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 129458 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 20000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 129458 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Clearly that's not it, what the values are doesn't matter. Went back to original line in the go script. Decided to see if there were any files setting ulimit on the flash drive:[\b] grep -ir ulimit /boot/* Only hits were files I already knew about (/boot/config/go, /boot/config/plugins/plexmediaserver/exportsettings, /boot/config/plugins/plexmediaserver/settings.ini, and /boot/packages/airvideo-unmenu-package.conf - though this last one was trying to set ulimit -v to 250000, which is unlimited) So now I need to determine how to best get it set system wide (and stay) Added my three ulimit lines to /boot/config/ident.cfg in an effort to hook into samba booting (Full file below) Immediately following a reboot things are looking promising, array is not even started up at this time. Data from Thu Jun 6 21:27:01 EDT 2013 (right after a reboot) core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 129458 max locked memory (kbytes, -l) 3000 max memory size (kbytes, -m) unlimited open files (-n) 30000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10000 cpu time (seconds, -t) unlimited max user processes (-u) 129458 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Decided to wait a little while and check to see if ulimit would set itself down again (like it does when I just manually run the commands during runtime). I have still not chosen to start the array yet. After 10 minutes of waiting (Thu Jun 6 21:37:02 EDT 2013), the ulimits had stayed core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 129458 max locked memory (kbytes, -l) 3000 max memory size (kbytes, -m) unlimited open files (-n) 30000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10000 cpu time (seconds, -t) unlimited max user processes (-u) 129458 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Decided to start the array at this time. Plex decided it was important enough to start running as well (I may have not disabled it prior). Waited till Thu Jun 6 21:41:28 EDT 2013, verified ulimits were still set how I wanted it with ulimit -a, Went into plex and told it to start refreshing my TV folder (~33 shows, total count of 2120 combined folders + files) All it really needs to do is download and store metadata for the shows. I've chosen not to add my Movies to the library as I've reset my Plex install a few times, and the shows alone have been enough to trigger the problem so far in past attempts. Annnd apparently the ulimits are *not* the main source of my problem. Thu Jun 6 21:45:13 EDT 2013 - -bash: cd: /mnt/user: Transport endpoint is not connected And all my user shares are unaccesible like before. Tail of my syslog, starting from the entry where samba started when I enabled the array: Jun 6 21:37:35 metacortex logger: Starting Samba: /usr/sbin/nmbd -D Jun 6 21:37:35 metacortex logger: /usr/sbin/smbd -D Jun 6 21:37:35 metacortex emhttp: shcmd (58): ps axc | grep -q rpc.mountd Jun 6 21:37:35 metacortex emhttp: _shcmd: shcmd (58): exit status: 1 Jun 6 21:37:35 metacortex emhttp: shcmd (59): /usr/local/sbin/emhttp_event svcs_restarted Jun 6 21:37:35 metacortex emhttp_event: svcs_restarted Jun 6 21:38:34 metacortex su[7544]: Successful su for unraid-plex by root Jun 6 21:38:34 metacortex su[7544]: + root:unraid-plex Jun 6 21:40:01 metacortex su[7647]: Successful su for unraid-plex by root Jun 6 21:40:01 metacortex su[7647]: + root:unraid-plex Jun 6 21:40:56 metacortex sshd[7778]: Accepted password for root from 192.168.1.120 port 52381 ssh2 Jun 6 21:45:59 metacortex emhttp: get_filesystem_status: statfs: /mnt/user/movies Transport endpoint is not connected Jun 6 21:45:59 metacortex emhttp: get_filesystem_status: statfs: /mnt/user/mysql Transport endpoint is not connected Jun 6 21:45:59 metacortex emhttp: get_filesystem_status: statfs: /mnt/user/plex Transport endpoint is not connected Jun 6 21:45:59 metacortex emhttp: get_filesystem_status: statfs: /mnt/user/tv Transport endpoint is not connected Jun 6 21:45:59 metacortex emhttp: get_filesystem_status: statfs: /mnt/user/xbmc Transport endpoint is not connected Jun 6 21:45:59 metacortex pms: Stopping Plex... Jun 6 21:46:06 metacortex pms: Stopping Plex ... forced! Not a single error leading up to it until suddenly I can't access any shares anymore. ulimit -a shows everything still set as I wanted it: core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 129458 max locked memory (kbytes, -l) 3000 max memory size (kbytes, -m) unlimited open files (-n) 30000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10000 cpu time (seconds, -t) unlimited max user processes (-u) 129458 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Neither restarting samba nor stopping and starting the array bring the shares back up - I have had to do a full reboot every time I'm suspecting I should now rename my thread / start a new one / latch on to an existing as I've at least gotten around the ulimit settings not sticking (by shoving them into /boot/config/ident.cfg). My core problem to resolve is Plex triggering the shares to be unavailable and the Transport Endpoint not connected errors. Will most likely start driving a thread on the Plex forums as well (existing searches have only yielded things I have already tried changing) **** ADDING MORE NOTES**** More notes - I kept working. My .../plexmediaserver/settings.ini #START_CONFIGURATION #Set autostart with array ENABLED="false" RUNAS="unraid-plex" #Set home of Plex Media Server PLEX_MEDIA_SERVER_HOME="/usr/local/plexmediaserver" #Set home for Plex metadata PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR="/mnt/user/plex/library" #the number of plugins that can run at the same time PLEX_MEDIA_SERVER_MAX_PLUGIN_PROCS=6 #ulimit -s PLEX_MEDIA_SERVER_MAX_STACK_SIZE=10000 #ulimit -l #PLEX_MEDIA_SERVER_MAX_LOCK_MEM=3000 #ulimit -n PLEX_MEDIA_SERVER_MAX_OPEN_FILES=30000 #where the mediaserver should store the transcodes PLEX_MEDIA_SERVER_TMPDIR="/mnt/user/plex/temp" #STOP_CONFIGURATION Found somewhere that some people have needed to change the RUNAS to root instead....Decided to give it a shot as it might be ignoring my ulimits (Just noting my change to /boot/config/ident.cfg is still keeping my ulimits set - hurrah) Started Plex. Started a media refresh again on the TV dir @ Thu Jun 6 22:10:01 EDT 2013 Well that was shortlived - Thu Jun 6 22:11:36 EDT 2013 - Transport endpoint not connected Thought to capture `ls -l` from /mnt while things were down - /bin/ls: cannot access user: Transport endpoint is not connected total 0 drwxrwxrwx 8 nobody users 176 2013-05-30 22:06 disk1/ drwxrwxrwx 6 nobody users 128 2013-05-31 09:39 disk2/ drwxrwxrwx 4 nobody users 80 2013-05-29 22:28 disk3/ drwxrwxrwx 4 nobody users 80 2013-05-29 22:33 disk4/ drwxrwxrwx 4 nobody users 80 2013-05-29 22:37 disk5/ drwxrwxrwx 4 nobody users 80 2013-05-29 22:41 disk6/ drwxrwxrwx 4 nobody users 80 2013-06-05 11:22 disk7/ d??? ? ? ? ? ? user/ Not super helpful though. Will leave it running as Root for now anyways until I get it fixed (and will slowly back out changes at that time to find out exactly what is needed) Link to comment
kontu Posted June 7, 2013 Author Share Posted June 7, 2013 Enabled `shfsExtra= "-logging 3"` , reproduced. Just noticed I can't attach syslog as it is 156MB now..looking into alternatives where to shove it. Link to comment
kontu Posted June 7, 2013 Author Share Posted June 7, 2013 Compressed brought it down to 4.1 at least Syslog - https://dl.dropboxusercontent.com/u/3911145/syslog-2013-06-07.txt.zip Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.