wdawgnc

The errors column is showing a Zero for all devices.

Something still isn't right. The system isn't at 100% CPU, the GUI is responsive and i can SSH in, but the mover has been running for 8.5 hours trying to move a 1GB file. I've attached another diagnostics file in case that helps anyone track down what's going on. Here's a quick snippet of the syslog Feb 4 00:00:01 MediaLibrary logger: mover started Feb 4 00:00:01 MediaLibrary logger: moving "TVShows" Feb 4 00:00:01 MediaLibrary logger: ./TVShows/Chicago P.D/Season 03/Chicago.P.D.-S03E13-Hit.Me-HDTV-720p.mkv Feb 4 00:00:01 MediaLibrary logger: .d..t...... ./ Feb 4 00:15:32 MediaLibrary kernel: mdcmd (61): spindown 0 Feb 4 00:15:32 MediaLibrary kernel: mdcmd (62): spindown 1 Feb 4 00:38:59 MediaLibrary kernel: mdcmd (63): spindown 7 Feb 4 02:18:43 MediaLibrary kernel: mdcmd (64): spindown 7 Feb 4 08:01:27 MediaLibrary avahi-daemon[17364]: server.c: Packet too short or invalid while reading response record. (Maybe a UTF-8 problem?) Feb 4 08:23:23 MediaLibrary kernel: mdcmd (65): spindown 0 Feb 4 08:23:23 MediaLibrary kernel: mdcmd (66): spindown 4 Feb 4 08:24:55 MediaLibrary sshd[20086]: Accepted none for root from 10.15.98.122 port 63304 ssh2 medialibrary-diagnostics-20160204-0828.zip

Update: The mover ran and didn't try to move the TVShows folder and so far the system hasn't hung up.

Excellent, thanks for the replies everyone, I appreciate the assistance. There were two files on the cache drive under the TVShows path. They were new as of this morning, but I think I'm following the logic of checking on the cache drive being clear. So, I manually ran the mover and it moved all of the TVShows files (and the folder itself) from the cache drive to the share. At this moment, the cache drive is empty. I verified that by SSH'ing in and using 'ls -la' on /mnt/cache I've re-enabled the mover so we can see if it hangs the system with the same syslog message about TVShows or not. I want the mover to execute via cron vs. manually so give me around an hour and I'll report back what happens. Thanks again

The TVShows share is not located on the cache drive, it wouldn't fit. Yes I have the share set to use the cache drive, as is the Movies share. The issue is, when the mover starts, it's trying to move TVShows/. It's not moving a show on the cache into/onto the TVShows share, it's trying to migrate the entire 8TB+ TVShows share - where it's moving it I have no idea. When it does this, the entire system locks up for 48+ hours and is unusable. Well I'm guessing that's what it's trying to do based on what I've seen in the syslog just before the lockups happen. Here's a log capture from last month when it happened ---=== tail of syslog begins here ===--- root@MediaLibrary:~# tail -35 /var/log/syslog Jan 2 18:03:36 MediaLibrary avahi-daemon[1555]: server.c: Packet too short or invalid while reading response record. (Maybe a UTF-8 problem?) Jan 2 18:05:46 MediaLibrary kernel: md: sync done. time=43545sec Jan 2 18:05:46 MediaLibrary kernel: md: recovery thread sync completion status: 0 Jan 2 18:20:47 MediaLibrary kernel: mdcmd (102): spindown 0 Jan 2 19:39:53 MediaLibrary kernel: sdc: sdc1 Jan 2 19:39:59 MediaLibrary kernel: sdc: sdc1 Jan 2 20:00:01 MediaLibrary logger: mover started Jan 2 20:00:01 MediaLibrary logger: mover finished Jan 2 21:30:31 MediaLibrary avahi-daemon[1555]: server.c: Packet too short or invalid while reading response record. (Maybe a UTF-8 problem?) Jan 2 22:00:01 MediaLibrary logger: mover started Jan 2 22:00:01 MediaLibrary logger: mover finished Jan 2 22:31:24 MediaLibrary kernel: mdcmd (103): spindown 0 Jan 2 22:31:25 MediaLibrary kernel: mdcmd (104): spindown 3 Jan 2 22:34:02 MediaLibrary kernel: mdcmd (105): spindown 1 Jan 2 22:34:04 MediaLibrary kernel: mdcmd (106): spindown 4 Jan 2 22:34:13 MediaLibrary kernel: mdcmd (107): spindown 2 Jan 2 23:34:39 MediaLibrary sshd[22350]: Accepted none for root from 10.15.98.122 port 56608 ssh2 Jan 2 23:45:57 MediaLibrary kernel: mdcmd (108): spindown 4 Jan 2 23:45:59 MediaLibrary kernel: mdcmd (109): spindown 7 Jan 2 23:46:03 MediaLibrary emhttp: shcmd (121): /usr/sbin/hdparm -y /dev/sda &> /dev/null Jan 2 23:46:05 MediaLibrary kernel: mdcmd (110): spindown 0 Jan 2 23:46:06 MediaLibrary kernel: mdcmd (111): spindown 2 Jan 2 23:49:10 MediaLibrary sshd[25586]: Accepted none for root from 10.15.98.122 port 57095 ssh2 Jan 3 00:00:01 MediaLibrary logger: mover started Jan 3 00:00:01 MediaLibrary logger: moving "TVShows" Jan 3 00:03:00 MediaLibrary sshd[25586]: Received disconnect from 10.15.98.122: 11: disconnected by user Jan 3 00:03:07 MediaLibrary sshd[22350]: Received disconnect from 10.15.98.122: 11: disconnected by user Jan 3 01:35:42 MediaLibrary sshd[16819]: Accepted none for root from 10.15.98.122 port 60914 ssh2 Jan 3 01:40:06 MediaLibrary sshd[16819]: Received disconnect from 10.15.98.122: 11: disconnected by user Jan 3 04:40:01 MediaLibrary apcupsd[9212]: apcupsd exiting, signal 15 Jan 3 04:40:01 MediaLibrary apcupsd[9212]: apcupsd shutdown succeeded Jan 3 04:40:04 MediaLibrary apcupsd[24946]: apcupsd 3.14.13 (02 February 2015) slackware startup succeeded Jan 3 04:40:04 MediaLibrary apcupsd[24946]: NIS server startup succeeded Jan 3 05:12:56 MediaLibrary kernel: sdc: sdc1 Jan 3 11:22:15 MediaLibrary sshd[20327]: Accepted none for root from 10.15.98.122 port 62767 ssh2 ---=== tail of syslog ends here ===--- The only other thing that could be useful is I have a monthly parity check scheduled and this behavior seems to have started when the parity check started.

This is a continuation of the issue I posted about last month. For some reason the last 3 days, whenever the mover runs (currently randomly every 2-ish hours) the system is now trying to move the entire TVShows share. Whenever this happens, the cpu pegs at 100%, nothing is accessible (no Web UI, no file shares work, etc.). I can SSH into the system but the Diagnostics won't complete, I can't even get "shutdown -r now" to work. About the only way to bring the system back up is to hard power it off. I've attached the diagnostics after the reboot. I'm hoping someone can see a setting/config I have wrong that's causing the mover to think it needs to move the TVShows share. This system was using 5.x for 3 years without issue. Once I upgraded to v6 this started happening. For the moment I've scheduled the mover to run Weekly so I can hopefully troubleshoot this and resolve it. medialibrary-diagnostics-20160203-1038.zip

I just saw your thread. Sure seems to be a similar issue. SHFS is doing something lengthy that's causing 100% CPU utilization. Are we just being impatient and it's "supposed" to do what it's doing?

I have an unRaid system I've been running for about 3 years now with v5. Decided to upgrade to 6, read lots of threads before attempting the upgrade. On Friday I performed the upgrade and didn't run into any known issues. I let the system run for 24hrs and it even did a scheduled parity check (i forgot I configured it to do so every Saturday). All of that worked fine, no errors or issues. I installed a few plugins of interest, specifically the preclear utility script, and started a preclear on a 4TB drive in the system. About 5 hrs into the preclear, the system became unresponsive. I happened to have the GUI in a tab and noticed the CPU went to 100% before it stopped responding or updating. I SSH'd into the system in the hopes of getting the diagnostics to run so I would have something to show all of you, but the diagnostics fails to complete. The system is still up and responding to SSH, I can tail the syslog (below) and i can run top (also below). I notice in the syslog that the mover has started to move "TVShows" - which is what leads me to think I've configured something wrong. TVShows is the root of about 10TB's of content - which might explain the 100% CPU and non-responsive GUI, and the inability to stream content from the array. What I don't understand is WHY the mover is moving TVShows. It's not on the cache drive, it's stored on various disks in the array. Back in v5 I had set specific drives to be used for TVShows, and different drives to be used for Movies. When I migrated to v6, I no longer see the option to pick specific drives (or I haven't found it yet) so I basically left the drive settings at "default". Is there a setting/configuration in v6 that I missed that is causing the system to copy TVShows to other drives or something? Any help would be appreciated. Regards, Jeff ---=== tail of syslog begins here ===--- root@MediaLibrary:~# tail -35 /var/log/syslog Jan 2 18:03:36 MediaLibrary avahi-daemon[1555]: server.c: Packet too short or invalid while reading response record. (Maybe a UTF-8 problem?) Jan 2 18:05:46 MediaLibrary kernel: md: sync done. time=43545sec Jan 2 18:05:46 MediaLibrary kernel: md: recovery thread sync completion status: 0 Jan 2 18:20:47 MediaLibrary kernel: mdcmd (102): spindown 0 Jan 2 19:39:53 MediaLibrary kernel: sdc: sdc1 Jan 2 19:39:59 MediaLibrary kernel: sdc: sdc1 Jan 2 20:00:01 MediaLibrary logger: mover started Jan 2 20:00:01 MediaLibrary logger: mover finished Jan 2 21:30:31 MediaLibrary avahi-daemon[1555]: server.c: Packet too short or invalid while reading response record. (Maybe a UTF-8 problem?) Jan 2 22:00:01 MediaLibrary logger: mover started Jan 2 22:00:01 MediaLibrary logger: mover finished Jan 2 22:31:24 MediaLibrary kernel: mdcmd (103): spindown 0 Jan 2 22:31:25 MediaLibrary kernel: mdcmd (104): spindown 3 Jan 2 22:34:02 MediaLibrary kernel: mdcmd (105): spindown 1 Jan 2 22:34:04 MediaLibrary kernel: mdcmd (106): spindown 4 Jan 2 22:34:13 MediaLibrary kernel: mdcmd (107): spindown 2 Jan 2 23:34:39 MediaLibrary sshd[22350]: Accepted none for root from 10.15.98.122 port 56608 ssh2 Jan 2 23:45:57 MediaLibrary kernel: mdcmd (108): spindown 4 Jan 2 23:45:59 MediaLibrary kernel: mdcmd (109): spindown 7 Jan 2 23:46:03 MediaLibrary emhttp: shcmd (121): /usr/sbin/hdparm -y /dev/sda &> /dev/null Jan 2 23:46:05 MediaLibrary kernel: mdcmd (110): spindown 0 Jan 2 23:46:06 MediaLibrary kernel: mdcmd (111): spindown 2 Jan 2 23:49:10 MediaLibrary sshd[25586]: Accepted none for root from 10.15.98.122 port 57095 ssh2 Jan 3 00:00:01 MediaLibrary logger: mover started Jan 3 00:00:01 MediaLibrary logger: moving "TVShows" Jan 3 00:03:00 MediaLibrary sshd[25586]: Received disconnect from 10.15.98.122: 11: disconnected by user Jan 3 00:03:07 MediaLibrary sshd[22350]: Received disconnect from 10.15.98.122: 11: disconnected by user Jan 3 01:35:42 MediaLibrary sshd[16819]: Accepted none for root from 10.15.98.122 port 60914 ssh2 Jan 3 01:40:06 MediaLibrary sshd[16819]: Received disconnect from 10.15.98.122: 11: disconnected by user Jan 3 04:40:01 MediaLibrary apcupsd[9212]: apcupsd exiting, signal 15 Jan 3 04:40:01 MediaLibrary apcupsd[9212]: apcupsd shutdown succeeded Jan 3 04:40:04 MediaLibrary apcupsd[24946]: apcupsd 3.14.13 (02 February 2015) slackware startup succeeded Jan 3 04:40:04 MediaLibrary apcupsd[24946]: NIS server startup succeeded Jan 3 05:12:56 MediaLibrary kernel: sdc: sdc1 Jan 3 11:22:15 MediaLibrary sshd[20327]: Accepted none for root from 10.15.98.122 port 62767 ssh2 ---=== tail of syslog ends here ===--- ---=== top capture begins here ===--- top - 12:42:57 up 2 days, 13 min, 1 user, load average: 1292.30, 1292.26, 1289.78 Tasks: 1497 total, 1 running, 1496 sleeping, 0 stopped, 0 zombie Cpu(s): 2.1%us, 62.1%sy, 0.0%ni, 0.0%id, 35.6%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 4050692k total, 3405136k used, 645556k free, 349668k buffers Swap: 0k total, 0k used, 0k free, 608800k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2761 root 20 0 2918m 23m 2176 S 198 0.6 796:00.57 shfs 24237 root 20 0 53068 48m 1060 D 14 1.2 0:00.44 readvz 24007 root 20 0 14352 3376 1984 R 1 0.1 0:00.17 top 1 root 20 0 4368 1648 1548 S 0 0.0 0:07.26 init 2 root 20 0 0 0 0 S 0 0.0 0:00.07 kthreadd 3 root 20 0 0 0 0 S 0 0.0 0:17.81 ksoftirqd/0 5 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/0:0H 7 root 20 0 0 0 0 S 0 0.0 0:25.17 rcu_preempt 8 root 20 0 0 0 0 S 0 0.0 0:00.00 rcu_sched 9 root 20 0 0 0 0 S 0 0.0 0:00.00 rcu_bh 10 root RT 0 0 0 0 S 0 0.0 0:02.82 migration/0 11 root RT 0 0 0 0 S 0 0.0 0:02.82 migration/1 12 root 20 0 0 0 0 S 0 0.0 0:17.30 ksoftirqd/1 14 root 0 -20 0 0 0 S 0 0.0 0:00.00 kworker/1:0H 15 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper 16 root 20 0 0 0 0 S 0 0.0 0:00.00 kdevtmpfs 17 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns 20 root 0 -20 0 0 0 S 0 0.0 0:00.00 perf 252 root 0 -20 0 0 0 S 0 0.0 0:00.00 writeback 254 root 25 5 0 0 0 S 0 0.0 0:00.00 ksmd 255 root 39 19 0 0 0 S 0 0.0 0:39.39 khugepaged 256 root 0 -20 0 0 0 S 0 0.0 0:00.00 crypto 257 root 0 -20 0 0 0 S 0 0.0 0:00.00 kintegrityd 258 root 0 -20 0 0 0 S 0 0.0 0:00.00 bioset 260 root 0 -20 0 0 0 S 0 0.0 0:00.00 kblockd 304 nobody 20 0 282m 13m 11m S 0 0.4 0:00.00 smbd 311 nobody 20 0 282m 13m 11m S 0 0.3 0:00.00 smbd 316 nobody 20 0 282m 13m 11m S 0 0.3 0:00.00 smbd 345 nobody 20 0 281m 13m 11m S 0 0.3 0:00.00 smbd 346 nobody 20 0 281m 13m 11m S 0 0.3 0:00.00 smbd 392 root 0 -20 0 0 0 S 0 0.0 0:00.00 ata_sff 401 nobody 20 0 282m 13m 11m S 0 0.3 0:00.00 smbd 408 nobody 20 0 286m 15m 14m D 0 0.4 0:00.00 smbd 410 root 0 -20 0 0 0 S 0 0.0 0:00.00 devfreq_wq 452 root 20 0 282m 13m 11m S 0 0.3 0:00.00 smbd 504 nobody 20 0 281m 13m 11m S 0 0.3 0:00.00 smbd 510 root 0 -20 0 0 0 S 0 0.0 0:00.00 rpciod ---=== top capture ends here ===---