Copy stops, drive not full

I'm doing initial seeding of the box, and I'm getting stuck trying to copy files to a share. No matter which file I try to copy, it stops and times out after 85.1MB copied.

 

I have this share spread on all disks, excluding Disk1. All disks are set on high water. I'm not using split levels. With the copy command unRAID is trying to write to Disk3, which is a 2TB disk with 498MB free. If my understanding of high water is correct, that means the system started writing again on the drive 2MB ago. Therefore the reason shouldn't be that the high water mark has been reached and it's not changing drives properly, and there's certainly enough space on the disk.

 

Any ideas?

 

edit: I tried copying directly to the same share on another drive - I believe this is ok? -, and it's going fine even after the first 85.1MB.

Posting a syslog might help determine if this was a one-time thing (such as a network busy) or a real problem (like a bad Disk2)

I tried with different files, and those files copy just fine to another disk.

 

Prior to installing the disk3 into the array, I ran one round of pre-clear. It had been in use on my desktop prior to pre-clearing. Below extract from the preclear. edit: had a wrong disk in the previous version, below the correct one. Looks ok to me.

 

============================================================================
** Changed attributes in files: /tmp/smart_start_sde  /tmp/smart_finish_sde
                ATTRIBUTE   NEW_VAL OLD_VAL FAILURE_THRESHOLD STATUS      RAW_VALUE
      Raw_Read_Error_Rate =   113     100            6        ok          54501848
         Spin_Retry_Count =   100     100           97        near_thresh 0
         End-to-End_Error =   100     100           99        near_thresh 0
  Airflow_Temperature_Cel =    63      65           45        near_thresh 37
      Temperature_Celsius =    37      35            0        ok          37
   Hardware_ECC_Recovered =    37     100            0        ok          54501848
No SMART attributes are FAILING_NOW

0 sectors were pending re-allocation before the start of the preclear.
0 sectors were pending re-allocation after pre-read in cycle 1 of 1.
0 sectors were pending re-allocation after zero of disk in cycle 1 of 1.
0 sectors are pending re-allocation at the end of the preclear,
    the number of sectors pending re-allocation did not change.
0 sectors had been re-allocated before the start of the preclear.
0 sectors are re-allocated at the end of the preclear,
    the number of sectors re-allocated did not change. 
============================================================================

 

Posting a syslog might help determine if this was a one-time thing (such as a network busy) or a real problem (like a bad Disk2)

 

I tried a few times, and there is no syslog entry when the copy fails. But there are a few of these at various times today, maybe that's related? I wasn't manually logging in at these times - or even awake at some times.

 

Jan 20 16:11:55 TheMonolith sSMTP[18842]: Creating SSL connection to host
Jan 20 16:11:55 TheMonolith sSMTP[18842]: SSL connection using RC4-SHA
Jan 20 16:11:55 TheMonolith sSMTP[18842]: Authorization failed (454 4.7.0 Too many login attempts, please try again later. eg1sm303355wib.0 - gsmtp)

Jan 20 16:11:55 TheMonolith sSMTP[18842]: Creating SSL connection to host
Jan 20 16:11:55 TheMonolith sSMTP[18842]: SSL connection using RC4-SHA
Jan 20 16:11:55 TheMonolith sSMTP[18842]: Authorization failed (454 4.7.0 Too many login attempts, please try again later. eg1sm303355wib.0 - gsmtp)

That looks like someone may be trying to hack in to your server. Unraid is not meant to be exposed to the internet, it needs to be behind a router, with no ports forwarded unless you know exactly what you are doing. How is your unraid box connected to your network?

 

Scratch that, after thinking a little more on it, that may be an add-on trying to send mail, and failing. Do you have any notification add-on's installed?

Scratch that, after thinking a little more on it, that may be an add-on trying to send mail, and failing. Do you have any notification add-on's installed?

 

Hmm, that's probably it. I have a Dynamix plugin Email Notify on, but I was unable to configure it correctly. I've uninstalled it, and will reboot once pre-clear on another disk is done.

 

But that reminds me that I need to secure my array. It is connected to a router, but I need to study what else is needed to ensure it can't be accessed remotely.

 

I had the wrong pre-clear report in my previous mail. Disk3 that is refusing to be written to had no issues in the pre-clear report. I'll re-test copying after reboot.

But that reminds me that I need to secure my array. It is connected to a router, but I need to study what else is needed to ensure it can't be accessed remotely.

In general, you just need to make sure that you don't forward any ports to the unraid box. Home routers pretty much block any unsolicited outside traffic by default, you shouldn't need to really do anything. The exception is uPnP, if that is enabled on the router, and you have a uPnP aware application, it can request port forwarding dynamically.

But that reminds me that I need to secure my array. It is connected to a router, but I need to study what else is needed to ensure it can't be accessed remotely.

In general, you just need to make sure that you don't forward any ports to the unraid box. Home routers pretty much block any unsolicited outside traffic by default, you shouldn't need to really do anything. The exception is uPnP, if that is enabled on the router, and you have a uPnP aware application, it can request port forwarding dynamically.

 

No, haven't forwarded any ports.

 

I take the uPnP-aware application would have to be on unRAID box for it to be a security threat? How do I find out? I have Dynamix installed, and failed installations of utorrent, unmenu and screen. Can't wait for an updated plugin framework...

I take the uPnP-aware application would have to be on unRAID box for it to be a security threat? How do I find out? I have Dynamix installed, and failed installations of utorrent, unmenu and screen. Can't wait for an updated plugin framework...

utorrent is uPnP, as many torrent clients are. Whether it's a risk you can deal with is up to you. Private trackers generally require an open incoming port, whether manually forwarded or uPnP. So if you are running a torrent client and want to be "connectable", you will need that port open. It's only as risky as the coding in the torrent client you choose. If there is a vulnerability in that specific torrent client, you could get hacked.

This is getting curiouser and curiouser:

 

I tried copying files to another share folder directly to disk3. This is as opposed to copying to the share itself, and letting unRAID determine the drive to use. It worked! That is curious: copying directly to the disk3 doesn't work on one share, but it does on the other 

 

edit: yet another test. I copied a video file to the share that worked. Then I moved the file to the share that wouldn't allow copying, and that also worked, and the file plays fine - I will run an md5 check to confirm (further edit: confirmed ok). After this the non-functional share started taking files again. end edit

 

There were no other disk activities going on in the above tests.

 

SMART report for disk3 shows RAW value of zero for Reallocated_Sector_Ct and Current_Pending_Sector, which according to the wiki are the ones to look at - so the disk should be ok.

 

I take the uPnP-aware application would have to be on unRAID box for it to be a security threat? How do I find out? I have Dynamix installed, and failed installations of utorrent, unmenu and screen. Can't wait for an updated plugin framework...

utorrent is uPnP, as many torrent clients are. Whether it's a risk you can deal with is up to you. Private trackers generally require an open incoming port, whether manually forwarded or uPnP. So if you are running a torrent client and want to be "connectable", you will need that port open. It's only as risky as the coding in the torrent client you choose. If there is a vulnerability in that specific torrent client, you could get hacked.

 

Thanks, that clears it up for me!

Well, the problem is back. Can't copy to a share on disk3 directly or via share, although the disk has plenty of space, SMART doesn't report any errors, and other shares on the same disk work fine. Rebooting the server didn't help, either.

 

I'm out of ideas. Anyone?

Attach a syslog and note the time of a failed transfer.

Attach a syslog and note the time of a failed transfer.

 

There is no entry in syslog when the failed transfer happens. Below my entire syslog since reboot - and I re-tried it a few times during the hours covered by the log.

 

Jan 21 04:40:01 TheMonolith syslogd 1.4.1: restart.
Jan 21 05:04:44 TheMonolith dhcpcd[1094]: eth0: renewing lease of 192.168.1.177
Jan 21 05:04:44 TheMonolith dhcpcd[1094]: eth0: acknowledged 192.168.1.177 from 192.168.1.1
Jan 21 05:04:44 TheMonolith dhcpcd[1094]: eth0: leased 192.168.1.177 for 43200 seconds
Jan 21 10:30:07 TheMonolith kernel:  sdf: sdf1
Jan 21 10:41:20 TheMonolith dhcpcd[1094]: eth0: renewing lease of 192.168.1.177
Jan 21 10:41:20 TheMonolith dhcpcd[1094]: eth0: invalid UDP packet from 192.168.1.165
Jan 21 10:41:20 TheMonolith dhcpcd[1094]: eth0: acknowledged 192.168.1.177 from 192.168.1.1
Jan 21 10:41:20 TheMonolith dhcpcd[1094]: eth0: leased 192.168.1.177 for 43200 seconds
Jan 21 16:14:55 TheMonolith dhcpcd[1094]: eth0: renewing lease of 192.168.1.177
Jan 21 16:14:55 TheMonolith dhcpcd[1094]: eth0: acknowledged 192.168.1.177 from 192.168.1.1
Jan 21 16:14:55 TheMonolith dhcpcd[1094]: eth0: leased 192.168.1.177 for 43200 seconds
Jan 21 19:51:21 TheMonolith in.telnetd[2616]: connect from 192.168.1.165 (192.168.1.165)
Jan 21 19:51:24 TheMonolith login[2617]: ROOT LOGIN  on '/dev/pts/1' from 'SpacemanSpiff.lan'

That is not a complete syslog. Attach the entire syslog. zip if needed. Attach screen shots of unRAID main and the share configuration.

That is not a complete syslog. Attach the entire syslog. zip if needed. Attach screen shots of unRAID main and the share configuration.

 

That's what I get following the wiki, which tells me to do use the command "cp  /var/log/syslog  /boot".

 

I found a longer syslog under /boot/logs, but it's from yesterday.

 

So, how do I capture the verbose syslog from today?

Try "cp /var/log/syslog* /boot"

Try "cp /var/log/syslog* /boot"

 

I get two files: "syslog" with the entries below showing me logging in using putty, in between where I tried to copy to the share twice, resulting in connection time out as usual. Also, "syslog.1" which is from yesterday.

 

Jan 21 23:28:37 TheMonolith in.telnetd[12201]: connect from 192.168.1.165 (192.168.1.165)
Jan 21 23:28:44 TheMonolith login[12202]: ROOT LOGIN  on '/dev/pts/0' from 'SpacemanSpiff.lan'
Jan 21 23:40:07 TheMonolith in.telnetd[20746]: connect from 192.168.1.165 (192.168.1.165)
Jan 21 23:40:11 TheMonolith login[20752]: ROOT LOGIN  on '/dev/pts/0' from 'SpacemanSpiff.lan'

Yeah... something's not right.. a typical boot will put almost 100 lines in your syslog, so i'd expect to see WAY more then you are getting. What happens if you type in "cat /var/log/syslog" in the console? How many lines of text do you get then?

 

I can not offer advise without a syslog. Please attach the log in text format. Zip if needed. See SYSTEM LOG in my sig.

Attached the full syslog since boot, using the command "cp /var/log/syslog /boot/syslog.txt". I did two file copying attempts to disk3 between lines 466/467 at the end of the file. cat command shows the same content, of course.

 

There are only a few DHCP entries after boot, but not hundreds of lines. The only other entries are for pre-clearing an unrelated disk. This is not the type of syslog described above.

 

Is there a verbose mode of syslog I have to enable? I'm using 5.0.4.

syslog.txt

Enter "cp /var/log/syslog* /boot". Additional syslog files should appear on the flash.

Enter "cp /var/log/syslog* /boot". Additional syslog files should appear on the flash.

 

As on previous times and described earlier, I get two syslog files, "syslog" and "syslog.1". Former is the one that I attached earlier, latter is a file from two days ago.

 

edit: I took a closer look at syslog.1 from two days ago, and it has dozens upon dozens of entries below. It looks like some fallout from a botched Transmission install attempt. Could this be related?

 

edit2: I deleted both syslogs from /var/log to see if that would clear things up. Then I tried copying a file to disk3, it timed out as usual, but no new syslog was created. I imagine unRAID will create a new syslog file automatically as soon as it needs to write to it?

 

I further re-re-tested by copying a file to another share on the same disk, and it worked. Then I copied the same file to the share that's having the tantrum, and it worked! But once again, copying timed out in the next file. So it doesn't appear to be a certain physical section of the drive, but some other issue.

 

Jan 20 23:04:49 TheMonolith logger: 
Jan 20 23:04:49 TheMonolith logger: Warning: simplexml_load_file(): /boot/config/plugins/transmission_unplugged.plg:1213: parser error : Entity 'nbsp' not defined in /usr/local/sbin/installplg on line 13
Jan 20 23:04:49 TheMonolith logger: 
Jan 20 23:04:49 TheMonolith logger: Warning: simplexml_load_file(): ete-dir": "//Downloads", </div><div class='line' id='LC93'>  in /usr/local/sbin/installplg on line 13
Jan 20 23:04:49 TheMonolith logger: 
Jan 20 23:04:49 TheMonolith logger: Warning: simplexml_load_file():                                                                                ^ in /usr/local/sbin/installplg on line 13

Unable to advise without a complete syslog.

Unable to advise without a complete syslog.

 

I understand that, but I've tried every suggestion in this thread numerous times, and the only syslog unRAID provides me is the one attached in a previous post.

We're not trying to be a pain...

 

unRAID syslogs are held in memory. They are NOT written to the flash drive unless you specifically copy them from memory and write them to flash.

(that's what the "cp /var/log/syslog* /boot" was doing...copying the syslog*  to 'boot' (boot is the flash))

 

Your Syslog.1 appears to be just a preclear operation.

 

The syslog you mention as posted first isn't complete, its the extract of a few lines around the problem area.  What's needed is the complete syslog.

That will show the complete boot up sequence of unRAID, the RAM, the network, all the disks, all the plugins, all the shares, etc., as they are loaded and configured.

It probably runs about 500 lines, starts out with a hundred lines prefixed with 'TheMonolith kernel: ' commands,

followed by 'TheMonolith logger: ', and then 'TheMonolith emhttp: ', etc.

 

Try either of these ways to get it:

0. Erase any 'syslog*' files on the root of your flash drive.

1. from the unRAID console, or Telnet, issue the copy command again, "cp /var/log/syslog*    /boot"

2. if you are familiar with MC, browse to var/log and view the syslog you see there in the window.  Copy it out to flash3

We're not trying to be a pain...

 

I know that

 

The thing is, the file in this post is the complete, unedited syslog file as given out by my unRAID box. But it is clear that it is not what you guys are looking for.

 

I don't know why it doesn't list all the things it should. Maybe there's a verbose version of the file somewhere, but that's the only syslog file I get using the several iterations of the commands you and others have given in this thread.

 

Try either of these ways to get it:

0. Erase any 'syslog*' files on the root of your flash drive.

1. from the unRAID console, or Telnet, issue the copy command again, "cp /var/log/syslog*    /boot"

2. if you are familiar with MC, browse to var/log and view the syslog you see there in the window.  Copy it out to flash3

 

I already tried 0, after which the "cp /var/log/syslog*    /boot" command gives "file not found" error - ie. the file hasn't been re-created, I presume since there has been nothing to report to syslog. I just tried it again, after several hours since I deleted the syslog file, nothing. As you can see from the syslog file above, there are breaks of several hours between entries, but there were file transfers going on throughout the night - and I believe this should have been visible on the log.

 

I also tried 2, and the syslog file at /var/log was the same as in the link above.

 

 

I'm currently in the process of moving all files off disk3, and I will hose it down thoroughly with a pre-clear tomorrow to see if that helps. That's the last resort: I would prefer to get this sorted out, as it is a very peculiar bug/issue, and the syslog file is also something that concerns me.

 

My entirely uneducated hunch is that the latter is a result of a botched unMenu/Transmission/uTorrent install - none of the guides on the wiki and forum worked on those, and I don't know how to clear/uninstall them. Dynamix worked and works.