A "mover" issue?


Recommended Posts

i am having the same problem:

 

Jun 11 16:34:29 UNRAID logger: mover started
Jun 11 16:34:29 UNRAID logger: skipping apps/
Jun 11 16:34:29 UNRAID logger: moving unsorted/
Jun 11 16:34:29 UNRAID logger: ./unsorted/._Screen Shot 2013-05-25 at 12.39.03 PM.jpg
Jun 11 16:34:29 UNRAID logger: .d..t...... ./
Jun 11 16:34:29 UNRAID logger: rsync: get_xattr_data: lgetxattr("unsorted","user.org.netatalk.supports-eas.4VNO7Q",4) failed: Input/output error (5)
Jun 11 16:34:29 UNRAID logger: .d..t...... unsorted/
Jun 11 16:34:29 UNRAID logger: >f+++++++++ unsorted/._Screen Shot 2013-05-25 at 12.39.03 PM.jpg
Jun 11 16:34:29 UNRAID kernel: REISERFS warning (device md1): jdm-20001 reiserfs_xattr_get: Invalid magic for xattr (user.org.netatalk.supports-eas.4VNO7Q) associated with [1919251317 1735552814 0x74656e2e UNKNOWN]
Jun 11 16:34:51 UNRAID logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7]

Link to comment

i am having the same problem:

 

Jun 11 16:34:29 UNRAID logger: mover started
Jun 11 16:34:29 UNRAID logger: skipping apps/
Jun 11 16:34:29 UNRAID logger: moving unsorted/
Jun 11 16:34:29 UNRAID logger: ./unsorted/._Screen Shot 2013-05-25 at 12.39.03 PM.jpg
Jun 11 16:34:29 UNRAID logger: .d..t...... ./
Jun 11 16:34:29 UNRAID logger: rsync: get_xattr_data: lgetxattr("unsorted","user.org.netatalk.supports-eas.4VNO7Q",4) failed: Input/output error (5)
Jun 11 16:34:29 UNRAID logger: .d..t...... unsorted/
Jun 11 16:34:29 UNRAID logger: >f+++++++++ unsorted/._Screen Shot 2013-05-25 at 12.39.03 PM.jpg
Jun 11 16:34:29 UNRAID kernel: REISERFS warning (device md1): jdm-20001 reiserfs_xattr_get: Invalid magic for xattr (user.org.netatalk.supports-eas.4VNO7Q) associated with [1919251317 1735552814 0x74656e2e UNKNOWN]
Jun 11 16:34:51 UNRAID logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7]

This is slightly different.  Can you run 'reiserfsck' on all your disks and then see if this issue persists?

Link to comment

This is slightly different.  Can you run 'reiserfsck' on all your disks and then see if this issue persists?

 

will do.

should i run it on /dev/sdX or /dev/mdX?

For array disks always on /dev/mdX to preserve parity.  For the cache disk use /dev/sdX

Link to comment

May I add this:

 

http://lime-technology.com/forum/index.php?topic=27720.msg246379#msg246379

 

Mover issue too? No pun intended.

This problem is something different.  The reason the 'mover' wouldn't run is because it looks like previously you changed a share setting, or maybe added a user share.  When this kind of config change happens the s/w stops the network protocols and unmounts the user share file system, makes the change, then re-mounts and re-starts the network protocols.  But when the s/w tried to do this the user share file system wouldn't completely unmount, probably due to an add-on holding a file open or 'cd'ing somewhere within /mnt/user/... Does that ring a bell?  This is a bug that I'll  have to think about how to solve.

 

Here is the clue from your system log:

Jun 10 17:38:45 Tower1 emhttp: shcmd (131): set -o pipefail ; umount /mnt/user |& logger
Jun 10 17:38:45 Tower1 logger: umount: /mnt/user: device is busy.
Jun 10 17:38:45 Tower1 logger:         (In some cases useful info about processes that use
Jun 10 17:38:45 Tower1 logger:          the device is found by lsof( or fuser(1))
Jun 10 17:38:45 Tower1 emhttp: _shcmd: shcmd (131): exit status: 1
Jun 10 17:38:45 Tower1 emhttp: shcmd (132): rmdir /mnt/user |& logger
Jun 10 17:38:45 Tower1 logger: rmdir: failed to remove `/mnt/user': Device or resource busy

 

Link to comment

For array disks always on /dev/mdX to preserve parity.  For the cache disk use /dev/sdX

Thanks. There was no corruption found on any of array disks or cache.

###########
reiserfsck --check started at Fri Jun 14 06:52:58 2013
###########
Replaying journal: Done.
Reiserfs journal '/dev/md2' in blocks [18..8211]: 0 transactions replayed
Checking internal tree.. finished                                
Comparing bitmaps..finished
Checking Semantic tree:
finished                                                                       
No corruptions found
There are on the filesystem:
Leaves 332238
Internal nodes 2113
Directories 528
Other files 2196
Data block pointers 335897724 (0 of them are zero)
Safe links 0
###########
reiserfsck finished at Fri Jun 14 09:20:42 2013
###########

 

I just manually triggered the mover, here is the log.

Jun 14 10:02:39 UNRAID logger: mover started
Jun 14 10:02:39 UNRAID logger: skipping apps/
Jun 14 10:02:39 UNRAID logger: moving unsorted/
Jun 14 10:02:39 UNRAID logger: ./unsorted/test.file
Jun 14 10:02:39 UNRAID logger: rsync: get_xattr_data: lgetxattr("unsorted","user.org.netatalk.supports-eas.4VNO7Q",4) failed: Input/output error (5)
Jun 14 10:02:39 UNRAID logger: >f+++++++++ unsorted/test.file
Jun 14 10:02:39 UNRAID kernel: REISERFS warning (device md1): jdm-20001 reiserfs_xattr_get: Invalid magic for xattr (user.org.netatalk.supports-eas.4VNO7Q) associated with [1919251317 1735552814 0x74656e2e UNKNOWN]
Jun 14 10:02:43 UNRAID logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7]
Jun 14 10:02:43 UNRAID logger: mover finished

 

Also, I have a /mnt/cache/apps share that is set to "cache disk only" - it is skipped by the mover, as expected, but it is always orange and showing pending cache. The "apps" folder is also present in /mnt/user - is this desired behaviour? If I delete a file from /mnt/user/apps it is also deleted from /mnt/cache/apps.

Link to comment

Also, I have a /mnt/cache/apps share that is set to "cache disk only" - it is skipped by the mover, as expected, but it is always orange and showing pending cache.

I think that Orange is used to indicate a share that contains data that is unprotected and by definition the cache (currently at least) falls into this category.

The "apps" folder is also present in /mnt/user - is this desired behaviour? If I delete a file from /mnt/user/apps it is also deleted from /mnt/cache/apps.

Yes.  Files are deleted regardless of whether they are on the cache disk or not if they show up under the user shares.

Link to comment

Thanks! I probably expected 'apps' folder set as 'cache disk only' not to be visible in the user shares, but glad to know its working as indended :)

 

Back to the topic - Tom, I should probably mention that I'm on rc13, running Basic version of unRAID.

Link to comment

For array disks always on /dev/mdX to preserve parity.  For the cache disk use /dev/sdX

Thanks. There was no corruption found on any of array disks or cache.

###########
reiserfsck --check started at Fri Jun 14 06:52:58 2013
###########
Replaying journal: Done.
Reiserfs journal '/dev/md2' in blocks [18..8211]: 0 transactions replayed
Checking internal tree.. finished                                
Comparing bitmaps..finished
Checking Semantic tree:
finished                                                                       
No corruptions found
There are on the filesystem:
Leaves 332238
Internal nodes 2113
Directories 528
Other files 2196
Data block pointers 335897724 (0 of them are zero)
Safe links 0
###########
reiserfsck finished at Fri Jun 14 09:20:42 2013
###########

 

I just manually triggered the mover, here is the log.

Jun 14 10:02:39 UNRAID logger: mover started
Jun 14 10:02:39 UNRAID logger: skipping apps/
Jun 14 10:02:39 UNRAID logger: moving unsorted/
Jun 14 10:02:39 UNRAID logger: ./unsorted/test.file
Jun 14 10:02:39 UNRAID logger: rsync: get_xattr_data: lgetxattr("unsorted","user.org.netatalk.supports-eas.4VNO7Q",4) failed: Input/output error (5)
Jun 14 10:02:39 UNRAID logger: >f+++++++++ unsorted/test.file
Jun 14 10:02:39 UNRAID kernel: REISERFS warning (device md1): jdm-20001 reiserfs_xattr_get: Invalid magic for xattr (user.org.netatalk.supports-eas.4VNO7Q) associated with [1919251317 1735552814 0x74656e2e UNKNOWN]
Jun 14 10:02:43 UNRAID logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7]
Jun 14 10:02:43 UNRAID logger: mover finished

 

How did you create "test.file"?  Please type these commands and paste the result in a post:

 

getfattr -d /mnt/user/unsorted
getfattr -d /mnt/user/unsorted/test.file

 

Also, I have a /mnt/cache/apps share that is set to "cache disk only" - it is skipped by the mover, as expected, but it is always orange and showing pending cache.

Right, orange just means there are files on the share at risk of being lost if the cache drive fails.  It will always be orange for a cache-only share (until "cache pool" feature is added).

 

The "apps" folder is also present in /mnt/user - is this desired behaviour? If I delete a file from /mnt/user/apps it is also deleted from /mnt/cache/apps.

Yes this is correct.  The "/mnt/user/<sharename>" tree provides a composite (combined) view of

/mnt/cache/<sharename>

/mnt/disk1/<sharename>

:

/mnt/diskN/<sharename>  where N is the highest disk number you have installed.

 

You can access a particular file/directory via

  /mnt/user/<sharename>/<file/directory name>

or if you know which disk the object is on via

  /mnt/user/diskX/<sharename>/<file/directory>

or if you know the object is on cache disk via

  /mnt/user/cache/<sharename>/<file/directory>

 

Make sense?

 

But here's something I should probably point out: Above is ok if you access any method other than AFP.  If you have exported a particular usershare via AFP, you must only access the share via AFP and in particular do not write new files into the share via any other protocol.  AFP is an "exclusive club" because netatalk maintains a separate database.  If you start accessing files on an AFP share outside the AFP protocol, netatalk won't know about it and funny things can happen.

Link to comment

 

How did you create "test.file"?  Please type these commands and paste the result in a post:

 

getfattr -d /mnt/user/unsorted
getfattr -d /mnt/user/unsorted/test.file

i created it with 'touch test.file' command, inside /mnt/cache/unsorted

getfattr command returned nothing:

root@UNRAID:~# getfattr -d /mnt/user/unsorted
root@UNRAID:~# getfattr -d /mnt/user/unsorted/test.file

screenshot file that was in the log from my first post was uploaded via SMB

Make sense?

yep :) thanks, it's all clear now.

 

But here's something I should probably point out: Above is ok if you access any method other than AFP.  If you have exported a particular usershare via AFP, you must only access the share via AFP and in particular do not write new files into the share via any other protocol.  AFP is an "exclusive club" because netatalk maintains a separate database.  If you start accessing files on an AFP share outside the AFP protocol, netatalk won't know about it and funny things can happen.

right, i don't know if it makes any difference, but i used to export my shares via AFP - it was slow, unreliable and causing all sorts of trouble, so right now AFP is turned off (Enable AFP: no) and I'm using SBM to view my shares

Link to comment

But here's something I should probably point out: Above is ok if you access any method other than AFP.  If you have exported a particular usershare via AFP, you must only access the share via AFP and in particular do not write new files into the share via any other protocol.  AFP is an "exclusive club" because netatalk maintains a separate database.  If you start accessing files on an AFP share outside the AFP protocol, netatalk won't know about it and funny things can happen.

 

Yes and No, the AFP DB behavior is separate to the issue of the damaged/corrupt apple xattr's.

 

WARNING

 

In order to be able to run -rf reconstructing the CNIDs in the database from the AppleDouble files, make sure you've run a -r rebuild sometimes before, where the CNIDs then would have been synched between database and AppleDouble files.

 

Also be careful about the option nocnidcache. Avoid this option if at all possible, because if prevents you from being able to use -f.

 

CNID background

 

The CNID backends maintains name to ID mappings. If you change a filename outside afpd(8) (shell, samba), the CNID db will not know and not reflect that change. Netatalk tries to recover from such inconsistencies as gracefully as possible. The mechanisms to resolve such inconsistencies may fail sometimes, though, as this is not an easy task to accomplish. E.g. if several names in the path to the file or directory have changed, things may go wrong.

 

If you change a lot of filenames at once, chances are higher that the afpds fallback mechanisms fail, i.e. files will be assigned new IDs, even though the file hasn't changed.

 

I have seen this behavior "If you change a lot of filenames at once, chances are higher that the afpds fallback mechanisms fail, i.e. files will be assigned new IDs, even though the file hasn't changed." My Sabnzb, Sickbeard are Windows hosts working off unRAID via SMB. they handle the media placement on unRAID via SMB. My clients that read/play (only) media are Mac's via AFP. My Mac server running plex scrapes the media and it is this query for the files via AFP and unRAID user shares that kick off the AFP DB to update the inodes. With adding MAXDEBUG, I am able to monitor how the DB handles these changes, it not to bad, it replaces some, creates new, and deletes all as required and quite fast, but I have a permanent DB (which lives on a raid 10 array), that DB closes on array shutdowns so nothing is lost with reboots. Something you considered doing on the cache drive in the past. And I personally feel you should and place in your release notes that AFP requires a cache drive (period) in order to utilize (again just my 2 cents). I also wanted to discuss if there would be a way to have a AFP CLIENT running on unRAID there by all file changes by other protocols would be picked up in realtime and DB instantly updated. Not sure if I explained that all to well.

 

But back to the topic on AFP extended attribute damage corruption "user.org.netatalk.supports-eas.xxx      failed: Exec format error (8)"

 

A quick search displayed how far back these damaged/corrupted apple xattr have been logged in syslog's (all unique and individual posts):

http://lime-technology.com/forum/index.php?topic=22831.msg202446#msg202446

http://lime-technology.com/forum/index.php?topic=23846.msg210467#msg210467

http://lime-technology.com/forum/index.php?topic=26642.msg233956#msg233956

http://lime-technology.com/forum/index.php?topic=22012.msg195067#msg195067

http://lime-technology.com/forum/index.php?topic=23255.msg205387#msg205387

http://lime-technology.com/forum/index.php?topic=25689.msg227410#msg227410

http://lime-technology.com/forum/index.php?topic=20596.msg182557#msg182557

http://lime-technology.com/forum/index.php?topic=16135.msg149265#msg149265

http://lime-technology.com/forum/index.php?topic=16885.msg153967#msg153967

http://lime-technology.com/forum/index.php?topic=19972.msg177808#msg177808

http://lime-technology.com/forum/index.php?topic=25306.msg233774#msg233774

 

I believe that with the last xattr additions in the last few RC we are where we need to be, I just need to verify that this is the case and will update you with info as soon as I do and what I did to repair. It will be either tonight or after Sunday, best that I can do. Everyone needw a break and a great fathers day bash! Cheers

 

Link to comment

Tom, I have created new user share (lets say 'ABC'), set it to use cache, created a test file inside /mnt/cache/ABC and started the mover - it finished fine, without any errors. Next i exported ABC via SMB, uploaded a file from OS X and triggered the mover - it finished fine, again. So I guess the problem applies to my 'old' shares only, possibly those, that were previously exported via AFP?

Link to comment

Tom, I have created new user share (lets say 'ABC'), set it to use cache, created a test file inside /mnt/cache/ABC and started the mover - it finished fine, without any errors. Next i exported ABC via SMB, uploaded a file from OS X and triggered the mover - it finished fine, again. So I guess the problem applies to my 'old' shares only, possibly those, that were previously exported via AFP?

"Watches madburg doing a victory dance at his PC"
Link to comment

Tom, I have created new user share (lets say 'ABC'), set it to use cache, created a test file inside /mnt/cache/ABC and started the mover - it finished fine, without any errors. Next i exported ABC via SMB, uploaded a file from OS X and triggered the mover - it finished fine, again. So I guess the problem applies to my 'old' shares only, possibly those, that were previously exported via AFP?

 

Ok try this.  Make a list of your data disks that contain the top-level directory "unsorted".  For each one list the xattrs, for example, let's say 'unsorted' is on disk1, disk3 and disk7, you would type this:

 

getfattr -d /mnt/disk1/unsorted
getfattr -d /mnt/disk3/unsorted
getfattr -d /mnt/disk7/unsorted

Link to comment

Tom, I have created new user share (lets say 'ABC'), set it to use cache, created a test file inside /mnt/cache/ABC and started the mover - it finished fine, without any errors. Next i exported ABC via SMB, uploaded a file from OS X and triggered the mover - it finished fine, again. So I guess the problem applies to my 'old' shares only, possibly those, that were previously exported via AFP?

"Watches madburg doing a victory dance at his PC"

 

Its not like that... I am very upset with this whole AFP thing I wont deny that. Its been treated as a bastard child in unRAID no doubt, BUT no one here can change the past, nor me get back all the wasted time, postings, troubleshooting, with besides Speeding_Ant no help from anyone.

So if Tom does the right thing(s) then at the end of the day we'll all be good. Most important is we fix this particular issue and make some tweaks so it works "better", for a lack of words at the moment.

 

I want to add, the reason I have not published publicly what I did to resolve this (for months) is because it can possibly  ruin your data (lose of extended attribute you may have in your data, not everyone only has Movie, TV shows, there's plenty of data that carries extended data), so I rather share it with Tom, who could script up the remedy in a safe® manner (as Linux scripting, I am a beginner as best).

 

The only puzzling thing was the amount and lack of these apple ea. attr's on various user shares that had me stumped (random no pattern). Tom has shared that info, even better the source, that showed they should be written, followed by deleted. So we shouldn't have any damaged/corrupted apple ea's nor good written ones!

So I can now verify my fix, the behavior of the apple ea's after the fix (have the shfs debug flag now to see something else as well) and report back to Tom.

 

 

Link to comment

OK

 

You pushed me over the edge!  ;D  see attachment.

 

I don't care for apple, but I like plex and especially plex on mac, the skins, and the direct play from a mac plex client to the nas (only available via plex for mac over AFP) versus throught the plex server to the nas. I personally hope AFP die's a horrible death, now their posting how wonderful SMB2 is.. a$$holes finally figured out they cant do sh^t with this protocol with retarded DB.

Finally.jpg.4a43ddaf79dbc9d85fae167eeec31a20.jpg

Link to comment

For -rc15 I changed a setting in AppleVolumes.default having to do with the "ea" setting:

 

# ea                  -> none|auto|sys|ad
#                        Specify how Extended Attributes are stores. default
#                        is auto.
#                        auto: try "sys" (by setting an EA on the shared
#                              directory itself), fallback to "ad".  Requires
#                              writable volume for performing the test.
#                              Note: options:ro overwrites "auto" with "none."
#                        sys:  Use filesystem EAs
#                        ad:   Use files in AppleDouble directories
#                        none: No EA support
#

 

In prior versions the "ea" setting wasn't specified at all, so it defaulted to "auto", which means it's going to write a funny xattr to each exported directory (but then immediately delete it).  Since our file system (reiserfs), and any file system we would be using in the future, supports extended attributes, I explicitly set "ea:sys".

Link to comment

Sounds like a good idea, as of which unRAID version Tom? sorry just re-read you mentioned starting in RC15.

I was watching that and noticed some shares were being set differently then others and should not have been the case (this is a while back), so I think with all the update and hardcoding this, is a good thing.

 

I couldn't wait so I VPN'ed in and work on all this now. I want to kiss u and hit u at the same time, so many emotions right now  ;D. Once I backed up (just a dump for history) the currently assigned apple ea' per share and removed them all, enable shfs logging, wish I knew about this long ago, as I want to see what was going on with all the reads and write upon array start when AFP was enabled. Guess what, NO MORE thousands reads and write to the array, quick set and remove of the apple ea's by shfs being logged. Moving on with the Plex Server now, which is going to mount the volumes. I'll have the entire syslog from boot -> on emailed to you.

Link to comment