joyless Posted June 11, 2013 Share Posted June 11, 2013 i am having the same problem: Jun 11 16:34:29 UNRAID logger: mover started Jun 11 16:34:29 UNRAID logger: skipping apps/ Jun 11 16:34:29 UNRAID logger: moving unsorted/ Jun 11 16:34:29 UNRAID logger: ./unsorted/._Screen Shot 2013-05-25 at 12.39.03 PM.jpg Jun 11 16:34:29 UNRAID logger: .d..t...... ./ Jun 11 16:34:29 UNRAID logger: rsync: get_xattr_data: lgetxattr("unsorted","user.org.netatalk.supports-eas.4VNO7Q",4) failed: Input/output error (5) Jun 11 16:34:29 UNRAID logger: .d..t...... unsorted/ Jun 11 16:34:29 UNRAID logger: >f+++++++++ unsorted/._Screen Shot 2013-05-25 at 12.39.03 PM.jpg Jun 11 16:34:29 UNRAID kernel: REISERFS warning (device md1): jdm-20001 reiserfs_xattr_get: Invalid magic for xattr (user.org.netatalk.supports-eas.4VNO7Q) associated with [1919251317 1735552814 0x74656e2e UNKNOWN] Jun 11 16:34:51 UNRAID logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7] Link to comment
dikkiedirk Posted June 11, 2013 Share Posted June 11, 2013 May I add this: http://lime-technology.com/forum/index.php?topic=27720.msg246379#msg246379 Mover issue too? No pun intended. Link to comment
limetech Posted June 13, 2013 Author Share Posted June 13, 2013 i am having the same problem: Jun 11 16:34:29 UNRAID logger: mover started Jun 11 16:34:29 UNRAID logger: skipping apps/ Jun 11 16:34:29 UNRAID logger: moving unsorted/ Jun 11 16:34:29 UNRAID logger: ./unsorted/._Screen Shot 2013-05-25 at 12.39.03 PM.jpg Jun 11 16:34:29 UNRAID logger: .d..t...... ./ Jun 11 16:34:29 UNRAID logger: rsync: get_xattr_data: lgetxattr("unsorted","user.org.netatalk.supports-eas.4VNO7Q",4) failed: Input/output error (5) Jun 11 16:34:29 UNRAID logger: .d..t...... unsorted/ Jun 11 16:34:29 UNRAID logger: >f+++++++++ unsorted/._Screen Shot 2013-05-25 at 12.39.03 PM.jpg Jun 11 16:34:29 UNRAID kernel: REISERFS warning (device md1): jdm-20001 reiserfs_xattr_get: Invalid magic for xattr (user.org.netatalk.supports-eas.4VNO7Q) associated with [1919251317 1735552814 0x74656e2e UNKNOWN] Jun 11 16:34:51 UNRAID logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7] This is slightly different. Can you run 'reiserfsck' on all your disks and then see if this issue persists? Link to comment
joyless Posted June 13, 2013 Share Posted June 13, 2013 This is slightly different. Can you run 'reiserfsck' on all your disks and then see if this issue persists? will do. should i run it on /dev/sdX or /dev/mdX? Link to comment
limetech Posted June 13, 2013 Author Share Posted June 13, 2013 This is slightly different. Can you run 'reiserfsck' on all your disks and then see if this issue persists? will do. should i run it on /dev/sdX or /dev/mdX? For array disks always on /dev/mdX to preserve parity. For the cache disk use /dev/sdX Link to comment
limetech Posted June 13, 2013 Author Share Posted June 13, 2013 May I add this: http://lime-technology.com/forum/index.php?topic=27720.msg246379#msg246379 Mover issue too? No pun intended. This problem is something different. The reason the 'mover' wouldn't run is because it looks like previously you changed a share setting, or maybe added a user share. When this kind of config change happens the s/w stops the network protocols and unmounts the user share file system, makes the change, then re-mounts and re-starts the network protocols. But when the s/w tried to do this the user share file system wouldn't completely unmount, probably due to an add-on holding a file open or 'cd'ing somewhere within /mnt/user/... Does that ring a bell? This is a bug that I'll have to think about how to solve. Here is the clue from your system log: Jun 10 17:38:45 Tower1 emhttp: shcmd (131): set -o pipefail ; umount /mnt/user |& logger Jun 10 17:38:45 Tower1 logger: umount: /mnt/user: device is busy. Jun 10 17:38:45 Tower1 logger: (In some cases useful info about processes that use Jun 10 17:38:45 Tower1 logger: the device is found by lsof( or fuser(1)) Jun 10 17:38:45 Tower1 emhttp: _shcmd: shcmd (131): exit status: 1 Jun 10 17:38:45 Tower1 emhttp: shcmd (132): rmdir /mnt/user |& logger Jun 10 17:38:45 Tower1 logger: rmdir: failed to remove `/mnt/user': Device or resource busy Link to comment
joyless Posted June 14, 2013 Share Posted June 14, 2013 For array disks always on /dev/mdX to preserve parity. For the cache disk use /dev/sdX Thanks. There was no corruption found on any of array disks or cache. ########### reiserfsck --check started at Fri Jun 14 06:52:58 2013 ########### Replaying journal: Done. Reiserfs journal '/dev/md2' in blocks [18..8211]: 0 transactions replayed Checking internal tree.. finished Comparing bitmaps..finished Checking Semantic tree: finished No corruptions found There are on the filesystem: Leaves 332238 Internal nodes 2113 Directories 528 Other files 2196 Data block pointers 335897724 (0 of them are zero) Safe links 0 ########### reiserfsck finished at Fri Jun 14 09:20:42 2013 ########### I just manually triggered the mover, here is the log. Jun 14 10:02:39 UNRAID logger: mover started Jun 14 10:02:39 UNRAID logger: skipping apps/ Jun 14 10:02:39 UNRAID logger: moving unsorted/ Jun 14 10:02:39 UNRAID logger: ./unsorted/test.file Jun 14 10:02:39 UNRAID logger: rsync: get_xattr_data: lgetxattr("unsorted","user.org.netatalk.supports-eas.4VNO7Q",4) failed: Input/output error (5) Jun 14 10:02:39 UNRAID logger: >f+++++++++ unsorted/test.file Jun 14 10:02:39 UNRAID kernel: REISERFS warning (device md1): jdm-20001 reiserfs_xattr_get: Invalid magic for xattr (user.org.netatalk.supports-eas.4VNO7Q) associated with [1919251317 1735552814 0x74656e2e UNKNOWN] Jun 14 10:02:43 UNRAID logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7] Jun 14 10:02:43 UNRAID logger: mover finished Also, I have a /mnt/cache/apps share that is set to "cache disk only" - it is skipped by the mover, as expected, but it is always orange and showing pending cache. The "apps" folder is also present in /mnt/user - is this desired behaviour? If I delete a file from /mnt/user/apps it is also deleted from /mnt/cache/apps. Link to comment
itimpi Posted June 14, 2013 Share Posted June 14, 2013 Also, I have a /mnt/cache/apps share that is set to "cache disk only" - it is skipped by the mover, as expected, but it is always orange and showing pending cache. I think that Orange is used to indicate a share that contains data that is unprotected and by definition the cache (currently at least) falls into this category. The "apps" folder is also present in /mnt/user - is this desired behaviour? If I delete a file from /mnt/user/apps it is also deleted from /mnt/cache/apps. Yes. Files are deleted regardless of whether they are on the cache disk or not if they show up under the user shares. Link to comment
joyless Posted June 14, 2013 Share Posted June 14, 2013 Thanks! I probably expected 'apps' folder set as 'cache disk only' not to be visible in the user shares, but glad to know its working as indended Back to the topic - Tom, I should probably mention that I'm on rc13, running Basic version of unRAID. Link to comment
limetech Posted June 14, 2013 Author Share Posted June 14, 2013 For array disks always on /dev/mdX to preserve parity. For the cache disk use /dev/sdX Thanks. There was no corruption found on any of array disks or cache. ########### reiserfsck --check started at Fri Jun 14 06:52:58 2013 ########### Replaying journal: Done. Reiserfs journal '/dev/md2' in blocks [18..8211]: 0 transactions replayed Checking internal tree.. finished Comparing bitmaps..finished Checking Semantic tree: finished No corruptions found There are on the filesystem: Leaves 332238 Internal nodes 2113 Directories 528 Other files 2196 Data block pointers 335897724 (0 of them are zero) Safe links 0 ########### reiserfsck finished at Fri Jun 14 09:20:42 2013 ########### I just manually triggered the mover, here is the log. Jun 14 10:02:39 UNRAID logger: mover started Jun 14 10:02:39 UNRAID logger: skipping apps/ Jun 14 10:02:39 UNRAID logger: moving unsorted/ Jun 14 10:02:39 UNRAID logger: ./unsorted/test.file Jun 14 10:02:39 UNRAID logger: rsync: get_xattr_data: lgetxattr("unsorted","user.org.netatalk.supports-eas.4VNO7Q",4) failed: Input/output error (5) Jun 14 10:02:39 UNRAID logger: >f+++++++++ unsorted/test.file Jun 14 10:02:39 UNRAID kernel: REISERFS warning (device md1): jdm-20001 reiserfs_xattr_get: Invalid magic for xattr (user.org.netatalk.supports-eas.4VNO7Q) associated with [1919251317 1735552814 0x74656e2e UNKNOWN] Jun 14 10:02:43 UNRAID logger: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7] Jun 14 10:02:43 UNRAID logger: mover finished How did you create "test.file"? Please type these commands and paste the result in a post: getfattr -d /mnt/user/unsorted getfattr -d /mnt/user/unsorted/test.file Also, I have a /mnt/cache/apps share that is set to "cache disk only" - it is skipped by the mover, as expected, but it is always orange and showing pending cache. Right, orange just means there are files on the share at risk of being lost if the cache drive fails. It will always be orange for a cache-only share (until "cache pool" feature is added). The "apps" folder is also present in /mnt/user - is this desired behaviour? If I delete a file from /mnt/user/apps it is also deleted from /mnt/cache/apps. Yes this is correct. The "/mnt/user/<sharename>" tree provides a composite (combined) view of /mnt/cache/<sharename> /mnt/disk1/<sharename> : /mnt/diskN/<sharename> where N is the highest disk number you have installed. You can access a particular file/directory via /mnt/user/<sharename>/<file/directory name> or if you know which disk the object is on via /mnt/user/diskX/<sharename>/<file/directory> or if you know the object is on cache disk via /mnt/user/cache/<sharename>/<file/directory> Make sense? But here's something I should probably point out: Above is ok if you access any method other than AFP. If you have exported a particular usershare via AFP, you must only access the share via AFP and in particular do not write new files into the share via any other protocol. AFP is an "exclusive club" because netatalk maintains a separate database. If you start accessing files on an AFP share outside the AFP protocol, netatalk won't know about it and funny things can happen. Link to comment
joyless Posted June 14, 2013 Share Posted June 14, 2013 How did you create "test.file"? Please type these commands and paste the result in a post: getfattr -d /mnt/user/unsorted getfattr -d /mnt/user/unsorted/test.file i created it with 'touch test.file' command, inside /mnt/cache/unsorted getfattr command returned nothing: root@UNRAID:~# getfattr -d /mnt/user/unsorted root@UNRAID:~# getfattr -d /mnt/user/unsorted/test.file screenshot file that was in the log from my first post was uploaded via SMB Make sense? yep thanks, it's all clear now. But here's something I should probably point out: Above is ok if you access any method other than AFP. If you have exported a particular usershare via AFP, you must only access the share via AFP and in particular do not write new files into the share via any other protocol. AFP is an "exclusive club" because netatalk maintains a separate database. If you start accessing files on an AFP share outside the AFP protocol, netatalk won't know about it and funny things can happen. right, i don't know if it makes any difference, but i used to export my shares via AFP - it was slow, unreliable and causing all sorts of trouble, so right now AFP is turned off (Enable AFP: no) and I'm using SBM to view my shares Link to comment
madburg Posted June 14, 2013 Share Posted June 14, 2013 But here's something I should probably point out: Above is ok if you access any method other than AFP. If you have exported a particular usershare via AFP, you must only access the share via AFP and in particular do not write new files into the share via any other protocol. AFP is an "exclusive club" because netatalk maintains a separate database. If you start accessing files on an AFP share outside the AFP protocol, netatalk won't know about it and funny things can happen. Yes and No, the AFP DB behavior is separate to the issue of the damaged/corrupt apple xattr's. WARNING In order to be able to run -rf reconstructing the CNIDs in the database from the AppleDouble files, make sure you've run a -r rebuild sometimes before, where the CNIDs then would have been synched between database and AppleDouble files. Also be careful about the option nocnidcache. Avoid this option if at all possible, because if prevents you from being able to use -f. CNID background The CNID backends maintains name to ID mappings. If you change a filename outside afpd( (shell, samba), the CNID db will not know and not reflect that change. Netatalk tries to recover from such inconsistencies as gracefully as possible. The mechanisms to resolve such inconsistencies may fail sometimes, though, as this is not an easy task to accomplish. E.g. if several names in the path to the file or directory have changed, things may go wrong. If you change a lot of filenames at once, chances are higher that the afpds fallback mechanisms fail, i.e. files will be assigned new IDs, even though the file hasn't changed. I have seen this behavior "If you change a lot of filenames at once, chances are higher that the afpds fallback mechanisms fail, i.e. files will be assigned new IDs, even though the file hasn't changed." My Sabnzb, Sickbeard are Windows hosts working off unRAID via SMB. they handle the media placement on unRAID via SMB. My clients that read/play (only) media are Mac's via AFP. My Mac server running plex scrapes the media and it is this query for the files via AFP and unRAID user shares that kick off the AFP DB to update the inodes. With adding MAXDEBUG, I am able to monitor how the DB handles these changes, it not to bad, it replaces some, creates new, and deletes all as required and quite fast, but I have a permanent DB (which lives on a raid 10 array), that DB closes on array shutdowns so nothing is lost with reboots. Something you considered doing on the cache drive in the past. And I personally feel you should and place in your release notes that AFP requires a cache drive (period) in order to utilize (again just my 2 cents). I also wanted to discuss if there would be a way to have a AFP CLIENT running on unRAID there by all file changes by other protocols would be picked up in realtime and DB instantly updated. Not sure if I explained that all to well. But back to the topic on AFP extended attribute damage corruption "user.org.netatalk.supports-eas.xxx failed: Exec format error (" A quick search displayed how far back these damaged/corrupted apple xattr have been logged in syslog's (all unique and individual posts): http://lime-technology.com/forum/index.php?topic=22831.msg202446#msg202446 http://lime-technology.com/forum/index.php?topic=23846.msg210467#msg210467 http://lime-technology.com/forum/index.php?topic=26642.msg233956#msg233956 http://lime-technology.com/forum/index.php?topic=22012.msg195067#msg195067 http://lime-technology.com/forum/index.php?topic=23255.msg205387#msg205387 http://lime-technology.com/forum/index.php?topic=25689.msg227410#msg227410 http://lime-technology.com/forum/index.php?topic=20596.msg182557#msg182557 http://lime-technology.com/forum/index.php?topic=16135.msg149265#msg149265 http://lime-technology.com/forum/index.php?topic=16885.msg153967#msg153967 http://lime-technology.com/forum/index.php?topic=19972.msg177808#msg177808 http://lime-technology.com/forum/index.php?topic=25306.msg233774#msg233774 I believe that with the last xattr additions in the last few RC we are where we need to be, I just need to verify that this is the case and will update you with info as soon as I do and what I did to repair. It will be either tonight or after Sunday, best that I can do. Everyone needw a break and a great fathers day bash! Cheers Link to comment
joyless Posted June 14, 2013 Share Posted June 14, 2013 Tom, I have created new user share (lets say 'ABC'), set it to use cache, created a test file inside /mnt/cache/ABC and started the mover - it finished fine, without any errors. Next i exported ABC via SMB, uploaded a file from OS X and triggered the mover - it finished fine, again. So I guess the problem applies to my 'old' shares only, possibly those, that were previously exported via AFP? Link to comment
JonathanM Posted June 14, 2013 Share Posted June 14, 2013 Tom, I have created new user share (lets say 'ABC'), set it to use cache, created a test file inside /mnt/cache/ABC and started the mover - it finished fine, without any errors. Next i exported ABC via SMB, uploaded a file from OS X and triggered the mover - it finished fine, again. So I guess the problem applies to my 'old' shares only, possibly those, that were previously exported via AFP? "Watches madburg doing a victory dance at his PC" Link to comment
limetech Posted June 14, 2013 Author Share Posted June 14, 2013 Tom, I have created new user share (lets say 'ABC'), set it to use cache, created a test file inside /mnt/cache/ABC and started the mover - it finished fine, without any errors. Next i exported ABC via SMB, uploaded a file from OS X and triggered the mover - it finished fine, again. So I guess the problem applies to my 'old' shares only, possibly those, that were previously exported via AFP? Ok try this. Make a list of your data disks that contain the top-level directory "unsorted". For each one list the xattrs, for example, let's say 'unsorted' is on disk1, disk3 and disk7, you would type this: getfattr -d /mnt/disk1/unsorted getfattr -d /mnt/disk3/unsorted getfattr -d /mnt/disk7/unsorted Link to comment
madburg Posted June 14, 2013 Share Posted June 14, 2013 Tom, I have created new user share (lets say 'ABC'), set it to use cache, created a test file inside /mnt/cache/ABC and started the mover - it finished fine, without any errors. Next i exported ABC via SMB, uploaded a file from OS X and triggered the mover - it finished fine, again. So I guess the problem applies to my 'old' shares only, possibly those, that were previously exported via AFP? "Watches madburg doing a victory dance at his PC" Its not like that... I am very upset with this whole AFP thing I wont deny that. Its been treated as a bastard child in unRAID no doubt, BUT no one here can change the past, nor me get back all the wasted time, postings, troubleshooting, with besides Speeding_Ant no help from anyone. So if Tom does the right thing(s) then at the end of the day we'll all be good. Most important is we fix this particular issue and make some tweaks so it works "better", for a lack of words at the moment. I want to add, the reason I have not published publicly what I did to resolve this (for months) is because it can possibly ruin your data (lose of extended attribute you may have in your data, not everyone only has Movie, TV shows, there's plenty of data that carries extended data), so I rather share it with Tom, who could script up the remedy in a safe® manner (as Linux scripting, I am a beginner as best). The only puzzling thing was the amount and lack of these apple ea. attr's on various user shares that had me stumped (random no pattern). Tom has shared that info, even better the source, that showed they should be written, followed by deleted. So we shouldn't have any damaged/corrupted apple ea's nor good written ones! So I can now verify my fix, the behavior of the apple ea's after the fix (have the shfs debug flag now to see something else as well) and report back to Tom. Link to comment
JonathanM Posted June 14, 2013 Share Posted June 14, 2013 "Watches madburg doing a victory dance at his PC" Its not like that...Aww, cmon, it had to feel a little bit good to see that your pet issue was finally getting some quality attention from Tom. Link to comment
madburg Posted June 14, 2013 Share Posted June 14, 2013 OK You pushed me over the edge! see attachment. I don't care for apple, but I like plex and especially plex on mac, the skins, and the direct play from a mac plex client to the nas (only available via plex for mac over AFP) versus throught the plex server to the nas. I personally hope AFP die's a horrible death, now their posting how wonderful SMB2 is.. a$$holes finally figured out they cant do sh^t with this protocol with retarded DB. Link to comment
limetech Posted June 14, 2013 Author Share Posted June 14, 2013 Please let's stay on topic guys. Link to comment
limetech Posted June 14, 2013 Author Share Posted June 14, 2013 For -rc15 I changed a setting in AppleVolumes.default having to do with the "ea" setting: # ea -> none|auto|sys|ad # Specify how Extended Attributes are stores. default # is auto. # auto: try "sys" (by setting an EA on the shared # directory itself), fallback to "ad". Requires # writable volume for performing the test. # Note: options:ro overwrites "auto" with "none." # sys: Use filesystem EAs # ad: Use files in AppleDouble directories # none: No EA support # In prior versions the "ea" setting wasn't specified at all, so it defaulted to "auto", which means it's going to write a funny xattr to each exported directory (but then immediately delete it). Since our file system (reiserfs), and any file system we would be using in the future, supports extended attributes, I explicitly set "ea:sys". Link to comment
madburg Posted June 14, 2013 Share Posted June 14, 2013 Sounds like a good idea, as of which unRAID version Tom? sorry just re-read you mentioned starting in RC15. I was watching that and noticed some shares were being set differently then others and should not have been the case (this is a while back), so I think with all the update and hardcoding this, is a good thing. I couldn't wait so I VPN'ed in and work on all this now. I want to kiss u and hit u at the same time, so many emotions right now . Once I backed up (just a dump for history) the currently assigned apple ea' per share and removed them all, enable shfs logging, wish I knew about this long ago, as I want to see what was going on with all the reads and write upon array start when AFP was enabled. Guess what, NO MORE thousands reads and write to the array, quick set and remove of the apple ea's by shfs being logged. Moving on with the Plex Server now, which is going to mount the volumes. I'll have the entire syslog from boot -> on emailed to you. Link to comment
madburg Posted June 14, 2013 Share Posted June 14, 2013 (but then immediately delete it) You did fix the ability to properly delete until recently though, right? Link to comment
limetech Posted June 14, 2013 Author Share Posted June 14, 2013 (but then immediately delete it) You did fix the ability to properly delete until recently though, right? The fix so that ea's would be properly deleted from user share directories was added in -rc13. Link to comment
madburg Posted June 14, 2013 Share Posted June 14, 2013 Check your inbox, syslog and notes sent. That's all I could do for now. Looks very positive. Got to head on home now. Link to comment
Recommended Posts