EGOvoruhk Posted December 6, 2013 Share Posted December 6, 2013 Can I make a simple request for the next release (or help changing it on my own)? I want the mover script to preserve timestamps when moving the files, a la: cp -r --no-preserve=mode --preserve=timestamps Quote Link to comment
prostuff1 Posted December 6, 2013 Share Posted December 6, 2013 Can I make a simple request for the next release (or help changing it on my own)? I want the mover script to preserve timestamps when moving the files, a la: cp -r --no-preserve=mode --preserve=timestamps I guess the question is why? and rsync is used in the mover, not cp. Quote Link to comment
NAS Posted December 6, 2013 Share Posted December 6, 2013 I always assumed the timestamps were preserved. It seems the only "right" way to do it to me. Is there some gotcha I am not considering? Quote Link to comment
mr-hexen Posted December 6, 2013 Share Posted December 6, 2013 i think they are preserved because if they weren't XBMC would see the movie again and add another copy (before/after mover). Quote Link to comment
NAS Posted December 6, 2013 Share Posted December 6, 2013 i think they are preserved because if they weren't XBMC would see the movie again and add another copy (before/after mover). Luckily XBMC uses a CRC of the path as the unique identifier so whilst it might be detected as changed it wont be added twice. Quote Link to comment
EGOvoruhk Posted December 6, 2013 Author Share Posted December 6, 2013 I guess the question is why? I use time stamps for organizational purposes, and it gets confusing when some data looks newer than others Example: If I put a folder of content on a cache enable share right now, and then a second folder 12 hours from now, when I look at my share tomorrow, after the script has run, it will look like they were created moments apart,. Plus the first folder I created might actually appear as the most recently created. Now multiple that by 50 folders, or however much stuff I've been working with, and you can see how easily it is to get lost and rsync is used in the mover, not cp. I'm aware of that, I actually looked into the script before posting, I just wanted to provide an example of what I was looking for. Trying to be thorough I always assumed the timestamps were preserved. It seems the only "right" way to do it to me. Is there some gotcha I am not considering? I guess I should have clarified the request is for folder timestamps. File timestamps are kept just fine Quote Link to comment
NAS Posted December 6, 2013 Share Posted December 6, 2013 As far as I am concerned the mover should not modify any timestamps in any way. If it does then I suspect it is a bug. I can think of no logical reason why it would be otherwise. Quote Link to comment
EGOvoruhk Posted December 6, 2013 Author Share Posted December 6, 2013 As far as I am concerned the mover should not modify any timestamps in any way. If it does then I suspect it is a bug. I can think of no logical reason why it would be otherwise. Must be a bug then, unless there's a conflict with the switches that are currently in place (rsync -i -dIWRpEAXogtO --numeric-ids --inplace). My mover is set to run at 3:40AM in the Web GUI, and I just checked the folders it moved over this morning. The first thing it moved has a folder creation date of 3:40AM, the next one 3:43AM, and it keeps going from there I also had a folder on my cache drive from earlier this morning, went and manually clicked to invoke the mover, and sure enough, it updated the timestamp of the folder to 11:01AM Quote Link to comment
limetech Posted December 6, 2013 Share Posted December 6, 2013 Yes, this is a problem. In the 5.0 release, I added the -O option to rsync command in the mover script, but this was wrong. What this does is cause all directories that get moved to have their mtime's changed to the time of move. I will remove this option in 5.0.4, but this will not solve this reported bug. The issue is that when an object is moved from a containing-directory on the cache drive, to an array disk that where the containing-directory already exists, the mtime of the containing-directory on the array disk will get updated, and I'm not sure how to prevent that... Quote Link to comment
WeeboTech Posted December 9, 2013 Share Posted December 9, 2013 The issue is that when an object is moved from a containing-directory on the cache drive, to an array disk that where the containing-directory already exists, the mtime of the containing-directory on the array disk will get updated, and I'm not sure how to prevent that... The mover is working on a file by file basis rather than a directory basis. This is why the directory mtimes on the destination get updated. We would either need to cache the directory mtime or change the mover to do rsync based on the directory name. The issue then becomes, how to check if the file is busy. I suppose you could do the rsync based on each file, then the directory, however once the file is moved, the source directory has been updated, so this would not work. I'll try to think of other ideas. for Share in */ ; do if [ -d "/mnt/user0/$Share" ]; then echo "moving $Share" find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \ -exec rsync -i -dIWRpEAXogtO --numeric-ids --inplace {} /mnt/user0/ \; -delete else echo "skipping $Share" fi done Quote Link to comment
limetech Posted December 9, 2013 Share Posted December 9, 2013 The issue is that when an object is moved from a containing-directory on the cache drive, to an array disk that where the containing-directory already exists, the mtime of the containing-directory on the array disk will get updated, and I'm not sure how to prevent that... The mover is working on a file by file basis rather than a directory basis. This is why the directory mtimes on the destination get updated. We would either need to cache the directory mtime or change the mover to do rsync based on the directory name. The issue then becomes, how to check if the file is busy. I suppose you could do the rsync based on each file, then the directory, however once the file is moved, the source directory has been updated, so this would not work. I'll try to think of other ideas. for Share in */ ; do if [ -d "/mnt/user0/$Share" ]; then echo "moving $Share" find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \ -exec rsync -i -dIWRpEAXogtO --numeric-ids --inplace {} /mnt/user0/ \; -delete else echo "skipping $Share" fi done Yeah, it's a pretty complex problem. Have to do this: for each file moved capture the mtime of the file and each component in the files path copy the file from cache to array [creating parent dirs as necessary] set the mtime of the new file set the mtime of each component of the files path on the array to the mtime captured above delete the file on the cache [this will cause files immediate parent directory mtime to change] set the mtime of the files parent directory back to what it was before endfor Might be easier to just write a program to do this. Quote Link to comment
WeeboTech Posted December 9, 2013 Share Posted December 9, 2013 Might be easier to just write a program to do this. I thought the same thing. Only I might write a wrapper for rsync since I'm a lil lazy. Cache the directory mtime, do the rsync, utime() the directory to the cached value. There's also FTW64() which could traverse a tree. Quote Link to comment
WeeboTech Posted December 9, 2013 Share Posted December 9, 2013 Also consider... While I know this is an issue which can be programmed. This might be one of those topics that the community could assist with. I bet people would be more inclined to prioritize 64bit unRAID then this. I could be wrong. Quote Link to comment
limetech Posted December 9, 2013 Share Posted December 9, 2013 Also consider... While I know this is an issue which can be programmed. This might be one of those topics that the community could assist with. I bet people would be more inclined to prioritize 64bit unRAID then this. I could be wrong. Agreed, which is why I haven't done it. Also the copy has to preserve ownership, mode and extended attributes, which is why rsync was used (cp doesn't preserve xattrs). Quote Link to comment
EGOvoruhk Posted April 13, 2015 Author Share Posted April 13, 2015 Any updates to this? I'm thinking about switching back to cache drive to resolve the "out of disk space" errors I'm getting when running low on most of my drives Quote Link to comment
limetech Posted June 5, 2015 Share Posted June 5, 2015 Any updates to this? I'm thinking about switching back to cache drive to resolve the "out of disk space" errors I'm getting when running low on most of my drives This can only be fixed with a custom mover program that patches directory mtimes. There's hope for this to get in to 6.x because we have other functions to be implemented in a mover program. Quote Link to comment
dboonthego Posted March 8, 2017 Share Posted March 8, 2017 On 12/6/2013 at 11:45 AM, limetech said: The issue is that when an object is moved from a containing-directory on the cache drive, to an array disk that where the containing-directory already exists, the mtime of the containing-directory on the array disk will get updated, and I'm not sure how to prevent that... Does this include the root of a share? It seems like that will always exist. I have tried a few times to copy data from an old computer to an EMPTY cache enabled unRAID share. I have managed to preserve the directory timestamps while copied to the cache, but every time the mover executes, the directory timestamps reflect the time the mover executed. On 6/5/2015 at 2:03 PM, limetech said: This can only be fixed with a custom mover program that patches directory mtimes. There's hope for this to get in to 6.x because we have other functions to be implemented in a mover program. Any update to integrating in 6.x? Quote Link to comment
limetech Posted March 8, 2017 Share Posted March 8, 2017 11 minutes ago, dboonthego said: Any update to integrating in 6.x? I was just looking for this topic, and here you are posting in a 2-year old thread... This related to: Quote Link to comment
dboonthego Posted March 8, 2017 Share Posted March 8, 2017 26 minutes ago, limetech said: I was just looking for this topic, and here you are posting in a 2-year old thread... Good to know. Thank you. Quote Link to comment
dboonthego Posted December 16, 2017 Share Posted December 16, 2017 I was reading the change notes for 6.4.0 and jumped through the roof when I read there is a fix for this. A big thank you! Quote Improved shfs/mover (-rc1) The LimeTech user share file system (shfs) has been improved in two areas. First, we now make use of FUSE read_buf/write_buf methods. This should result in significant throughput increases. Second, the mover script/move program no longer uses rsync to move files/directories between the cache pool and the parity array. Instead the move program invokes a new shfs ioctl() call. This should result in complete preservation of all metadata including atime and mtime. Quote Link to comment
limetech Posted December 16, 2017 Share Posted December 16, 2017 1 hour ago, dboonthego said: I was reading the change notes for 6.4.0 and jumped through the roof when I read there is a fix for this. A big thank you! Where have you been, man? Quote Link to comment
dboonthego Posted December 19, 2017 Share Posted December 19, 2017 Been waiting patiently Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.