mover does not preserve directory mtime


Recommended Posts

i think they are preserved because if they weren't XBMC would see the movie again and add another copy (before/after mover).

 

Luckily XBMC uses a CRC of the path as the unique identifier so whilst it might be detected as changed it wont be added twice.

Link to comment
I guess the question is why?

 

I use time stamps for organizational purposes, and it gets confusing when some data looks newer than others

 

Example: If I put a folder of content on a cache enable share right now, and then a second folder 12 hours from now, when I look at my share tomorrow, after the script has run, it will look like they were created moments apart,. Plus the first folder I created might actually appear as the most recently created. Now multiple that by 50 folders, or however much stuff I've been working with, and you can see how easily it is to get lost

 

and rsync is used in the mover, not cp.

 

I'm aware of that, I actually looked into the script before posting, I just wanted to provide an example of what I was looking for. Trying to be thorough

 

I always assumed the timestamps were preserved. It seems the only "right" way to do it to me. Is there some gotcha I am not considering?

 

I guess I should have clarified the request is for folder timestamps. File timestamps are kept just fine

Link to comment
As far as I am concerned the mover should not modify any timestamps in any way. If it does then I suspect it is a bug. I can think of no logical reason why it would be otherwise.

 

Must be a bug then, unless there's a conflict with the switches that are currently in place (rsync -i -dIWRpEAXogtO --numeric-ids --inplace). My mover is set to run at 3:40AM in the Web GUI, and I just checked the folders it moved over this morning. The first thing it moved has a folder creation date of 3:40AM, the next one 3:43AM, and it keeps going from there

 

I also had a folder on my cache drive from earlier this morning, went and manually clicked to invoke the mover, and sure enough, it updated the timestamp of the folder to 11:01AM

Link to comment

Yes, this is a problem.  In the 5.0 release, I added the -O option to rsync command in the mover script, but this was wrong.  What this does is cause all directories that get moved to have their mtime's changed to the time of move.  I will remove this option in 5.0.4, but this will not solve this reported bug.

 

The issue is that when an object is moved from a containing-directory on the cache drive, to an array disk that where the containing-directory already exists, the mtime of the containing-directory on the array disk will get updated, and I'm not sure how to prevent that...

Link to comment

The issue is that when an object is moved from a containing-directory on the cache drive, to an array disk that where the containing-directory already exists, the mtime of the containing-directory on the array disk will get updated, and I'm not sure how to prevent that...

 

The mover is working on a file by file basis rather than a directory basis.

This is why the directory mtimes on the destination get updated.

 

We would either need to cache the directory mtime or change the mover to do rsync based on the directory name. The issue then becomes, how to check if the file is busy.

 

I suppose you could do the rsync based on each file, then the directory, however once the file is moved, the source directory has been updated, so this would not work. I'll try to think of other ideas.

 


for Share in */ ; do
  if [ -d "/mnt/user0/$Share" ]; then
    echo "moving $Share"
    find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \
      -exec rsync -i -dIWRpEAXogtO --numeric-ids --inplace {} /mnt/user0/ \; -delete
  else
    echo "skipping $Share"
  fi
done

Link to comment

The issue is that when an object is moved from a containing-directory on the cache drive, to an array disk that where the containing-directory already exists, the mtime of the containing-directory on the array disk will get updated, and I'm not sure how to prevent that...

 

The mover is working on a file by file basis rather than a directory basis.

This is why the directory mtimes on the destination get updated.

 

We would either need to cache the directory mtime or change the mover to do rsync based on the directory name. The issue then becomes, how to check if the file is busy.

 

I suppose you could do the rsync based on each file, then the directory, however once the file is moved, the source directory has been updated, so this would not work. I'll try to think of other ideas.

 


for Share in */ ; do
  if [ -d "/mnt/user0/$Share" ]; then
    echo "moving $Share"
    find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \
      -exec rsync -i -dIWRpEAXogtO --numeric-ids --inplace {} /mnt/user0/ \; -delete
  else
    echo "skipping $Share"
  fi
done

 

Yeah, it's a pretty complex problem.  Have to do this:

 

for each file moved

  capture the mtime of the file and each component in the files path

  copy the file from cache to array [creating parent dirs as necessary]

  set the mtime of the new file

  set the mtime of each component of the files path on the array to the mtime captured above

  delete the file on the cache [this will cause files immediate parent directory mtime to change]

  set the mtime of the files parent directory back to what it was before

endfor

 

Might be easier to just write a program to do this.

Link to comment

Also consider... While I know this is an issue which can be programmed.

This might be one of those topics that the community could assist with.

 

I bet people would be more inclined to prioritize 64bit unRAID then this. I could be wrong.

Agreed, which is why I haven't done it.  Also the copy has to preserve ownership, mode and extended attributes, which is why rsync was used (cp doesn't preserve xattrs).

Link to comment
  • 1 year later...
  • 1 month later...

Any updates to this?

 

I'm thinking about switching back to cache drive to resolve the "out of disk space" errors I'm getting when running low on most of my drives

 

This can only be fixed with a custom mover program that patches directory mtimes.  There's hope for this to get in to 6.x because we have other functions to be implemented in a mover program.

Link to comment
  • 1 year later...

 

On 12/6/2013 at 11:45 AM, limetech said:

The issue is that when an object is moved from a containing-directory on the cache drive, to an array disk that where the containing-directory already exists, the mtime of the containing-directory on the array disk will get updated, and I'm not sure how to prevent that...

 

Does this include the root of a share?  It seems like that will always exist.  I have tried a few times to copy data from an old computer to an EMPTY cache enabled unRAID share. I have managed to preserve the directory timestamps while copied to the cache, but every time the mover executes, the directory timestamps reflect the time the mover executed.

 

On 6/5/2015 at 2:03 PM, limetech said:

This can only be fixed with a custom mover program that patches directory mtimes.  There's hope for this to get in to 6.x because we have other functions to be implemented in a mover program.

 

Any update to integrating in 6.x?

Link to comment
  • 9 months later...

I was reading the change notes for 6.4.0 and jumped through the roof when I read there is a fix for this.   A big thank you!

 

Quote

Improved shfs/mover (-rc1)

 

The LimeTech user share file system (shfs) has been improved in two areas.  First, we now make use of FUSE read_buf/write_buf methods.  This should result in significant throughput increases.  Second, the mover script/move program no longer uses rsync to move files/directories between the cache pool and the parity array.  Instead the move program invokes a new shfs ioctl() call.  This should result in complete preservation of all metadata including atime and mtime.

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.