Re-writing file on user share not removing old copy

Can anyone explain the circumstances in which this might occur, and how to resolve?

 

Thunderbird (via Dovecot) keeps a file listing which mail folders are subscribed to.  If I update this list, the new subscriptions file always gets written to the last drive in the user share (most free).  However, the older copy of the file is still left on a lower number drive, and not deleted, leaving a duplicate.  The cache drive is not enabled for this user share.  When the file is read, it is always the older (lower drive number) version of the file which is read.  The old and new files both have the same ownership and permissions.  This is completely reproducible - it occurs every time.

 

There is nothing in the logfile relating to the file creation other than the standard

Jun 23 12:24:29 Tower shfs/user: duplicate object: /mnt/disk4/Maildir/ruby/subscriptions (Minor Issues)

messages.

 

Isn't unRAID meant to prevent duplicates, or is there a known set of circumstances which can allow this to occur?

 

Is there a way to resolve this?

 

I'm currently running v5rc4, but have no reason to believe that this is a v5 problem.

I'm not familiar with the inner workings of Thunderbird, but it sounds as if it is attempting to replace a file that it still has locked. This is possible on the host system, but I don't think any network protocol allows it. When you replace a file on UnRAID via smb, the new file is copied and then the old one is deleted. So, perhaps it is having trouble deleting the old file because Thunderbird still has it locked.

 

The only way I can think to fix this would be to keep your Thunderbird files locally and use rsync or another tool to sync these files to/from your NAS.

Thanks for the reply and suggestion ...

 

First of all, I'm not using smb, but nfs - if that makes any difference.

 

Secondly, I'm not seeing any error relating to a failure to delete.  However, the Thunderbird Error Console is full of these messages:

 

Error: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIMsgFolder.getStringProperty]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: chrome://messenger/content/folderPane.js :: getSmartFolderName :: line 2473"  data: no]
Source File: chrome://messenger/content/folderPane.js
Line: 2475

 

However, these messages are not generated specifically at the time I'm editing the subscriptions but are produced repeatedly whenever the main Thunderbird window is open.  A quick google shows that this is a reported and logged fault.

 

Thirdly, I cannot see a way to configure the location of the file ... if I could, I would simply move it to a disk share, rather than a user share.

 

Fourthly, if I ensure that the only copy of the file is on the highest drive number (most free), then the file gets updated (overwritten?) exactly as expected when I modify subscriptions.

 

If this is a problem due to a file lock being applied during the update, then it would seem, to me, that unRAID is not showing a transparent interface to its multiple drive user shares.  Also, if this is a problem with a file being locked during update, then it must be a very common situation and would affect many applications, not just Thunderbird/Dovecot.

This does sound like an issue with the introduction of non-cache user shares.

 

I'm going to move it to the "rc" forum for more visibility.  It does sound like a bug to me.

 

Joe L.

Thanks Joe ... it certainly feels like a bug - but whether old or new, I cannot say..

 

Just one, possibly important, point that I've just realised - my Maildir user share is not actually shared, neither by nfs nor smb.  All access is local access by Dovecot running on the unRAID server.  Therefor, it is always accessed as /mnt/user/Maildir and so any file locking is done locally on unRAID, not via any network protocol.  Since the directory is not shared, the mail client (Thunderbird) cannot access the files directly but only via IMAP to Dovecot.

PeterB-

Please re-test this using SMB as your connection method.  I know you hate SMB and love NFS, but nevertheless, this will provide some valuable information.

PeterB-

Please re-test this using SMB as your connection method.  I know you hate SMB and love NFS, but nevertheless, this will provide some valuable information.

 

As I said in my last post, this isn't actually an smb/nfs issue - the user share isn't really shared ... it's just being accessed locally and simply created as a user share to allow disk spanning.

 

I guess that I could work around the problem by re-distributing my data so that my Maildir is held on a single drive (not a share) ... however, I do believe that there is an issue here, and I will do what I can to assist you in tracking it down.

PeterB-

Please re-test this using SMB as your connection method.  I know you hate SMB and love NFS, but nevertheless, this will provide some valuable information.

 

As I said in my last post, this isn't actually an smb/nfs issue - the user share isn't really shared ... it's just being accessed locally and simply created as a user share to allow disk spanning.

 

I guess that I could work around the problem by re-distributing my data so that my Maildir is held on a single drive (not a share) ... however, I do believe that there is an issue here, and I will do what I can to assist you in tracking it down.

 

Sorry missed that... to help me figure this out, please do this:

 

Copy the attached file, called "extra.cfg", to the 'config' directory of your flash.  There is a line in there that turns on debugging information for the user share file system.  With this file in place, Stop the array and the Start it again.  Next, do your file update that causes this issue, and then capture the system log and send to me: [email protected]

 

The log will be filled with numerous debug messages and your system might run slower because of it, but this will generate a trace of the user share file system operations.

 

To restore normal operation, Stop the array and delete the file, then Start array again.

extra.cfg

Just a quick note to say that Tom has isolated the problem (which has always been present in unRAID), and it will be fixed in the next RC.

 

Briefly, as the 'subscriptions' file is updated, the new one is created as 'subscriptions.lock'.  When the new file has been completely written, it is renamed to 'subscriptions'.  If the two files happen to be on the same drive, all is well and the old file is implicitly deleted by the rename.  If the two files are on different drives, the implicit delete does not happen - hence duplicate files.

Good to hear. So, if you create a file with a unique name on a user share and then rename it, intending to replace another file, a duplicate will be created? What is the fix? Replace the file or generate error?

The fix is to wait until it's fixed by LimeTech in their user-space filesystem handler, 'shfs'.

 

The likely fix internally to shfs is an additional check on file rename/move operations to check if a file already exists by that exact name on another disk. If it does, then you'd have to delete the original file before renaming the new file (quicker to do) OR you'd have to move the entire contents of the file to be renamed into the original file (takes longer to do). A third possibility is in changing the way shfs deals with duplicate files, perhaps instead of returning the first file found when searching sequentially from disk1 through diskn is to return the most-recently modified file.

Automatically pruning old duplicates is desired.

Automatically pruning old duplicates is desired.

I'm hesitant to advocate automatically deleting anything.

If it were on the same disk, the automatically deleting the old content is what would have happened. Fixing the situation so it occurs if on separate disks would seem to be the desired result.

If it were on the same disk, the automatically deleting the old content is what would have happened. Fixing the situation so it occurs if on separate disks would seem to be the desired result.

I agree. In this particular situation, the external application implicitly expects that when it overwrites a file, it's really gone. I was responding to the generic statement that duplicates should automatically be removed by unraid, without regard to how they were created.

Oh yes. In this particular case I can see the expected behavior being to delete the duplicate file.

 

However, in all other cases where a move/rename operation is not done via shfs, I would absolutely not want shfs to delete the duplicate file(s) when it detects one. That would be crazy. 

However, in all other cases where a move/rename operation is not done via shfs, I would absolutely not want shfs to delete the duplicate file(s) when it detects one. That would be crazy. 

agreed...