Weird SyncToy issues when syncing to unRAID - unchanged files are synced

Hi there. Make yourself comfortable, this is a long and curious tale...

I recently bought an HP MicroServer gen8 and have loaded it up with unRAID Basic.

I used to use an ARM-based Cubietruck (with 2TB internal HDD) running Armbian (based on Debian) to store my music and control playback around the house using Logitech Media Server (LMS). I also have my music stored on my Windows 7 Pro PC, where I manage its metadata. I would use SyncToy to sync the music from my PC to the Cubietruck whenever I bought a new album.

I thought the same thing would work with unRAID, however SyncToy insists on syncing albums that haven't been modified - even for 5 or so years.

Some background:

I set up my unRaid with 1x 4TB Parity disk and 1x 4TB storage disk, then copied all my movies and TV shows over from my PC to an unRAID user share ('video') using robocopy. I then removed the 2TB HDD from my PC (which used to contain my movies etc), pre-cleared it, and added it to unRAID.

I then used SyncToy to copy the entire contents (ie from the root of the drive) of my 2TB 'audio' HDD over to another unRAID user share ('audio' - about 1.4TB of data).

I then installed a 500GB SSD cache drive, and set up Plex and LMS Dockers, storing the Docker image on the root of the cache drive, and creating a special cache only 'appdata' share for the configs.

However, I noticed that when I used LMS that both the 4TB (disk1) and the 2TB (disk3 - there is no disk2, as I'm keeping that bay free for a future upgrade) disk spun up, as my albums were spread across both, due to my disk allocation settings (high water, automatically split any directory as required).

So I used consld8 from the diskmv utilities to move all of the music I access using LMS onto disk 1. Gigabytes of tunes poured over to disk1.

I also arranged my other miscellaneous audio bits n pieces (things that I don't access through LMS) onto disk3 (my 2TB HDD) to free up some room on disk1.

It was about then that I realised how much musical junk I had stored, so I went through the HDD on my PC and decided to clean out a heap of the music I no longer listen to, and ran SyncToy in Synchronisation mode to propagate those changes and deletions to unRAID.

After SyncToy was running for a while, I noticed that it was replacing a heap of music on my PC's HDD - even though none of those files had been edited. I freaked out and hit stop.

I spent most of the day yesterday trying to get the two to happily sync, and even ran WinMerge. What was curious about WinMerge (I set the folder Compare Method to 'Modified Date' as it would take more than a day to do a full contents comparison) is that it only saw the expected differences - ie those files I had edited since I last synced, not the ones SyncToy wanted to move and which hadn't been touched for years. So I manually aligned the two folders by deleting anything I had deleted in my clean up, and copying over anything new. A manual sync, in effect (WinMerge made this surprisingly easy, as I could compare the folders between the two locations).

After this manual sync, unRAID had once again spread the files across multiple disks (I neglected to change the share settings), so I ran consld8 again, and then created a new user share ('mymusic') for the albums that LMS uses, forcing it to only use disk1. I then used midnight commander (KB and monitor plugged directly into unRAID, logged in as root) to move the files between the two user shares (almost instantaneous). I then renamed the old 'audio' share to 'miscaudio'.

I then ran SyncToy on the 'mymusic' user share, expecting it to say that the two folders were 'in sync' but it just started copying files over to the PC again.

So I decided to run it in Echo mode (meaning that it would only copy from the PC to unRAID), hoping that it would just align the two and work from then on. It copied gigabytes and gigabytes of old audio files from my PC to unRAID (those that I think were moved from unRAID to my PC during the Sync I ran after cleaning up my music folders? It's hard to know - these aren't files that should have been modified in any way). When it finished, I thought I'd then run it in sync (ie two-way) mode again (to see if the two disks were now truly 'in sync'), and curiously, it started copying unchanged files back over from unRAID to my PC, just like it had that morning.

I can't figure out what has changed on those files that makes SyncToy think that they've been modified, and I can't figure out what process caused that change. Most importantly, I can't figure out how to get SyncToy to work properly again.

Worse, I fear that maybe the files have been damaged during some copy (consld8, maybe?), and that's why SyncToy thinks they've been modified. As such, I'm not too interested in running a sync from unRAID > PC, just in case they are somehow corrupt. I now only have one backup of these files around (a Mac-encrypted external HDD - stupidly, I ran a sync on my ARM-based server yesterday, propagating the modified PC files to it, too), and it's a few weeks out of date/currency. I'm currently running a Quick compare using WinMerge, to see if there really are differences between my unRAID copies and those on my PC, but it's going to take hours to run.

Some possible causes for SyncToy to confuse unmodified files for modified ones:

• Consld8 changed the modified times of the files, causing SyncToy to think that they'd changed (even though the author said that all attributes are maintained).
  
• The Mover changed the modified times of the files, causing SyncToy to think they'd changed.
  
• Having SyncToy run on a User share, rather than a disk share (which I don't have turned on), causes something funny in the way SyncToy looks at files.
  

Note that after fiishing copying any files to unRAID (whether the manual sync I did or a SyncToy run), I would always run The Mover before running another sync.

I guess I'm posting here to find out if anyone else has had a similar problem with SyncToy, or can shed light onto why SyncToy thinks the files have changed, or can offer a way to get SyncToy to work again (I really need a reliable method to sync from my PC to unRAID - so I'm open to alternative syncing methods, too).

I regularity use FreeFileSync to sync files between my PC and unRAID and it only copies changed files as far as all my tests show.

I regularity use FreeFileSync to sync files between my PC and unRAID and it only copies changed files as far as all my tests show.

Thanks, I'll look into that.

I'm going to wait for WinMerge's file comparison to complete first, though.

FWIW, the quick comparison run by WinMerge shows all the files as being identical, so it seems that there have been no corruptions. Further, I can't find any files that have different time stamps, so I have NO IDEA why SyncToy insists on syncing extra files.

I shall try running FreeFile Sync now and see if it also thinks that files need to be synced. Thanks itimpi for the tip.

EDIT: FreeFileSync works a treat! Looks like a great (and fast) program. Also handy that it runs from the command line, and on Mac. Thanks again, itimpi!