For a long time I have been using Microsoft’s SyncToy to backup data on my Windows boxes over the network to my Linux box. Every few weeks (in reality it was months) I would also use it to copy that same data to an external drive for the off-site backup.
Not any more.
When I first started using SyncToy I was satisfied that it was copying all files. Recently I discovered one of two things: back in the beginning I didn’t check properly, or the behaviour of SyncToy has changed since then.
So what is the problem?
The SyncToy setting I have been using (at least on the recent versions) is ‘Echo’ which is described as:
“New and updated files are copied left to right. Renames and deletes on the left are repeated on the right.”
At face value this is what I wanted, a mirror of the local files to a network share. Unfortunately I didn’t take this description literally enough, SyncToy will ONLY echo changes that are made on the ‘left’ side. What I need (for example when rotating through external hard drives) is a proper sync that analyses both source and destination to determine the differences that need to be copied (you know, like rsync).
So if for some reason files on the destination (‘right’ in SyncToy terminology) go missing or get corrupted, SyncToy doesn’t care. In the case where I am using a pair of identical external drives that I swap between home and work every couple of weeks, data that is copied to one drive is then not copied to the other drive a few weeks later.
What really confuses me is a step that the latest version of SyncToy no longer performs, which is how I noticed this (and then found that many others already knew). It used to be that when the sync ran (immediately after login) I could see it walking the destination file tree, both via network activity and in the samba logs. Why? If SyncToy doesn’t care about the destination, what is the point of this scan? Obviously they figured out that it was redundant and it was removed.
So what have I done?
Ideally I wanted a realiable win32 port of rsync that didn’t require me to install Cygwin. But without that I started looking into alternatives and I settled on Robocopy. Yes, another tool from Microsoft. For XP it is obtained from the Windows Server 2003 Resource Kit, but it is standard for Vista and 7.
Robocopy is a command line tool (there is a GUI available) which is fine with me as I want to script it. Which I have done and I now have two scripts. One to run at login which backs up local data across the network, and a second script which backs up the same data to an external hard drive. This second script also pulls other data (such as my email, etc) from the Linux box to the external hard drive.
One important option that I need to specify is /FFT which tells it to ‘assume FAT File Times’ as apparently the FAT file times are not as accurate as you would expect. But I’m copying from NTFS to ext3, FAT or FAT32 is not involved, but in between those two file systems is Samba, whose SMB implementation has similar time accuracy problems as FAT.
It has now been a week and the backups are working correctly. Hopefully it stays that way.