A history of rsync on Mac OS X

I think Apple included the rsync binary with Mac OS X since at least version 10.2 Jaguar, but it couldn’t be used for backups of most Mac data due to its lack of support for resource forks, which were and are prevalent on the Mac.

Rsync first got attention on Mac OS X when Kevin Boyd of University of Michigan added support for resource forks to rsync with a port he called RsyncX. RsyncX also included a simple GUI for setting up backups. The GUI generated shell scripts and scheduled them in cron. The GUI was buggy and generated scripts with minor, but serious flaws. The scripts also lacked some necessary features that a backup solution would provide. But with the lack of a good backup solution (just say no to Retrospect), some admins including myself took the time to customize and extend the scripts that RsyncX created, or just created new ones from scratch. I added features to my scripts like checking for free disk space, rotating incremental backups with hard links, checking for a mounted volume before running, failure notifications, etc.

RsyncX had other flaws though. It was based on older rsync code which contained bugs and at least one serious security vulnerability. One of the bugs results in rsync getting stuck in an endless loop if files are changing while rsync is running. If you have logging enabled, rsyncx will quickly fill your drive while it is stuck in this loop — really bad news. My workaround for this was to spawn rsync off and watch its log every few seconds for telltale signs that it was stuck in a loop. I would then kill rsync and restart it. Ugly, but it actually works. Oh yeah, it also doesn’t know how to handle locked files (uchg), so I have to unlock all files on the destination before the sync. Ugh. Oh and it throws lchown errors on symlinks. Just grep -v them out.

When Apple began talking about the features of Mac OS X 10.4 Tiger, one of the big features for command line unix geeks was support for resource forks in all command line tools, including rsync. By this time those of us using RsyncX were getting pretty tired of all of the workarounds and bugs, so an Apple supported rsync that handled resource forks was awesome news.

When rsync was released with 10.4.0, it has serious bugs. It was basically unusable. It would crash unexpectedly with large data sets. It would incorrectly set modification dates, which would then cause all future syncs to re-copy all files. I stayed with rsyncx to handle production syncs with the additional slap in the face of deprecated warnings filling the system log. With each new release, I would test Apple’s included rsync. I filed bugs. I used a developer support incident. Finally around 10.4.9 and later, rsync did actually seem to work, at least well enough. In real world tests though, Apple’s rsync speed was and is dismal on large data sets. The problem lies in the fact that resource forks do not have a modification date to compare when syncing. Without this key piece of data, there is no easy way of knowing whether the resource fork of a file has changed or not. Apple’s solution to this problem was to ALWAYS COPY the resource fork. That’s right, if your data has resource forks, you copy the resource data every time rsync runs. Yes, resource forks are typically small, but they add up, and for terabytes of small files, the I/O causes an rsync of unchanged data that should take about 30 minutes to instead take 4 hours.

So again I stick with RsyncX.

Leopard’s rsync appears to be virtually unchanged from Tiger’s at least to me. It still copies the resource fork every time.

Mike Bombich apparently likes rsync too. He is using it for the sync engine in Carbon Copy Cloner 3. He includes an updated patched version in the bundle here: Carbon\ Copy\ Cloner/Carbon\ Copy\ Cloner.app/Contents/Resources/ccc_helper.app/Contents/Resources/rsync

One nice thing about this binary is that it has a patch to optionally checksum resource forks (–ea-checksum) to prevent them from being copied unnecessarily. Cool, but checksumming adds time, so I’ll need do some real world tests.

Rsync version 3.0 is in prerelease and includes built-in ACL and extended attribute support. Hopefully this includes some way of handling unchanged resource forks.

Until Apple adds some sort of date stamp to resource forks, checksums may be the only safe way to handle them. I suspect that addressing the issue properly will have to wait for a new filesystem.

Advertisements

7 thoughts on “A history of rsync on Mac OS X

  1. Pingback: Pants in Motion » Blog Archive » Rsync: This is Your Life

  2. Pingback: rsync 3 report update « Pattern Buffer:

  3. Pingback: Rsyncx to backup Mac | Tongfamily.com

  4. My experience on rsync : Actually I compiled a recent version rsync 3.0.3 in both server an client machines. It was working fine with logs generating at server side too. But all of sudden the server side log generation is stopped. I checked backup is fine. so now I started taking the logs generating in the client side. Also rsync fails to copy/stat the files which are not having enough file permission, in that i used changing the file permission and copied again.

    It will happy news if Apple look into these kind of issues with the rsync and support.

  5. I have been using Rsync on my Windows/Ubuntu network to backup to FreeNAS backup and ftp servers in the attic at home for a year or so now.
    I have just ordered a new Mac Pro and had been wondering about rsync on Mac since then.
    Shame to hear it is still a bit buggy, but I only have max of about 600Gb data to backup, not TB’s

    Really submitting this comment so I can tick the “Notify me of follow-up comments” check box

  6. Pingback: seven geek wonders » Blog Archive » Understanding Leopard’s Time Machine

  7. Pingback: rSync when Time Machine fails « macsynology

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s