Ed's Sandbox

Scripting RAR for Automated Backup

I've directed some effort into finding a reliable but painless way to back-up my more important data in case one of my hard-drives dies, as they occasionally do from time to time (I've actually been very lucky with my drives but I don't want to leave years worth of data to chance)

Since I have two hard-drives in my PC, my ideal solution will simply mirror the data I care about such that it exists on both drives. If either drive fails, the data can be recovered from the other one. Of course this doesn't help if the whole computer goes up in flames, but it covers the most common case.

My first idea was to use ntbackup, which comes with Windows (at least my XP Pro has it.) You can launch it most easily by going to Start->Run->ntbackup.exe It lets you check off the files/directories you desire to be backed up, as well as specifying a location (in my case, on another drive) where the archive will go.

The result of ntbackup is a .bkf file ("Windows Backup File") which appears to be a compressed archive, which opens with ntbackup as well.

This approach works but I didn't like it for a few reasons:

• (Seemingly) poor session handling - I want to backup the same paths every time, but while ntbackup appears to let me save the session, the entire contents of the drive appear selected when I load it. Maybe I am missing something - it looks like I have to manually select what I want to back up each time.



• ntbackup has "incremental backup" support but in a way that seems better suited for tape backup than for what I am trying to do. Incremental backup is simply an additional backup file meant as a "patch" to the original full backup. The next incremental backup is a patch to the prior incremental backup file, etc. Meaning the backup files are not self-contained and it may require multiple backup files to restore a particular file. This also bloats up the size of the archive, if there's a frequently modified large file in the set (e.g. my Outlook folder). Doing a full backup each time takes too long.



• The .bkf format will probably be hard to restore from on a non-windows system, if I should ever need to. Also, it's not clear if I could open it on an earlier or later version of Windows itself.

With all of the above in mind, I set out searching for a better back strategy. My first thought was that I could simply use the Windows shell command xcopy to mirror some folders onto another drive but this presented a few difficulties - mainly the apparent need to copy everything each time (there's some magic for copying based on the archive flag or creation time but it didn't seem very straight forward. Plus, this option lacked compression.

The solution I ended up with is to use rar.exe which comes with WinRar, using command line options to make sure the following happens:

• Files which have not changes since the last archival should not be touched in the archive.


• Files should be compressed within the archive However, to minimize archival times, files which are already compressed (.RARs, specifically) should be simply stored within the archive.
  
• Files should be tested for integrity as they're added to the archive.


• The maximum amount of recovery data should be stored to give me a shot of recovering my files even if some of the archive is corrupt.

Here's the syntax:
"C:\Program Files\WinRAR\rar.exe" u -r -rr10 -t -m3 -msrar "e:\MIRROR of STUFF for BACKUP\DriveDbackup.rar" D:\Documents D:\Pictures D:\Outlook_archive.rar Outlook_main.pst

The 'u' parameter tells RAR to Update files within the archive - i.e. replace modified files with their newest version, and add new files. I believe this option does NOT remove files from the archive if they had been removed from the target. This is not a big deal, but if I decide to rework the file layout of my data, I will probably end up building a new archive from scratch, because if I move 500mb of pictures from D:\Pictures\Misc\Folder001 to D:\Pictures\Vacations\2005\Paris, the Update operation of the archive will end up keeping it in the old location and adding it in the new place. A new archive can be created by simply replacing 'u' with 'a', OR deleting/moving the archive from its location.

The -r option tells RAR to recursively look at the contents of subdirectories of the paths I've specified.

-rr10 makes it store some additional recovery data into the archive.

-t makes it test the integrity of the archive when all is said and done.

-m3 makes it compress at "Normal" level. The -m value could range from 0 (store: fast but no size reduction) to 5 (max: slow but maximum size reduction.)

-msrar tells RAR to ignore the compression level for files with the .rar extension, since these files do not benefit from another compression pass. I decided to allow t to compress other semi-compressed files (e.g. .zip and .jpg) because the could benefit from a RAR pass, and this is a one time expense since these files will not be compressed the next time around unless they had changed. If I wanted to omit .jpg and .zip in addition to .rar, I could have specified -msrar;zip;jpg Of course these files will STILL be stored into the archive.

The next argument is the location and name where my compressed archive will be stored, and all subsequent arguments are names of directories and files which I want to store into the archive.

Results:

It takes a little while to run this the first time. It took about half an hour to compress 6000 files (2.48 gigs) into the RAR file (2.2 gigs). Most of the time was spent compressing JPGs which probably yielded very little real savings in space. Also, quite a bit of my data is already compressed as RAR files so RAR didn't attempt any further compression. It's up to you whether you want to try compressing or using -m0 for store. The savings in space you get depend heavily on the kind of data you're backing up.

The next time you run this command, RAR spends quite a bit of time saying Updating archive e:\MIRROR of STUFF for BACKUP\DriveDbackup.rar while it scans for any differences between what you want to back up and what you already have. If there are absolutely no changes, it will say WARNING: No files and exit.

If there ARE any files that have changed or were added, RAR will say Adding and list the names of the files as it finds them. Afterwards, there's about 5 minutes of testing, as RAR checks not only the files it has changed/added, but the entire archive. If everything went OK, it will print All OK. Done and exit.