Sometimes we assume that the software we use is infallible. It works most of the time, so we just think that it'll always work.
Unfortunately, that's not always the case. Even if the software is 100% perfect, there are still things that can cause us problems. Disk drives are physical machines, and as with anything physical they wear out over time. Eventually, they break. And when that happens, you may find yourself in the same situation I was in yesterday: a broken Subversion repository
I did an svnadmin verify and it would return errors all over the place (in a different place each time, too). Eventually, I had to concede that the repository was un-recoverable.
Luckily for me, I do weekly backups and my backup from last week was still usable. My backup strategy is actually very simple: hot-backup.py every sunday night. Technically, I probably don't have to use hot-backup.py, since it's unlikely that I'm going to be doing anything at 4am on a Sunday morning, but you never know :)
So I bought a new hard drive and kicked off the restore process. I don't have a huge repository (~900 revisions [and counting!] and about 900MB.) - mostly the size comes from the fact that I've got copies of a few open source projects in there... I don't know why; I should probably tidy up my repository one of these days...
Anyway, the restore process is really simple: just bunzip the file that hot-backup.py created last Sunday. It took a few minutes (a bit longer actually, cause I took the opportunity to upgrade to Subversion 1.6.11 while I was at it).
The most annoying thing was that after it was all complete, my repository was complete only up until last Sunday: I had to take my current working directory and manually apply all the changes that I had made since then to get it back to what it was. Isn't there a corollary to Murphey's Law that states that when the worst thing that could happen happens, it happens at the worst possible time? In my case, it was because my drive failed on a Sunday: just hours before the next backup would've occurred and I had a full 7 days worth of changes to manually reapply.
So it's always a good thing when a hardware failure shows that all your obsessive backing up is actually worthwhile, but even with the best plans in place, there's always a lesson to learn. In my case, I've decided to augment my weekly backups with some daily incremental backups. I've borrowed this script and set it up so that it backs up incrementally each night as well.
I think the next step would be to do some off-site backups. I've just got my backups happening to a different drive in the same computer (I was lucky, I guess, that only one drive failed). I really need to get my backups saved somewhere a little more isolated. But maybe I'll wait until the next failure precipitates it...