Backups: A Cautionary Tale
Posted on Tuesday, August 07, 2007
I’m really embarassed about what I’m about to write. Seriously. My ego is bruised. My competence challenged. My sanity in question. What could do this to poor me, you ask? A dead hard drive and a backup system gone wrong in the most stupid way imaginable.
This morning I woke up to find my Macbook’s fan revving, the infamous rainbow cursing spinning and the hard drive…clicking. Bad news, right? No big deal, I thought, because I’ve got a good backup system in place. Let me tell you about it:
Each night at 4 a.m. a backup script I wrote (using rsync) mounts an encrypted disk image off my “server” (my trusty old ibook + an external hard drive), runs rsync to sync the disk image to my hard drive and then unmounts. It all works automatically with no intervention from me. Very slick. I was proud of myself.
It wasn’t always automatic though: only about two months ago did I automate the process by configuring cron to run the backup script every night. This was amazingly easy…with one glitch: I needed the script to run as root and the password for the encrypted disk image was stored in my personal keychain, so root wasn’t finding it. Now there is a command line way to extract a password from a user’s keychain, but I didn’t know this at the time.* Since I needed cron to be able to mount the image on its own, the easiest solution was to just type the password into the backup script in plain text. However, I’d made the original disk image password the same as my regular user (and admin) account password. I wasn’t too crazy about just storing that in clear text right there on my hard drive.** So I just generated a new password and put it right in the script.*** And I use good passwords, ok? So it’s not my cat’s name or anything.
You can probably guess what happened already. In my haste to finish up the script, I forgot to save the new password. Well, I did save it…on my Macbook hard drive, with the intent to write it down or save a copy elsewhere. It sat there in clear text in my backup script for two months. At any point during this time, I could’ve done something about this. Oh, I have a backup of the script…inside the encrypted image. I even have an unencrypted backup of the script that includes everything except the password: I’d saved a copy for James a few weeks back (who was putting together his own home backup system…read and learn, James!) though, of course, I cut the password out of the copy I sent him.
So now I’ve got a perfect image of my Macbook hard drive trapped inside a 50GB disk image that I cannot open. And to twist the knife a bit more: I was aware of this problem. It had occured to me once while I was out for a run a few weeks ago, but I completely forgot to ever do anything about it.
There is a bit of a silver lining. About two months ago, almost on a whim, I backed up my home area to the hard drive of my work computer. It had plenty of spare space because it has a 250 GB hard drive that could get away with only being 20 GB b/c I save all my data on the server at work (there’s a lesson in there, too…). So, thankfully, I haven’t lost everything, just the last two months worth of work, which — and at no point in the last two months did I think I’d be happy to say this — hasn’t amounted to much. I’ve been a complete slacker the last few months and now it’s paid off because I would’ve lost it all anyway.
I also called Apple and they’re going to replace the hard drive. (Well, I had to call twice…the first guy was a total idiot, so I just hung up and called back and got someone both polite and competent who immediately began filling out paperwork for a repair…note: if you’re buying an Apple laptop, get Apple Care, and keep calling if they don’t help you at first.) The hard drive had a particularly nasty death: I tried Disk Utility off the Install DVD, Firewire Disk Mode with another Mac, a Linux Live CD and a USB-SATA connector thingy and nothing would even let me see the drive, let alone let me recover any data from it. Apparently this is a relatively common death for Macbook hard drives, which doesn’t exactly discourage my temptation to buy an evil Dell and switch to Linux.
So do you see why I’m so distraught? Understand why I’m so embarassed and angry at myself? If I’d done nothing about backing up my data — which is what most people do, I guess — I would be angry and disappointed right now. However, the fact that I went through the process of developing a well thought-out backup plan and then screwed up one critical detail that made it all for nothing….that really stings.
* In case you’re wondering, I believe this will store the value of your password in the variable $password:
password=`security 2>&1 >/dev/null \ find-generic-password -ga test | grep password | \ sed -e "s/password: \"//" -e "s/\"//"`
James shared this with me weeks ago, yet I did not take the time to implement it…
** This illustrates an interesting dilemma about data protection: by increasing security on one front (password security) you also decrease security on another (human stupidity and forgetfullness). Another example: I had enough room on my external hard drive for either a single, full nightly image of my laptop’s drive (best in the case of hardware failure) or incremental, archived backups of all the important folders (best in case of human error: overwriting files and not realizing it for a day or so). Care to guess whether or not I’ve ever overwritten files that could’ve been recovered a few days later if I’d opted for the latter scenario? (Or simply coughed up the cash for a bigger drive and done both?)
*** My logic for thinking this was a better choice was that the password was for a disk image stored on my home server, so it would'nt really help anyone who stole my laptop, unless they happened to steal it in my house and also figure out the password to my server. Of course, the best choice would’ve been to use the keychain tip above, or just write down the damn password and put it in a safe place.