The Ur-Guru pointed me to this sad post on Refreshing News:

A Calgary man is desperate to get his stolen laptop, with years of work on it, back.

It was stolen from the trunk of his car while he went for a run in Edworthy Park Wednesday night.

John Boldt is pleading for the return of his hard drive, which contains research and notes for the thesis he was writing for his master’s degree in history. He only had three chapters left to write before his paper was complete.

Unless he gets it back, Boldt will have to abandon his dream and quit at the University of Calgary.

“It’s so many years of my life just thrown away,” he says. […] Boldt says if he doesn’t get the data back, he won’t be able to return to the U of C when classes start next week.

I feel for John Boldt. My car was broken into in February, and it’s a terrible experience even when nothing of value is taken. Even if the laptop had been in plain sight on the passenger seat and not in the trunk, that wouldn’t have given anyone the right to steal it.

But how does someone get into, and nearly through, a master’s program without learning to back up his work, especially when it’s so important to him?

Back when I was in graduate school, in the late Eighties and early Nineties, I had all my papers, and then the draft chapters of my dissertation, on floppy disks. This was partly because I didn’t own the computers I worked on, and partly because computers didn’t have much internal storage. But there were always two sets of disks, the main set and the backup set. Always. I can’t even remember who taught me to do that, but the lesson was an old one.

Heck, even when I was an undergraduate using the mainframe as a glorified typewriter, they told us we could get a tape with our files on it. (I never did, but I can’t imagine what computer I would ever have been able to use it with, anyway. It would be easier for me to run my undergraduate honors thesis through a scanner and OCR it.)

Ah—according to Gawker, there was a backup drive in the trunk along with the laptop, and that got taken, too. This additional info makes Boldt look a bit more clueful. Some people do habitually store their backup drives in the car, in order not to have them in the office where the desktop computer is. I’m not sure that would be the best option if you’re a laptop person carrying your computer around a lot, but at least it suggests he was doing something to protect his work.

Many of the commenters on the Gawker post are protesting that the situation seems fishy to them, pointing out numerous ways that the chapters of the thesis could have been backed up and asking why Boldt’s advisor doesn’t have a copy of it. It’s a good question: I certainly gave chapters of my dissertation to my advisor as I wrote them, though I think I had to deliver them in hard copy. Having a printout would be a lot better than not having anything.

Others point out that Boldt had plenty of opportunity to use an online backup service like Mozy or Carbonite or even a tool like Dropbox. A history thesis would almost certainly be a Word doc, probably using a template provided by the university or the department. It might be large as Word docs go if it had illustrations in it, but it would be unlikely to be larger than the 2 GB limit for free accounts at Mozy.

But, as I suspected, Boldt would have had no need to resort to a commercial service if he wanted online backup. The University of Calgary offers all students a service called Webdisk File Storage:

Webdisk is a remote file storage system that allows you to store your files on a IT server making them easily accessible from any computer with an Internet connection. […] Webdisk files are automatically backed up every day, so they can be recovered if they are inadvertently deleted or changed.

So where was Boldt when Webdisk accounts were being handed out?

It’s easy to get sloppy about backups, to say you’ll put another system in place tomorrow, to forget to re-establish levels of protection when you get a new machine. But when you have really critical data, you can’t back it up too many ways or in too many places, and you can’t do it too soon.

And that will be true even if John Boldt turns out to be just as much a fiction as Whiteboard Jenny.

I was listening to Marketing Over Coffee a few weeks ago and heard a sad tale of woe from co-host John Wall. His blog, Ronin Marketeer, was down for four days. Hosed, in fact.

Literally.

On February 20th, there was an an accident during the annual inspection of the fire-suppression system at WestHost’s data center. If you rewrite WestHost’s account of the incident in the active voice, it amounts to “The vendor forgot to remove an actuator before the inspection, and this triggered a release of Inergen all over the data center.”

According to Wikipedia, Inergen is non-toxic…to humans. It is, alas, highly poisonous to computer servers. And John’s blog just happened to be on one of the worst-affected machines.

Bye-bye blog.

The good news is, WestHost actually had backups of its clients’ sites. Not all hosting companies actually back up your website for you. (You can usually make your own backups through the control panel, but you may or may not be able to automate this process. Of course, if you have a traditional HTML site and edit the files on your computer before uploading them, you should be able to back up your local versions easily.)

The better news (theoretically) was that John, being a smart guy with lots of IT experience, had recently made his own backup of his blog. That meant he was starting out in better shape than Jeff Atwood over at Coding Horror, who had to rely on other people to piece together bits of his lost blog for him.

But the first attempt to restore Ronin Marketeer left a bit to be desired. When I sent a sympathetic inquiry to John after hearing the podcast, he sent me a link to a post he had titled “When Even a Backup Is Not Enough.”

As you can see, everything is all f’d up here.

Over a week ago disaster struck at my hosting company, during a fire alarm test the suppression system was triggered, hosing all the servers. This blog was dead for a full week.

We were offered to move our hosting from the version 3 infrastructure to v4, and I took up the offer since it got my domain back 2 days earlier. Unfortunately the new environment is not the same – even though I have a full backup of my Database that supports this blog, the new system does not allow you access to the directory where that data is kept.

I’m no expert in MySQL, but it looks like I’ve gone from having my own instance to sharing one on the server with everyone else.

The end result is that all my archives are gone for now and my Google juice vanishing as there’s no access to any of my archives. It looks like my only path is to install WP and MySQL on a box of my own, then do a WordPress export so I can then import it back in. I cannot believe that having the actual files is not enough for me to do a restore.

“My god,” you may be thinking. “If having the backup is no good, why bother making one?”

But if he hadn’t had the backup, the story would not have had a happy ending, and it does. John had to do some heroically geeky things, but he was able to get the blog back up and running. He did lose some comments, probably due to the nature of the restore process, but everything else seems to be intact. John started Ronin Marketeer in November 2006, and he’s a pretty prolific blogger. It would have been a serious loss, and no fun to try to reconstruct from the Google cache and the Wayback Machine.

I’m betting John will be especially interested in the WordPress backup plugin I’m going to be writing about next week. Everyone else certainly seems to be, and I’m very impressed so far.

Yesterday the Ur-Guru pointed me to a post on Coding Horror entitled “International Backup Awareness Day.” Coding Horror is normally a blog, and the permalinks to posts don’t normally look like the one I just pasted in there. Depending on when you read this, clicking on that link might get you a “404: Page Not Found” error. Thanks to catastrophic data loss, Coding Horror is only pretending to be a blog right now.

Here’s the story. Jeff Atwood, geek extraordinaire, hosted his blog on a virtual machine (VM) on a server at a web hosting company. VMs are great for developers, because you can simulate different operating systems in order to test your software on them, and you can take snapshots and re-create them easily.

Unlike Jeff Atwood or the Ur-Guru, I’ve never worked with a VM. To use them you need more RAM than I have at my disposal. I can see how it might be handy to have one to test backup software on without cluttering up the registry of my main machine, though. But apparently backing up the contents of a VM doesn’t work quite the same way as backing up your ordinary operating system and files. This is a setup for a very dangerous situation: when you think you have backups, but don’t. (Have I said “Test your backups” lately? Test your backups.) And this, it seems, is exactly how Jeff lost his blog:

1. The server experienced routine hard drive failure. (Ed. note: hard drive failure is described as “routine.” In data centers, where drives are spinning 24/7, that’s exactly what it is.)
2. Because of the hard drive failure, the virtual machine image hosting this blog was corrupted.
3. Because the blog was hosted in a virtual machine, the standard daily backup procedures at the host were unable to ever back it up.
4. Because I am an idiot, I didn’t have my own (recent) backups of Coding Horror. Man, I wish I had read some good blog entries on backup strategies!
5. Because there were no good backups, there was catastrophic data loss. Fin, draw curtain, exeunt stage left.

Now, I don’t know what blogging platform Jeff was using. Given that he’s one of those extreme geek types, it could be something really obscure, even something he created himself. (Power geeks are like that; they’re as likely to insist on developing their own tools as to use anyone else’s.) I don’t even know what operating system his VM was running. But I know that there were ways to back up this blog when I was using Blogger (published by FTP to my own website), and there’s no excuse for not backing up WordPress blogs, since there are handy plugins to make it easier. And any offline blog editor like Windows Live Writer or Ecto will save local copies of your posts, so you can back them up along with the rest of the data on your hard drive. (Back in the days before Windows Live Writer, I used to write my blog posts in Microsoft Word, but you don’t actually want to paste from Word into anything that uses HTML. It did, however, mean that I had local copies.)

Jeff was able to re-build the text portion of his blog in HTML thanks to a fellow extreme geek who’s been archiving the Internet, but lost many of the images (which are not, apparently, on his hard drive, or not readily identifiable from among those on his hard drive). I shudder to think just how much work this must have been—and how much more work it will be to convert it back into blog format if he chooses to do that.

The lessons Jeff Atwood learned from the demise of Coding Horror are not just for geeks.

What can we all learn from this sad turn of events?

1. I suck.
2. No, really, I suck.
3. Don’t rely on your host or anyone else to back up your important data. Do it yourself. If you aren’t personally responsible for your own backups, they are effectively not happening.
4. If something really bad happens to your data, how would you recover? What’s the process? What are the hard parts of recovery? I think in the back of my mind I had false confidence about Coding Horror recovery scenarios because I kept thinking of it as mostly text. Of course, the text turned out to be the easiest part. The images, which I had thought of as a “nice to have”, were more essential than I realized and far more difficult to recover. Some argue that we shouldn’t be talking about “backups”, but recovery.
5. It’s worth revisiting your recovery process periodically to make sure it’s still alive, kicking, and fully functional.
6. I’m awesome! No, just kidding. I suck.

So when, exactly, is International Backup Awareness Day? Today. Yesterday. This week. This month. This year. It’s a trick question. Every day is International Backup Awareness Day. And the sooner I figure that out, the better off I’ll be.

Have you checked your backups lately? Now might be a really good time.

Would you entrust your data to a company called “Danger”? Microsoft and T-Mobile did. And it was your data, if you were a Sidekick user.

The adventure began on October 10th. The headline in TechCrunch read “T-Mobile Sidekick Disaster: Danger’s Servers Crashed, And They Don’t Have A Backup.” Jason Kincaid, author of the TechCrunch article, was absolutely scathing on the subject:

This goes beyond FAIL, face-palm, or any of the other internet memes we’ve come to associate with incompetence. The fact that T-Mobile and/or Microsoft Danger don’t have a redundant backup is simply inexcusable, especially given the fact that the Sidekick is totally reliant on the cloud because it doesn’t store its data locally.

I’ve never used a Sidekick, but a mobile device that doesn’t store phone numbers, etc locally at all seems bizarre, and in fact I’m not sure that statement is quite accurate, given suggestions in other articles that if you keep the Sidekick charged and turned on, you would at least save anything in its current memory.

But then, I still have a “dumbphone,” so what do I know about how these things work?

By October 11th, T-Mobile had posted the following discouraging notice on its user forums:

Regrettably, based on Microsoft/Danger’s latest recovery assessment of their systems, we must now inform you that personal information stored on your device — such as contacts, calendar entries, to-do lists or photos — that is no longer on your Sidekick almost certainly has been lost as a result of a server failure at Microsoft/Danger.

Not surprisingly, the media has been all over the story. “Microsoft has said that the hardware failure that caused the problem took out both the primary and backup copies of the database that contained Sidekick users’ information,” Ina Fried wrote on October 12th. “But the question remains, why wasn’t there a true independent backup of the data?”

That would certainly be my question. Rafe Needleman, also writing for CNET on October 12th, concluded that you can’t trust the cloud because you can’t trust the people running it. The problem, in other words, is not one of technology. Tech support staff often refer to problems that start “between the keyboard and the chair.”

If it’s possible to create independent, redundant backups in your own data center, it’s possible to do it in the data centers used by cloud computing companies. The only difference  is that you can’t walk down the hall and see that they’ve done it. Some people will slack off when you aren’t there to hold them accountable, but that’s not true of everyone. As Lance Ulanoff concluded in his October 13th article, “Don’t Blame Cloud Computing for the T-Mobile Mess,”

Obviously, something went very, very wrong with T-Mobile and Microsoft’s Sidekick data set-up, but let’s not throw out the baby with the bathwater (or the cloud with the rainwater). The cloud isn’t the problem. Instead, I blame the people—as always.

But the Register, with typically British enthusiasm for a pun, declared “Danger Lurks in the Clouds” on October 18th. The danger is that all mobile devices will rely increasingly on a working connection to provide any functions at all. Nevertheless, author Bill Ray concludes:

Cloud-based servers are still more reliable than most of the kit knocking around users’ homes – the life expectancy of an Apple Time Capsule, for example, is just over 17 months according to the Time Capsule Memorial Register, so even those who are backing up locally shouldn’t be too smug.

That article concludes, in the Register’s usual tongue-in-cheek fashion, that paper is the only safe storage medium.

By October 20th, Microsoft and Danger had in fact been able to restore some of the data, as reported in CNET and on T-Mobile’s user forum. That’s a happier ending than Sidekick owners had been led to expect. I’m glad they got their data back, but if I’d been affected, I’d want more.

I’d want to know what the company was going to do differently from now on so that this wouldn’t happen again. And I’d want a free application that would let me back up all my contacts, calendar entries, etc, onto my computer. It wouldn’t even have to sync with Outlook or Google or Mac-whatever, as long as I’d be able to restore the data to my mobile device.

Finally, as an occasional naming consultant, I want to see Microsoft Danger rebranded. What incentive do you have to entrust something valuable to a company called Danger? What incentive do employees of a company called Danger have to be careful? Danger is a fun name for a company that makes games, but for data storage, it just sounds unreliable.

I woke up at 2 AM this morning. Don’t ask why, because I haven’t the faintest idea, but there I was. Since by 3:00 it was clear I wasn’t going to get back to sleep, I turned on the computers and attempted to be productive.

By about 10:30 AM I was nearly finished creating a mind map for the website redesign for a corporate client, based on their suggestions and another site they thought was a good example. I use MindJet’s MindManager Pro 6 software for this. The latest version of MindManager is 8, but I never bothered to upgrade. The program already has more features than I know how to use.

Anyway, I had opened an earlier map to check something that I wanted to add to the current map. I was finished with the earlier map and went to close it, and got prompted with one of those “Do you want to save changes?” dialogs. I hadn’t made any changes to that file, so I said “No.”

And then discovered that I’d just closed the whole program, and lost the last 10 minutes of work I’d done. Four hours of sleep, yeah. Does wonders for the brain function.

I save my work often, and I have most of my programs set to autosave at least every ten minutes. But it turns out I can make a lot of changes to a document in ten minutes, at least if it’s a mind map.

Now, I have Mind Manager set to automatically create .BAK files the way I have Word set to automatically create .WBK files. In Word, the .WBK files preserve an earlier version of the file; you can return to it if the current version gets corrupted somehow. I assumed the same idea held true with MindManager, but I couldn’t even get it to open the .BAK file, so I had no way to tell whether it had preserved my new changes.

I also checked Free Agent Sync on my F:\ drive, but that copy of the file, like the one on my C:\ drive, had not preserved the changes. So I had to go back and re-create all the work I’d just done. And in the process of doing so, I misspelled someone’s name, a thing I normally pride myself on not doing, and probably left a few other things out.

I’ve since gone back and set the autosave in MindManager (it’s under Tools | Options | Save) to 5 minutes. That might keep me out of trouble. But only if I get enough sleep.