I Think I Just Dodged A Tactical Nuke

Almost.

Sakura, the third server in the mu.nu cluster, used for offsite backups and for offloading some sites during the election overload, just dropped dead.

Now the two main servers, Aoi and Midori, do daily backups and then exchange backup files.  I'd intended to apply that to Sakura as well, but Sakura went live about the time I went insane, and it never happened.

The first sign of trouble was earlier today, when Protein Wisdom started running slow, and then stopped entirely.  I logged in and poked around a bit, but couldn't see any problem.  Restarted Apache.  Ran a check on the database.  All fine.

Took a look at the system log.  Reporting 6 bad sectors on the disk.  Okay, that's not great, but not the end of the world.  I'll take a backup, copy it offsite, and get them to replace the disk.

Start the backup to the second disk, and everything's fine for a few minutes, and then it's nothing but I/O errors.  Uh-oh.  The system disk has gone offline, taking with it the databases and the operating system.

I'm still logged into the server, mind you.  But I've lost /bin, /usr/bin, everything bin.  What still works?  Well, let's see.  I have cd.  I have echo, because that's built into the shell.  And I have cat.

And that's about it.

And using cd, echo, and cat, I managed to rescue the main Protein Wisdom database from a disk that I couldn't access, because the entire thing was cached in RAM.

The reason why I only almost dodged the nuke is that it came up with a bunch of I/O errors, and I thought it hadn't worked, so I had tech support reboot the server.

Mistake.  The server wouldn't boot from the drive, so they had to replace it.  They put it in an external caddy, but it still wouldn't recognise it.  So all I had was a corrupted backup of one database.

But I was able - thanks to myisamchk and repair table and a spare copy of the Wordpress database structure - to rebuild that database and merge it into the last off-site backup.

If I hadn't rebooted at that point, I might have been able to do the same for the other databases.

And then I got a call from my business partner.  He had a site - a business site - hosted on that server as well.  And I'd just lost all his data...  Except not, I realised, after my blood pressure had hit 180/140.  He's running a hosted app that was linked from the site he has on that server, so all he needed was for me to restore the static content of his site, which was backed up safely.  Twice, in fact.

The third site on there wasn't so lucky, and lost a couple of month's data.  Fortunately, it's a less busy site, and a couple of hours spent trawling Google's caches let me dig out most of the missing posts; the site owner will just need to cut-and-paste those and re-load the associated media files.

I just spent five solid days - as in, 120 consecutive hours - panicking about the servers at my day job.  And the minute I get that fixed (and in the end, a lot better than it ever was before) and everyone is happy, boom!

And now I'm dead.  I'm so dead my deads have dead on them.  I hadn't even been grocery shopping for two weeks, until I hauled myself up the hill to the shops just now at 8:30 on a Sunday night.

I need some Popotan.

Posted by: Pixy Misa at 08:15 PM | Comments (10) | Add Comment | Trackbacks (Suck)
Post contains 610 words, total size 4 kb.