WOULD YOU CARE FOR SOME TEA?

I Think I Just Dodged A Tactical Nuke

Almost.

Sakura, the third server in the mu.nu cluster, used for offsite backups and for offloading some sites during the election overload, just dropped dead.

Now the two main servers, Aoi and Midori, do daily backups and then exchange backup files.  I'd intended to apply that to Sakura as well, but Sakura went live about the time I went insane, and it never happened.

The first sign of trouble was earlier today, when Protein Wisdom started running slow, and then stopped entirely.  I logged in and poked around a bit, but couldn't see any problem.  Restarted Apache.  Ran a check on the database.  All fine.

Took a look at the system log.  Reporting 6 bad sectors on the disk.  Okay, that's not great, but not the end of the world.  I'll take a backup, copy it offsite, and get them to replace the disk.

Start the backup to the second disk, and everything's fine for a few minutes, and then it's nothing but I/O errors.  Uh-oh.  The system disk has gone offline, taking with it the databases and the operating system.

I'm still logged into the server, mind you.  But I've lost /bin, /usr/bin, everything bin.  What still works?  Well, let's see.  I have cd.  I have echo, because that's built into the shell.  And I have cat.

And that's about it.

And using cd, echo, and cat, I managed to rescue the main Protein Wisdom database from a disk that I couldn't access, because the entire thing was cached in RAM.

The reason why I only almost dodged the nuke is that it came up with a bunch of I/O errors, and I thought it hadn't worked, so I had tech support reboot the server.

Mistake.  The server wouldn't boot from the drive, so they had to replace it.  They put it in an external caddy, but it still wouldn't recognise it.  So all I had was a corrupted backup of one database.

But I was able - thanks to myisamchk and repair table and a spare copy of the Wordpress database structure - to rebuild that database and merge it into the last off-site backup.

If I hadn't rebooted at that point, I might have been able to do the same for the other databases.

And then I got a call from my business partner.  He had a site - a business site - hosted on that server as well.  And I'd just lost all his data...  Except not, I realised, after my blood pressure had hit 180/140.  He's running a hosted app that was linked from the site he has on that server, so all he needed was for me to restore the static content of his site, which was backed up safely.  Twice, in fact.

The third site on there wasn't so lucky, and lost a couple of month's data.  Fortunately, it's a less busy site, and a couple of hours spent trawling Google's caches let me dig out most of the missing posts; the site owner will just need to cut-and-paste those and re-load the associated media files.

I just spent five solid days - as in, 120 consecutive hours - panicking about the servers at my day job.  And the minute I get that fixed (and in the end, a lot better than it ever was before) and everyone is happy, boom!

And now I'm dead.  I'm so dead my deads have dead on them.  I hadn't even been grocery shopping for two weeks, until I hauled myself up the hill to the shops just now at 8:30 on a Sunday night.

I need some Popotan.

Posted by: Pixy Misa at 08:15 PM | Comments (10) | Add Comment | Trackbacks (Suck)
Post contains 610 words, total size 4 kb.

Impeach Obama

They bumped House for an address from the Campaigner-in-Chief?

Posted by: Pixy Misa at 08:19 PM | Comments (6) | Add Comment | Trackbacks (Suck)
Post contains 11 words, total size 1 kb.

P-p-p-parsers

The Ping module has been coded and backported to 1.1, and is in testing.

Trackbacks are fairly simple in principle, until you get to autodiscovery.

With autodiscovery, I have to -

• Parse any markup in the post (BBCode, Textile, Markdown, Creole, reStructuredText*) to generate pure HTML
• Parse the HTML to extract links
• Parse the links to distinguish links to media and files from links to web pages
• Parse the web pages to find the RDF trackback data (which is likely to be in a comment, so an HTML parser won't even see it)
• Parse the RDF data for the trackback URL
• Actually send the ping
• Parse the XML response

Bleh.

* Yes, there's lots of goodies coming up.

Posted by: Pixy Misa at 02:05 PM | Comments (3) | Add Comment | Trackbacks (Suck)
Post contains 112 words, total size 1 kb.

1.2-alpha-c1

The new edition of Minx is working!

Okay, it's still in internal alpha testing, but after I ripped out its internals to provide for some of the major new features (scripting language, API support, safe third-party applets) it's been kind of broken for a while.

Now it's working fine, and seems to be at least close to the speed of 1.1.  I can optimise the common cases of my new tag dictionary class, which is very heavily used; that should help things along.

Hmm.

Before:

108kb generated in CPU 0.13, elapsed 0.1315 seconds.
65 queries taking 0.0324 seconds, 276 records returned.
Powered by Minx 1.1.2-beta.

After:

106kb generated in CPU 0.12, elapsed 0.1285 seconds.
70 queries taking 0.0426 seconds, 279 records returned.
Powered by Minx 1.2-alpha-c1.

But I just noticed that recent comments aren't working on 1.2.*

Based on the work I've been doing at my day job over the past year, I do have some neat ways to potentially speed up the query side of things (and reduce the number of queries).  But that's only one third the total time, so Amdahl's Law applies in spades.

* Doh!  It's that blasted cross-referencing indexy thing I implemented.  Efficient, yeah, but if the cross-reference table is empty it don't work.  And if the cross-reference table isn't accurate, it don't work right.  I'll have to find - or rewrite - the code to rebuild the index, but at least I know it's not a bug.  As such.

Posted by: Pixy Misa at 11:08 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 244 words, total size 2 kb.

Pig Bums

Broke my glasses.

Amazingly, I managed to find both my spare pairs despite my optically-challenged condition.  Now looking forward to raging headache kicking in about 4PM.

Posted by: Pixy Misa at 09:06 AM | Comments (10) | Add Comment | Trackbacks (Suck)
Post contains 28 words, total size 1 kb.

Avoiding Work



Making omelettes.

Posted by: Pixy Misa at 01:03 PM | Comments (12) | Add Comment | Trackbacks (Suck)
Post contains 4 words, total size 1 kb.