Well, That Was Horrible

I had Midori - the mu.nu server; there's only the one now - scheduled for a ram upgrade Saturday at 9 PM Central.  They actually tried to do it on Thursday, but fortunately stopped before they could screw anything up.

So, I'm sitting there, waiting for the upgrade to start, while frantically running backups just in case anything goes wrong and trying to get two million files copied to Aoi before they pull the plug.  And the maintenance window arrives...  And passes...  And my backups finish.  And I post a query as to whether anyone is home at Softlayer.

And they respond (after an hour) and the upgrade is rescheduled for the next maintenance window, 2 AM - 4 AM Sunday.  And a little before 3 AM, they shut down the server and upgrade the memory.

And lo, the server is inaccessible.

So my first, second and third responses are to panic, for not only is this currently the only server for mu.nu, it is currently the only DNS server for mee.nu, so the whole shebang is pretty much off the air.  Okay, it's 3 o'clock on Sunday morning and no-one cares much right now, but I've got to get it fixed, and fast. 

Which might not be easy, because #1 suspect is a failed system disk.  The user directories are on a different disk, and backups are on yet another disk, but if I can't log in they don't do me a lot of good.

Now, among the goodies Softlayer provides me with is KVM and text console access.  Unfortunately, this is via Supermicro's IPMI tool, and it sucks your choice of equine anatomy.  Most notably, the K part of the KVM fuction suffers from severe arrhythmia.  When you press a key, you get, more or less at random, between zero and thirty copies of that keystroke.

Oh, and the text console doesn't work at all.

It's something of a miracle that I was even able to log in, and more of a miracle that I was able to quickly work out what the problem was: noexec was set for the /usr filesystem.  I'd meant to set it for /var/tmp, and missed, and hadn't rebooted since.

Oops.

At precisely this moment, for extra points, the KVM widgie decided to invert my capslock setting, making it an interesting experience to navigate vi.  But I did, and I fixed fstab, and I saved, and I rebooted, and apache didn't start but I have an easy shortcut command for that because it's something I need to do every now and then and lo we were back again.

And I've spent the last seven hours on a five-minute memory upgrade.

On a Sunday.

Bleh.

Posted by: Pixy Misa at 07:41 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 455 words, total size 3 kb.