Dear Santa, thank you for the dolls and pencils and the fish. It's Easter now, so I hope I didn't wake you but... honest, it is an emergency. There's a crack in my wall. Aunt Sharon says it's just an ordinary crack, but I know its not cause at night there's voices so... please please can you send someone to fix it? Or a policeman, or...
Back in a moment.
Thank you Santa.

Whosoever

Finds a way to harness the power of bored computer programmers shall rule the world!

Posted by: Pixy Misa at 10:01 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 16 words, total size 1 kb.

Damn

Still having issues with memcached.  I might have to disable it for a while.

It just seems to stop responding, but apparently only on the new server.  We have way more than enough CPU power for now, so disabling it won't even be noticeable.

Update: Added a global switch to the cache, and set it to off.

Posted by: Pixy Misa at 02:52 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 58 words, total size 1 kb.

AAAAAAARGH!

An extraneous nohup in a startup script.

That was it.

That's what was behind the bug I've been tracing for seven hours.

I need a drink.

...

Okay, so while I was crawling around in the logfiles I was able to find and fix a couple of intermittent bugs that have been plaguing us for a while.  Including a nasty one where the session management code would, under certain circumstances, delete the database cursor for a thread and so ensure that the subsequent page request would drop dead. (The error management code would reinstate the cursor on the next request, but that particular page would fail.  It happened to me just now, editing this post.)

But still.  Seven hours for a nohup.

Posted by: Pixy Misa at 12:54 AM | Comments (5) | Add Comment | Trackbacks (Suck)
Post contains 123 words, total size 1 kb.

Got It!

Finally tracked down what's going on on the production web servers.

The system automatically flushes the cache when you update a page, so people don't get old copies of the page.  That's timing out - it eventually happens, but it takes more than ten seconds, so the reverse proxy times out and returns a "System not available" error.  Since the cache flush is handled by the redirect module (which all update pages have to call), you never receive the redirect because the proxy closes the connection first.

It was particularly confusing because (a) the updates actually worked and (b) because the error was inside the redirect module and redirect logging was at the start of the redirect module, the log files showed that the redirect was happening but no browser request was following it.

And the reason for that is that (a) I didn't have a local cache running and (b) the remote cache is blocked by the firewall.  It seems that cache lookups time out immediately, but cache flushes take much longer.

Which should have been obvious.

Posted by: Pixy Misa at 12:32 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 180 words, total size 1 kb.

Encouraging Signs

Just finishing the preparations for the launch of mee.nu.

I loaded a copy of the current beta database onto the production cluster.  By the time I could switch screens and bring up phpMyAdmin, it had already finished replicating.

Posted by: Pixy Misa at 02:44 AM | Comments (3) | Add Comment | Trackbacks (Suck)
Post contains 40 words, total size 1 kb.

No Joy In Mudville

The server that ate all of mee.nu's bandwidth just corrupted its own operating system and died.

Emails have been sent, you betcha.

Update: Evil server is no more; they've done a complete system swap.  Now I just have to reinstall everything.

Posted by: Pixy Misa at 04:24 PM | Comments (4) | Add Comment | Trackbacks (Suck)
Post contains 45 words, total size 1 kb.

Bugger

I just discovered that the two mee.nu production servers, which I'm about to put to work, had a little glitch in their reverse proxies and somehow managed to exchange a terabyte or so of wasted packets in the past few days.

Over our hosting company's public network.

And since it was a terabyte in each direction, and they bill for both incoming and outgoing data, that means I just used up the entire bandwidth quota for mee.nu for the month.*

Bugger.

Fortunately, Softlayer's excess bandwidth charges are among the lowest in the business, but still... Bugger.

Filed a support ticket throwing myself on the mercy of the accounting department.  You never know.

* Excluding the beta server, but I need to take that offline, get the drive replaced, and reinstall the operating system.

Posted by: Pixy Misa at 05:04 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 134 words, total size 1 kb.

Slashzonk

I took "Zonk" off my list of editors over at Slashdot because, let's face it, he's an idiot.

The front page didn't update for the next 24 hours.  Because none of the other dozen of so editors posted anything.

So I re-enabled Zonk, read the page, and then took him off the list again.  Gah.

I expect Slashdot's readers to be about 60% idiots, but I can ignore them, mock them, or (when I have mod points) put the boot in.  Can't do much to Zonk when he's the only editor who's actually doing anything.

Posted by: Pixy Misa at 01:49 PM | Comments (4) | Add Comment | Trackbacks (Suck)
Post contains 96 words, total size 1 kb.

Aargh! (Part 97 Of A Continuing Series)

Spend two hours debugging a 25-line SQL query only to find out that it is syntactically correct, but that one of the parameters is being passed as a string instead of an int, which causes the interface library to quote it.

The information is there in the error message, but it wouldn't hurt for it to be a little less... delphic:

You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''30'' at line 1.

The program generates the query as a single line; it only became 25 lines after I broke it down for readability.

Blahs.

Posted by: Pixy Misa at 01:02 AM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 117 words, total size 1 kb.