Saturday, December 23
So, What Happened There...
I was planning to do a server migration last weekend, but the new server died.
I was planning to do a server migration last weekend, but the new server died.
So it got pushed back until basically the last minute before the old server was set to get cancelled on Thursday - and then on Thursday Akismet, the spam filtering service provided by Wordpress, basically dropped dead. Don't know what's up with that, but it took out not only all commenting on mu.nu and mee.nu, but also the old server itself due to a huge backlog of requests (it was taking forever for Akismet to eventually time out).
This happened while I was migrating the server, and since everything was in flux I had no idea what, specifically, had gone wrong.
The remaining issues were due to an upgrade from MySQL 5.0 to 5.7, which by default runs in strict mode and didn't like some old, sloppy SQL queries and definitions; a couple of places where I'd specified canonical server names (whatever.mu.nu) instead of functional names (bloghost.mu.nu), and a proxy configuration error that meant the real IP address wasn't getting passed through to Minx.
And that meant, because Akismet is rejecting all my API requests, that every commenter was treated as having the same IP address, which was banned.
Splat.
Working on the recent comments and search functions now. I had to disable recent comments because MySQL 5.7 can't properly evaluate the query (it takes a minute and a half for something that used to take milliseconds). I'm not sure yet what happened to search.
On a related note, the MySQL query optimiser doesn't seem to improve over time. They just move the bugs around.
Update: I think Akismet didn't actually go down, they just changed something that we were depending on. They probably sent me an email six months ago...
Posted by: Pixy Misa at
07:19 PM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 300 words, total size 2 kb.
48kb generated in CPU 0.0636, elapsed 0.281 seconds.
56 queries taking 0.2676 seconds, 338 records returned.
Powered by Minx 1.1.6c-pink.
56 queries taking 0.2676 seconds, 338 records returned.
Powered by Minx 1.1.6c-pink.
