What happened? Twelve years! You hit me with a cricket bat! Ha! Twelve years!
Tuesday, May 01
New Toy: Arale
It's a Xeon E5-2620 v2 with 64GB of RAM and a RAID-6 array of SSDs.
It's very cheap because it's a 2013 system - not quite 5 years old - and they're small SSDs. But it will do nicely to replace a couple of our other servers that are running with single SSDs.
It will be the main server for the new platform. I was worried until the weekend that I'd need more CPU power than that, but the 5x performance gains I got from two days of intense optimisation mean that this little server will run like the wind. In Kansas. In May.
The 64GB RAM is nice too; the other server I got recently only has 16GB, which is adequate but not spacious. I had been considering Digital Ocean because of the ease of use, but it would cost nearly 5x as much if I wanted that much RAM in one place.
...
And I have that stupid boot loader problem with this one too. Digital Ocean does not suffer from boot loader problems.
Update: Well, that was weird. Five-year-old hardware but a two-year-old operating system won't boot. But a week-old operating system does.
Well, Ubuntu 18.04 here we go, I guess. While fiddling around I turned it into a RAID-1 for boot and a RAID-5 for storage. I'll probably keep that.
Posted by: Pixy Misa at
10:43 AM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 227 words, total size 2 kb.
Sunday, April 29
It's a PyPy MySQL DB Client Linux Benchmark Query Battle!
Been working on our new platform this weekend.
There's a query at the core of everything that pulls together the posts and comments from whatever you're looking at (a blog, your timeline, a forum thread) along with your interactions (likes, follows, bookmarks, votes) and attached data (replies, shares, and so on). It has four joins and twelve subqueries plus a variable number of parameters, but at least three.
It was taking 25 milliseconds to run.
Well, actually it was taking 45 seconds to run, but I fixed that pretty quickly.
Sometimes the best way to solve a subtle problem is to turn it into a huge problem with a clear solution. So if your database query is a little slow, you add more data until it's really slow and you can measure differences as you vary the parameters, and if you can't find where that strange smell is coming from, you burn your house down.
So after fixing that big problem, I was left with a 25 millisecond query. That's not terrible, but what I really wanted was a 5 millisecond query. Fiddling with the query on a small dataset didn't tell me anything, so I benchmarked the posting API overnight and added another 20 million records.
That found it for me, and I got the query down to about 10 milliseconds. That's a big improvement, and enough to make it viable for production. Most importantly, it moved the workload from about 50:50 between database and application to 1:4 database to application - a single database server could keep four application servers busy. Or if I deploy on DigitalOcean, one $40 database server could provide for 16 $5 application servers.
Then I took a closer look at the application to find out where the time was spent (it's safe to assume MySQL itself is pretty well optimised at this point). I checked out my ORM* but that was only taking 0.4 milliseconds on that 25 (now 10) millisecond query.
Some more poking found that a lot of time was being spent inside the MySQL client library. But that's written in C, so there's not much I can do to speed it up, is there?
As a test, I ran my benchmark under CPython (interpreted) instead of PyPy (compiled).
It was twice as fast.
The interpreter was twice as fast as the compiler.
Why? Because CPython has a much easier time binding to C libraries than does PyPy. The PyPy compiler does a lot of internal optimisations that mean it needs a translation layer for older-style C libraries. And that translation layer isn't very fast.
There's an alternative database library written in Python. Under the Python interpreter that's a really bad idea - C is much much faster - but under the PyPy compiler it's not so silly.
So I swapped out mysqlclient and imported PyMySQL and my performance instantly doubled.
So the heart of the platform is running five times faster now than it was yesterday.**
There is still a scaling problems that will crop up as the database grows, but (a) I'm testing with 20 million posts right now and it's fine so far and (b) I know how to solve that one.
I'm doing some more testing with a skewed dataset, where 1% of the channels (a channel is a blog or forum or whatever) have 50% of the posts, but right now it's looking good.
So I can spend a couple of hours now trying to fix our existing platform, which I've been neglecting for three months now.
Update: Oops. Ran out of disk space. Stupid search index.
I also tested out the brand new Ubuntu 18.04, but the outcome there was that it needs another three months in the oven before I'd use it for production.
* I wrote a little database abstraction layer called Mirai, because the usual libraries like SQLAlchemy do a lot more than I need and are rather slow. I wanted something simple and very fast. It's really more of a ROM than an ORM, since it translates relational query data to nested object structures, but inserts are mostly manual.
** And 10,000 times faster than the day before, but never mind that...
Posted by: Pixy Misa at
11:50 AM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 711 words, total size 5 kb.
Tuesday, April 10
Hello Blog, How Are You?
Why yes, I did get suspended on Twitter. How did you know?
The mob got me. Was having a nice civil discussion blazing row over a California State Senate bill legislating Orwell's Ministry of Truth and I slipped upspoke my mind reported scientifically established fact and called one of the fascists retarded:
Two things to note: First, it's transparently obvious that I didn't break those rules. Second, those rules are retarded. Third, so is the goddamn California state senate:
TITLE 14.5. False Information Strategic Plans
3085.
(a) Any person who operates a social media Internet Web site with physical presence in California shall develop a strategic plan to verify news stories shared on its Internet Web site.
(b) The strategic plan shall include, but is not limited to, all of the following:
(1) A plan to mitigate the spread of false information through news stories.
(2) The utilization of fact-checkers to verify news stories.
(3) Providing outreach to social media users regarding news stories containing false information.
(4) Placing a warning on a news story containing false information.
(c) As used in this section, "social media†means an electronic service or account, or electronic content, including, but not limited to, videos, still photographs, blogs, video blogs, podcasts, instant and text messages, email, online services or accounts, or Internet Web site profiles or locations.
I don't know if I have ever seen a more blatant First Amendment violation, but these retards are perfectly fine with handing this power to the government even when Donald Trump - who is Literally Hitler - is in power.
Update: Twitter has now locked all my alternate accounts and won't even let me read tweets any more, though the message clearly says I can read tweets.
Update 2: I'm back, and I've saved a copy of all my tweets. Not that anyone would ever want them, but anyway...
Posted by: Pixy Misa at
03:56 AM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 317 words, total size 3 kb.
Saturday, March 31
Twupdate
From an earlier post:
Also, I'm on my 19th20th21st22nd23rd24th25th26th27th28th29th30th31st32nd33rd34th35th36th37th38th39th40th41st42nd43rd44th 45th suspension with Twitter right now.
The only reason it's not more is you can't get suspended when you're already suspended. Pretty sure I got double-suspended at one point.
Posted by: Pixy Misa at
12:18 PM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 62 words, total size 1 kb.
Monday, March 26
Steven Pinker Was Jordan B. Peterson Before It Was Cool
Posted by: Pixy Misa at
10:00 AM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 10 words, total size 1 kb.
Saturday, March 24
Where Was I?
At my day job we've done our San Francisco and New York launch events (and both went very well), and that just leaves London this week. This is the first weekend in a month when I haven't had to work.
I was doing something about a new social network, as I recall.
....
Nope, gone now. Well I'm sure if it was important I'll remember it.
Posted by: Pixy Misa at
12:26 PM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 69 words, total size 1 kb.
Thursday, March 22
They Nuke Horses, Don't They?
ESTABLISHING SHOT OF HORSES GALLOPING INTO THE DISTANCE
CUT TO
Zuck, standing in front of barn door, enormous brass padlock in hands: We've shut the door, and now we're locking it.
Lock: KA-CHUNK.
Zuck: We've put top men on the problem.
SLOW ZOOM OUT SHOWING MILES AND MILES OF BARN AND THOUSANDS OF DOORS
We had a server hiccup just now; something happened to the SSD and it got pinned at 100% utilisation for several minutes. I had to flush some things, stop a backup, and restart the database to get it back to normal.
You will likely need to log back in.
Posted by: Pixy Misa at
11:59 AM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 53 words, total size 1 kb.
