You're Amelia!
You're late!
Amelia Pond! You're the little girl!
I'm Amelia, and you're late.

Friday, May 18


Glitch In The Matrix

Sorry, the backup script froze the database for a few minutes.  It's not supposed to do that, but it did.

Posted by: Pixy Misa at 09:13 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 24 words, total size 1 kb.

Saturday, May 05


Zoom Zoom

2000 users registered in 21.079s, avg 10.5ms
20000 friends added in 19.551s, avg 1.0ms
20000 idiots muted in 15.948s, avg 0.8ms
2000 lists added in 3.497s, avg 1.7ms
20000 members added to 2000 lists in 54.352s, avg 2.7ms
2000 channels added in 4.420s, avg 2.2ms
2000 mixes added in 9.542s, avg 4.8ms
2000 messages posted in 15.774s, avg 7.9ms
2000 stacks added in 3.227s, avg 1.6ms
20000 slots added to 2000 stacks in 63.906s, avg 3.2ms
[more messages posted...]
39867 messages viewed in 2000 requests in 17.310s, avg request 8.7ms 49595 bytes 5873 bytes gz


Doing some refactoring, and running the test suite repeatedly to check that nothing has blown up.  (Which it did earlier this evening.)  Performance is on a single-core Vultr VPS with 2GB RAM.  Production is a 6-core/12-thread Xeon 2620 v2 with 64GB RAM, so throughput will be quite a bit faster.

That's without any caching, too.

Posted by: Pixy Misa at 09:38 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 638 words, total size 5 kb.

Friday, May 04


The Return Of The Database

Sorry, got impatient while doing the changes and pushed the server too hard broke things.  Only broke them a little bit, but once it's broken at all it takes 45 minutes to check the indexes and bring the database back online.

There's something not quite right with the storage on these KVM servers I'm using.  I'm probably going to switch back to bare metal servers for the databases.  VPSes are fine for apps and files, but databases need consistency.

Posted by: Pixy Misa at 12:55 AM | Comments (1) | Add Comment | Trackbacks (Suck)
Post contains 84 words, total size 1 kb.

Thursday, May 03


The Return Of Recent Comments And Search Thingy

Um, they're back.

Posted by: Pixy Misa at 10:48 PM | Comments (6) | Add Comment | Trackbacks (Suck)
Post contains 11 words, total size 1 kb.


Pest Toast

Poast poast poast...

Posted by: Pixy Misa at 10:21 PM | Comments (3) | Add Comment | Trackbacks (Suck)
Post contains 5 words, total size 1 kb.

Tuesday, May 01


New Toy: Arale

It's a Xeon E5-2620 v2 with 64GB of RAM and a RAID-6 array of SSDs.

It's very cheap because it's a 2013 system - not quite 5 years old - and they're small SSDs.  But it will do nicely to replace a couple of our other servers that are running with single SSDs.

It will be the main server for the new platform.  I was worried until the weekend that I'd need more CPU power than that, but the 5x performance gains I got from two days of intense optimisation mean that this little server will run like the wind.  In Kansas.  In May.

The 64GB RAM is nice too; the other server I got recently only has 16GB, which is adequate but not spacious.  I had been considering Digital Ocean because of the ease of use, but it would cost nearly 5x as much if I wanted that much RAM in one place.


And I have that stupid boot loader problem with this one too.  Digital Ocean does not suffer from boot loader problems.

Update: Well, that was weird.  Five-year-old hardware but a two-year-old operating system won't boot.  But a week-old operating system does.

Well, Ubuntu 18.04 here we go, I guess.  While fiddling around I turned it into a RAID-1 for boot and a RAID-5 for storage.  I'll probably keep that.


Posted by: Pixy Misa at 10:43 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 227 words, total size 2 kb.

Sunday, April 29


It's a PyPy MySQL DB Client Linux Benchmark Query Battle!

Been working on our new platform this weekend.  

There's a query at the core of everything that pulls together the posts and comments from whatever you're looking at (a blog, your timeline, a forum thread) along with your interactions (likes, follows, bookmarks, votes) and attached data (replies, shares, and so on).  It has four joins and twelve subqueries plus a variable number of parameters, but at least three.

It was taking 25 milliseconds to run.

Well, actually it was taking 45 seconds to run, but I fixed that pretty quickly.  

Sometimes the best way to solve a subtle problem is to turn it into a huge problem with a clear solution.  So if your database query is a little slow, you add more data until it's really slow and you can measure differences as you vary the parameters, and if you can't find where that strange smell is coming from, you burn your house down.

So after fixing that big problem, I was left with a 25 millisecond query. That's not terrible, but what I really wanted was a 5 millisecond query.  Fiddling with the query on a small dataset didn't tell me anything, so I benchmarked the posting API overnight and added another 20 million records.

That found it for me, and I got the query down to about 10 milliseconds.  That's a big improvement, and enough to make it viable for production.  Most importantly, it moved the workload from about 50:50 between database and application to 1:4 database to application - a single database server could keep four application servers busy.  Or if I deploy on DigitalOcean, one $40 database server could provide for 16 $5 application servers.

Then I took a closer look at the application to find out where the time was spent (it's safe to assume MySQL itself is pretty well optimised at this point).  I checked out my ORM* but that was only taking 0.4 milliseconds on that 25 (now 10) millisecond query.

Some more poking found that a lot of time was being spent inside the MySQL client library.  But that's written in C, so there's not much I can do to speed it up, is there?

As a test, I ran my benchmark under CPython (interpreted) instead of PyPy (compiled).

It was twice as fast.

The interpreter was twice as fast as the compiler.

Why?  Because CPython has a much easier time binding to C libraries than does PyPy.  The PyPy compiler does a lot of internal optimisations that mean it needs a translation layer for older-style C libraries.  And that translation layer isn't very fast.

There's an alternative database library written in Python.  Under the Python interpreter that's a really bad idea - C is much much faster - but under the PyPy compiler it's not so silly.

So I swapped out mysqlclient and imported PyMySQL and my performance instantly doubled.

So the heart of the platform is running five times faster now than it was yesterday.**

There is still a scaling problems that will crop up as the database grows, but (a) I'm testing with 20 million posts right now and it's fine so far and (b) I know how to solve that one.

I'm doing some more testing with a skewed dataset, where 1% of the channels (a channel is a blog or forum or whatever) have 50% of the posts, but right now it's looking good.

So I can spend a couple of hours now trying to fix our existing platform, which I've been neglecting for three months now.

Update: Oops.  Ran out of disk space.  Stupid search index.

I also tested out the brand new Ubuntu 18.04, but the outcome there was that it needs another three months in the oven before I'd use it for production.

* I wrote a little database abstraction layer called Mirai, because the usual libraries like SQLAlchemy do a lot more than I need and are rather slow.  I wanted something simple and very fast.  It's really more of a ROM than an ORM, since it translates relational query data to nested object structures, but inserts are mostly manual.

** And 10,000 times faster than the day before, but never mind that...

Posted by: Pixy Misa at 11:50 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 711 words, total size 5 kb.

Friday, March 09


Glitch In The Matrix

We had a server hiccup just now; something happened to the SSD and it got pinned at 100% utilisation for several minutes.  I had to flush some things, stop a backup, and restart the database to get it back to normal.

You will likely need to log back in.

Posted by: Pixy Misa at 11:59 AM | Comments (3) | Add Comment | Trackbacks (Suck)
Post contains 53 words, total size 1 kb.

Wednesday, February 21



Getting away from politics to happier subjects for a moment, AMD have released their desktop Ryzen APUs - that is, processors with embedded graphics.  The first batch of Ryzen chips (like the R7 1700 I'm using now) had up to 8 cores but no graphics; these have only 4 cores but use the other half of the space to provide the best integrated graphics of any PC CPU ever.*

There are two models, the $99 R3 2200G, and the $169 R5 2400G.  The R5 is up to 30% faster on multi-threaded workloads and up to 50% faster on graphics, but there's a big factor of "it depends".  If you only have 4 active threads the R3 will perform within about 5% of the R5, and depending on the game you want to play, memory speed might be a more significant factor in frame rates than the APU itself.

The question really is, though, can I play games on this thing without having to buy a discrete graphics card which currently thanks to goddamn Blipcoin miners range upwards of eleventy billion dollars?

And the answer seems to be, hell yes.

You're not going to be playing at 4k 60Hz, but GTA5 is quite playable even on the $99 model, with stock cooler, slow memory, at 1080p and with quality settings mostly on high.  Some other games need you to turn the quality or resolution settings down a little to retain good frame rates, but it's still running at an integer multiple of Intel's integrated graphics.

* The best integrated graphics ever belong to the Xbox One X, the chip for which is also made by AMD.  Second place goes to Sony's Playstation 4 Pro, which uses a chip made by...  AMD.

Posted by: Pixy Misa at 04:53 PM | Comments (3) | Add Comment | Trackbacks (Suck)
Post contains 291 words, total size 2 kb.

Thursday, January 04


The Reboot Heard Round The World

There is apparently a security flaw in every recent Intel microprocessor, and by "recent" I mean "since November 1995".

There's a patch, but it's messy, and slows software down by 10%-30%, and requires the server to rebooted.

When you ask "which server", the answer is "all the server".  Alllll the server.

This weekend.

Expect some things to be offline.

Posted by: Pixy Misa at 10:40 AM | Comments (10) | Add Comment | Trackbacks (Suck)
Post contains 65 words, total size 1 kb.

<< Page 1 of 131 >>
69kb generated in CPU 0.07, elapsed 0.1974 seconds.
56 queries taking 0.1585 seconds, 300 records returned.
Powered by Minx 1.1.6c-pink.