Two main problems:

1. We need a wildcard certificate, because the rate limits for free certificates would take decades to secure all the blogs.
2. Forms have to be explicitly HTTPS; implicit form actions are HTTP (unencrypted), and browsers don't permit HTTP form actions from an HTTPS page.

There were a couple of glitches while this was going on, but mostly it was confined to my blog.

Update: DNS changes have propagated, and I've set this blog to always use HTTPS.  All forms across all blogs should be updated in the system now to switch to HTTPS as it becomes available (this caused me the most pain).

Will continue testing on this blog before switching everyone else across.

Update 2: Bugger, got one of the DNS updates wrong.  Sites will load slowly for the next hour or so because they're waiting for a script to load from an IP that's blocked by our hosting provider (long story).  I could try to fix that, but it would take an hour to make and test the changes.  Some of the fancy features might not work while we wait for that DNS change to propagate.

Update 3: That's fixed now.

Update 4: Broke mee.im, which was depending on the old mee.nu DNS servers.  Fixed that too.

Posted by: Mauser at Sunday, May 27 2018 07:28 AM (Ix1l6)

Posted by: J Greely at Sunday, May 27 2018 10:47 AM (tgyIO)

Posted by: Pixy Misa at Sunday, May 27 2018 01:47 PM (PiXy!)

Posted by: Pixy Misa at Sunday, May 27 2018 02:18 PM (PiXy!)

Posted by: Pixy Misa at Sunday, May 27 2018 02:43 PM (PiXy!)

Posted by: Pixy Misa at Sunday, May 27 2018 02:43 PM (PiXy!)

Posted by: Pixy Misa at Sunday, May 27 2018 02:43 PM (PiXy!)

Posted by: Pixy Misa at Sunday, May 27 2018 02:43 PM (PiXy!)

Posted by: Pete Zaitcev at Sunday, June 03 2018 08:12 AM (LZ7Bg)

Posted by: Pete Zaitcev at Sunday, June 03 2018 08:14 AM (LZ7Bg)

Posted by: Pixy Misa at Monday, June 04 2018 12:39 PM (PiXy!)

Posted by: Pixy Misa at Monday, June 04 2018 09:36 PM (PiXy!)

Hide Comments | Add Comment

Post more.

Post post post post.....

Foof!

Posted by: Pixy Misa at Saturday, May 26 2018 07:10 PM (PiXy!)

Posted by: Pixy Misa at Saturday, May 26 2018 07:10 PM (PiXy!)

Posted by: Pixy Misa at Saturday, May 26 2018 08:48 PM (PiXy!)

Posted by: Pixy Misa at Saturday, May 26 2018 08:48 PM (PiXy!)

Posted by: Wonderduck at Friday, June 01 2018 03:11 PM (ojeh3)

Posted by: Pixy Misa at Monday, June 04 2018 12:38 PM (PiXy!)

Hide Comments | Add Comment

Posted by: Pixy Misa at Saturday, May 26 2018 03:23 PM (2yngH)

Posted by: Pixy Misa at Saturday, May 26 2018 03:23 PM (2yngH)

Posted by: Pixy Misa at Saturday, May 26 2018 04:47 PM (PiXy!)

Hide Comments | Add Comment

Answers should include a stamped, self-addressed envelope and twenty bucks.  Or just have at it in the comments.  Neatest correct entry wins.

 

Doing some refactoring, and running the test suite repeatedly to check that nothing has blown up.  (Which it did earlier this evening.)  Performance is on a single-core Vultr VPS with 2GB RAM.  Production is a 6-core/12-thread Xeon 2620 v2 with 64GB RAM, so throughput will be quite a bit faster.

That's without any caching, too.

There's something not quite right with the storage on these KVM servers I'm using.  I'm probably going to switch back to bare metal servers for the databases.  VPSes are fine for apps and files, but databases need consistency.

Posted by: SamathaDonnelly at Monday, May 28 2018 08:58 PM (E+TF1)

Hide Comments | Add Comment

It's very cheap because it's a 2013 system - not quite 5 years old - and they're small SSDs.  But it will do nicely to replace a couple of our other servers that are running with single SSDs.

It will be the main server for the new platform.  I was worried until the weekend that I'd need more CPU power than that, but the 5x performance gains I got from two days of intense optimisation mean that this little server will run like the wind.  In Kansas.  In May.

The 64GB RAM is nice too; the other server I got recently only has 16GB, which is adequate but not spacious.  I had been considering Digital Ocean because of the ease of use, but it would cost nearly 5x as much if I wanted that much RAM in one place.

...

And I have that stupid boot loader problem with this one too.  Digital Ocean does not suffer from boot loader problems.

Update: Well, that was weird.  Five-year-old hardware but a two-year-old operating system won't boot.  But a week-old operating system does.

Well, Ubuntu 18.04 here we go, I guess.  While fiddling around I turned it into a RAID-1 for boot and a RAID-5 for storage.  I'll probably keep that.

There's a query at the core of everything that pulls together the posts and comments from whatever you're looking at (a blog, your timeline, a forum thread) along with your interactions (likes, follows, bookmarks, votes) and attached data (replies, shares, and so on).  It has four joins and twelve subqueries plus a variable number of parameters, but at least three.

It was taking 25 milliseconds to run.

Well, actually it was taking 45 seconds to run, but I fixed that pretty quickly.  

Sometimes the best way to solve a subtle problem is to turn it into a huge problem with a clear solution.  So if your database query is a little slow, you add more data until it's really slow and you can measure differences as you vary the parameters, and if you can't find where that strange smell is coming from, you burn your house down.

So after fixing that big problem, I was left with a 25 millisecond query. That's not terrible, but what I really wanted was a 5 millisecond query.  Fiddling with the query on a small dataset didn't tell me anything, so I benchmarked the posting API overnight and added another 20 million records.

That found it for me, and I got the query down to about 10 milliseconds.  That's a big improvement, and enough to make it viable for production.  Most importantly, it moved the workload from about 50:50 between database and application to 1:4 database to application - a single database server could keep four application servers busy.  Or if I deploy on DigitalOcean, one $40 database server could provide for 16 $5 application servers.

Then I took a closer look at the application to find out where the time was spent (it's safe to assume MySQL itself is pretty well optimised at this point).  I checked out my ORM* but that was only taking 0.4 milliseconds on that 25 (now 10) millisecond query.

Some more poking found that a lot of time was being spent inside the MySQL client library.  But that's written in C, so there's not much I can do to speed it up, is there?

As a test, I ran my benchmark under CPython (interpreted) instead of PyPy (compiled).

It was twice as fast.

The interpreter was twice as fast as the compiler.

Why?  Because CPython has a much easier time binding to C libraries than does PyPy.  The PyPy compiler does a lot of internal optimisations that mean it needs a translation layer for older-style C libraries.  And that translation layer isn't very fast.

There's an alternative database library written in Python.  Under the Python interpreter that's a really bad idea - C is much much faster - but under the PyPy compiler it's not so silly.

So I swapped out mysqlclient and imported PyMySQL and my performance instantly doubled.

So the heart of the platform is running five times faster now than it was yesterday.**

There is still a scaling problems that will crop up as the database grows, but (a) I'm testing with 20 million posts right now and it's fine so far and (b) I know how to solve that one.

I'm doing some more testing with a skewed dataset, where 1% of the channels (a channel is a blog or forum or whatever) have 50% of the posts, but right now it's looking good.

So I can spend a couple of hours now trying to fix our existing platform, which I've been neglecting for three months now.

Update: Oops.  Ran out of disk space.  Stupid search index.

I also tested out the brand new Ubuntu 18.04, but the outcome there was that it needs another three months in the oven before I'd use it for production.