Saturday, November 14
Scale Away Edition
- Fired up the Mana test environment this morning - an E3-1270 v6 with 32GB of RAM and two 256GB SSDs. Been working on performance since around 7AM. I never get up that early on the weekend, but there were back-to-back Hololive En Minecraft streams.
So far the biggest improvement (apart from the fact that the server itself is faster) is confirming that timeline queries are now efficient and I don't need to maintain default stacks anymore. That cut the time for a scaling test run - creating ten thousand users and a million messages, then reading them back via various combinations of API methods and parameters - from four hours to two. Still single-threaded; the API is fine with multi-threading but the test suite doesn't do it yet.
I have a third run of the scaling test about 80% complete now, with the default stacks disabled, and some new InnoDB and ZFS tuning. So far it's still trucking along at around 120 messages per second, which is five times faster than the dev environment.
I can get a server with twice the everything of this test system - so eight cores, 64GB RAM, and 1TB+ of NVMe storage - for under $100 per month. That should handle things nicely.
- One thing I still need to check is the load generated by the search indexes. Going over the MariaDB documentation while things were running and I was waiting for the next Minecraft stream, I was reminded that they support the Federated storage engine - or rather, the rewritten version called FederatedX.
That would let me put the search indexes on their own server should the need arise, with no code changes at all. And in theory other tables as well, but search is the top candidate for this because it's an isolated function - if search breaks, everything else keeps on running until you get around to fixing it.
(Unless developers write their apps to use search all the time instead of the more specific and efficient API methods designed for them. Not that that ever happens.)
- No, you idiots, Apple's M1 GPU is not faster than an RX580. (Tom's Hardware_
- X-NAND is NAND only X. (Tom's Hardware)
It seems to be QLC flash with pseudo-SLC caching, but with a lot more internal parallelism than typical flash dies, so you can read or write an order of magnitude faster. In terms of throughput; latency is comparable to existing chips.
- Target said "sorry, we fucked up" and unbanned the book they banned yesterday.
- Latest scaling run just completed. Only shaved another 15 minutes off the run time - about 12% - but it cut the write amplification issue drastically. That turned out to be a clash between the InnoDB and ZFS block sizes.
(I did set that up correctly previously, but then forgot about it because with TokuDB it makes no difference and works just fine at the default settings.)
Database size for this run was 3.7GB before compression, 2.1GB compressed.
I'm going to re-run it now with GZip compression instead of LZ4. Percona recommend this tradeoff if you have a fast CPU, not so much because it saves space, as because it improves performance by reducing I/O.
The nice thing with ZFS is that you can change compression algorithms and block sizes on the fly. The changes only take effect on new block writes, but it works just fine with a random mix of different sizes and compression methods.
Making the database and filesystem block sizes match already reduced I/O utilisation by 60%, so let's see what GZip can do to help.
Update: GZip is the way to go. Only shaved 6 minutes - 5% - off the previous run, but it cut the database size nearly 40% from 2.1GB to 1.3GB. Faster and smaller is good.
These times and sizes are all with temporal tables enabled as well. That's been painless so far.
- I've applied the same tweaks to my test environment now as well, and we'll see how that goes. If I can do the beta launch on a $24 per month VPS, with the ability to scale it up as needed, all the better.
Update: Short test run down from 17 minutes to 11. Long test run is, well, running. With these changes, performance looks like it will in fact be fine starting out on a $24 VPS.
Update: Yes, full test suite is down from 633 minutes to 123 minutes on the dev server. Which is running all sorts of other crap apart from Mana.
That is more than fast enough; that's over 10 million messages a day. Well, until it hits the next performance cliff and dives over it.
Update: Oops, that run was without the search index. So 10 million messages per day if the search index is on another server. Running it again with search active.
Terraria Roller Coaster Video of the Day
Everything is going well, she takes out her first boss, then the ctrl-click calamity strikes.
And then the storm clears away and the Sun comes out again.
Walking through my local Target store recently allows for the observation of a lot of books of questionable content, not the least of which masquerading as 'anti-racist' by being racist.
Posted by: cxt217 at Sunday, November 15 2020 08:28 AM (4i7w0)
Posted by: Phil Fraering at Sunday, November 15 2020 11:20 AM (Clxcy)
Also a slightly *larger* font would be nice.
And maybe updating the HTML so that whatever enables reading mode in modern browsers works.
Posted by: Rick C at Sunday, November 15 2020 02:17 PM (eqaFC)
I'll see if I can also add some options for default background and text colours. On some monitors black text on a white background is perfect, but on others a full screen of white is blinding.
Posted by: Pixy Misa at Sunday, November 15 2020 04:40 PM (PiXy!)
I'd send money, but I'm slightly underemployed at the moment.
Posted by: Phil Fraering at Tuesday, November 17 2020 01:54 AM (Clxcy)
58 queries taking 0.1376 seconds, 328 records returned.
Powered by Minx 1.1.6c-pink.