Monday, December 23


Daily News Stuff 23 December 2019

Christmas Eve Eve Edition

Tech News

Video of the Day

I recently - as in, six months ago - mentioned in passing that Hedetniemi's Conjecture had been proven false.  Here's all the tasty deets, as the kids say.

Disclaimer: There was a merry passenger,
A messenger, a mariner.
He built a gilded gondola
To wander in, and had in her
A load of golden oranges
And porridge for his provender.

Posted by: Pixy Misa at 11:34 PM | Comments (1) | Add Comment | Trackbacks (Suck)
Post contains 356 words, total size 4 kb.


Dirty Deeds Done Dirt Cheap

So, this is what I've done.  Virtual server with 24GB memory for $24 per month...  So long as you don't want to use all 24GB at once.

top - 05:43:22 up 2:06, 3 users, load average: 0.13, 0.05, 0.04
Tasks: 403 total, 1 running, 359 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.7 us, 0.7 sy, 0.0 ni, 98.1 id, 0.3 wa, 0.0 hi, 0.2 si, 0.0 st
KiB Mem : 4039484 total, 106752 free, 2530504 used, 1402228 buff/cache
KiB Swap: 20971512 total, 20000504 free, 971008 used. 1252856 avail Mem

It seems to be working pretty well.  I have multiple copies of MySQL running, plus MongoDB, Elasticsearch, Redis, and RabbitMQ, and while it has pushed stuff out to swap, it's not actively swapping, because no single test environment needs more than 4GB.  And it swaps back in pretty fast when needed.

Now I just need to make sure that all the nonsense I've done since yesterday hasn't wrecked the awesome performance I was getting.

Update: Seems to be just as fast as before - over 100 messages per second with a single worker, with all application features enabled, plus ZFS compression.  Read requests in the short-form test suite are slightly slower (up from 6.6ms to 7.7ms on average), but that might be because the PyPy jit hadn't fully optimised that code path.  Doing a full test run now.

Update: Yep, that gave me an average read request time of 6.6ms over the full run.  That's pulling the latest 20 messages from your feed along with all associated information.  With no caching - I have a caching module but it's disabled for now to make sure the core engine is fast.

Also took a look at the ZFS stats.  Both my deduplication ratio and my compression ratio are running at around 2:1, so for the 12.9GB used in my test environments so far, I'm using 3.56GB of actual disk space.

And that's when TokuDB writes its data in compressed format already.  ZFS helps by compressing the unused sections of the TokuDB files, because it allocates 2MB at a time by default.  Since every table and index on TokuDB is a separate file, that can add up.

So in short: The new VPS works great, it runs my test suite close to three times faster than before (which is very welcome given how many times I expect to run it over the next three weeks), ZFS works great, TokuDB works great, LXD works great except for the networking stuff, and even that is manageable now that I know the /etc/hosts trick, and it's all about $20 per month cheaper than the old server.

Posted by: Pixy Misa at 03:44 PM | Comments (1) | Add Comment | Trackbacks (Suck)
Post contains 445 words, total size 3 kb.

Sunday, December 22


Daily News Stuff 22 December 2019

Allegedly Edition

Tech News

Music Video of the Day

The film Johnny Dangerously came out 35 years ago today.

Disclaimer: You can tell I'm a living legend, not some ordinary creep.

Posted by: Pixy Misa at 10:14 PM | Comments (3) | Add Comment | Trackbacks (Suck)
Post contains 382 words, total size 4 kb.


The Vultrs Are Circling

I wanted to try out the new system on something closer to a deployment environment, but didn't feel like committing to another dedicated server.  Then I noticed that Vultr has rolled out their "high frequency" nodes worldwide - 3.8GHz CPUs and NVMe storage, starting at $6 per month.

I spun up a $24 instance (about 4¢ per hour) and loaded everything up, and set it to running the test suite.

And it immediately turned in performance 3x faster than the dev environment, which does run at 2.6GHz, yes, but has twice the cores and twice the RAM.

Turns out it's that NVMe drive that makes all the difference.  Write latency is typically between 200 and 300 Âµs on my dev server, but it's consistently 80 Âµs on the Vultr node.

Given that performance and the low pricing, this looks like a good option for production.

I'll try configuring one with ZFS and see how that goes; the default install is EXT4.  Since Vultr provide free full-server snapshot backups and fairly generous storage (the standard $20 server has 80GB of space; the $24 high frequency server has 128GB) ZFS isn't critical if it proves irksome.

(ZFS is brilliant for the dev environment though. because spinning up a whole new virtual server uses about 3MB of disk space, so long as it's the same version of Linux as an existing virtual server.  ZFS can both compress files automatically and merge identical files.)

Update: Thumbs up for Vultr.  Installing a custom ISO couldn't be easier; they even have a list of ready-to-go installers that do a standard setup that will work with your Vultr VPS but let you customise your filesystems.

Update 2: So I ended up spending the afternoon moving my dev environments over to Vultr, because I'm getting around twice the CPU performance and three times the I/O performance.  It will probably work out somewhat cheaper too, since I can make do with a smaller server. 

I hope.  It's configured with 4GB RAM and 16GB of swap, because there's a lot of stuff that gets used maybe once a month, and the idea is it will get swapped out to that nice fast NVMe storage and then swapped back in when it's needed.  If I do need to go for the same memory size as before it will end up costing more, but will also be faster and have more disk space.

Posted by: Pixy Misa at 09:54 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 405 words, total size 3 kb.

Saturday, December 21


Daily News Stuff 21 December 2019

Singing Chipmunk Edition

Tech News

  • If you put the names of your LXD containers in your hosts file or internal DNS, and match them up to LXD-managed IPs, LXD will actually do the right thing and assign those IPs, rather than its astoundingly unhelpful default behaviour of pulling an IP address out of its ass each time.

    If I'd known that a year ago it would have saved me, well, two or three hours, to be honest.

  • It really does look like the Ryzen 4700U will have eight cores.  (Tom's Hardware)

    The problem is that it's a laptop part and supposed to run at 15W.  That's not a lot of power to run eight cores and 832 graphics shaders.

  • The Intel 665p is a better version of the 660p, but is it better enough to fend off cheap DRAMless TLC NVMe SSDs?  (Tom's Hardware)

    The 665p is hampered by having only 2 PCIe lanes, meaning it can never break the 2GB per second barrier.  On the plus side, the 665p improves direct write performance (once the cache is full) by 50% over its predecessor.

  • Nitter is an alternative front-end to Twitter.  (GitHub)

    It acts as a proxy server - your browser talks to your Nitter node, and Nitter talks to the Twitter API.

    It uses no JavaScript.  Not on the server, and not in the browser.  That's pretty impressive.

    And it's written in Nim, the Python-like compiled language I've mentioned a couple of times when talking about Crystal.

  • Why Nitter?  Perhaps because the modern web is becoming an unusable, user-hostile wasteland.  (Abid Omar)

    Twitter is far from the worst culprit in that respect.  Well, the UI and API anyway, the content is another question.

  • Design your web pages to last.  (Jeff Huang)

    A good list:

    1. Use vanilla HTML and CSS.  If you need JavaScript, use it; if you don't need it, don't.

    2. Don't minimise.  Your page will get compressed automatically by the web server, so minimisation makes the files unreadable while only saving a few percent.

    3. Just have one great big page.  Uh.  Not so sure about that one.

    4. Don't hotlink.  That includes sites like CDNJS, convenient as they are.  At least make sure you have a copy of anything you load from another source.

    5. Don't use fancy webfonts.  Or if you do, put them on your own server.

    6. Minify your SVG.  Wait, what?  Use WebP, an image format that works with about 20% of apps on a good day.  Um, no.  

      Rule 6: There is no rule 6.

      But don't embed huge PNG files right in the page.

    7. Use a free monitoring service to tell you when your hosting provider has disappeared and your site is unrecoverable.

      So you can spend your weekend trying to extract it from archive.org.

  • Things done fast.  (Patrick Collison)

    And things done not fast.  San Francisco has been building a bus lane for 20 years.  The Alaska Highway took less than eight months.

  • Apple is working on top secret satellites.  (Bloomberg)

    I think people will work out what's going on when they launch the things into space on a huge fucking rocket.

  • I missed this one before: Data for 1.2 BILLION people was left in an unsecured Elasticsearch server connected directly to the internet.  (WCCFTech)

    Elasticsearch, until recently, had no security at all in the open source version.  I mean, sure, MongoDB installs by default without a password configured, but it does nag you to set a password.

    Elasticsearch didn't even have passwords in the free version.

  • Now data collected from million Facebook accounts by a third party has been found in another unsecured Elasticsearch server.  (WCCFTech)

    To be clear: This is not a server operated by Facebook.  Someone got their hands on the data somehow, put it in Elasticsearch to do, well, whatever, and left it connected to the internet with no password because until recently the free version of Elasticsearch didn't have passwords.

  • San Francisco accidentally banned smartphones.  (TechDirt)

    First worthwhile thing they've done since they burned the place down.

  • The Standard Model may be broken.  (Ars Technica)

    Have they tried turning the Universe off then on again?

Music Video of the Day

Disclaimer: We built this city on sausage rolls.

Posted by: Pixy Misa at 07:56 PM | Comments (5) | Add Comment | Trackbacks (Suck)
Post contains 705 words, total size 7 kb.


Daily News Stuff 20 December 2019

Working For The Man Edition

Tech News


Video of the Day

If you look into a mirror at midnight and say "Intel Core i9-10980XE" three times...

Bonus Video of the Day

Music Video of the Day

Disclaimer: I'm gonna be the man.

Posted by: Pixy Misa at 02:48 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 604 words, total size 6 kb.

Thursday, December 19


Daily News Stuff 19 December 2019

Christmas Hamper Full Of Poo Edition

Tech News

  • Broadcom is shipping the industry's first 25.6Tbps Ethernet switch.  (Tom's Hardware)

    That's 64 ports of 400GbE, so it's perfect for your next home networking project assuming that your home is the Large Hadron Collider.

  • Intel has released the i3-9350KF.  (Tom's Hardware)

    Four cores, four threads, no integrated graphics, buy AMD.

  • You can't index JSON arrays under TokuDB, only InnoDB.  Poot.

    Overall, though, TokuDB is a pretty clear winner for this application.  The database size is - hang on, let me measure it...

    For bulk-inserting 250,000 sample randomly-generated comments:

    InnoDB  168M  37.4 seconds
    MyISAM  115M  25.1 seconds
    TokuDB   79M  12.8 seconds

    InnoDB provides better transaction throughput than TokuDB on a high-end server, but TokuDB seems to be substantially better on low-end and virtual servers.

    It lacks some features (array and full-text indexes) but the benefits are too good to ignore.  And full-text indexes on InnoDB require a metric shit ton of system resources and are just not viable for my use case.

  • FISA judge slaps FBI.  (TechDirt)

    Not directly tech-related, but illustrates why I track TechDirt.  Yes, they're all dirty rotten liberals, but they are principled dirty rotten liberals.

  • Vivaldi now pretends to be Chrome on certain sites, because certain sites are retarded.  (Bleeping Computer)

    Shopify, for example.

Disclaimer: 42C here at PixyLab today and my air conditioner is dead.  I tried turning it on again despite the earlier flooding, but it would have none of it.  Blaargh.

Posted by: Pixy Misa at 11:47 PM | Comments (1) | Add Comment | Trackbacks (Suck)
Post contains 251 words, total size 3 kb.

Wednesday, December 18


Daily News Stuff 18 December 2019

A Cat May Look At A King Edition

Tech News

  • The secret word of the day is DELAY_KEY_WRITE.

    If you create a full-text index in MySQL and start shoving data into it, MySQL will try to flush the new index entries to disk as fast as it can.  This can easily produce a situation where posting a single comment writes a megabyte of data to disk.

    DELAY_KEY_WRITE says don't do that.  The new index entries sit in the key buffer (in RAM) until they need to be written out to make room for something else.

    Downside is that if the server crashes you need to run an index rebuild.

    Upside is that - coupled with moving transactional data from the InnoDB engine to TokuDB - it reduced the I/O load by 95%.  Full-text search went from being a pain in the bum to manage, to being a pain in the bum to manage if and when the server happens to crash.

    I can live with that.  Need to decouple things a little so that if the search index gets corrupted everything except search will keep running while it rebuilds, but that will take maybe three or four hours.

  • ADATA has 3GHz 32GB SO-DIMMs for all your 3GHz 32GB SO-DIMM needs.  (AnandTech)

    I want to try these in my Dell desktops, which currently only have 16GB.  I would upgrade, except that I have five computers with 16GB each so I just run things on different computers when I start to run out of RAM.

    Yes, it would have been a lot cheaper to buy more RAM.  I didn't plan to do it this way, it just sort of happened.

  • The new Mac Pro is actually repairable.  (iFixit)

    I mean, it should be - it's just an expensive PC in an expensive PC case.  But this is 2019, and it's Apple we're talking about, so it comes as something of a surprise.

  • That's not a whale.  (NewsBTC)

    Chinese scammers apparently stole - and then sold all at once - $185 million worth of Bitcoin.

  • The idiots at Microsoft are putting ads in their apps again.  (MS Power User)

    You can submit this as a suggestion, Microsoft.

  • Ghibli movies are now in the iTunes store.  (9to5Mac)

    I could wish they weren't copy-protected to Hell and back, but at least they are available.

Disclaimer: It might be very much like a whale, but it's not a whale.

Posted by: Pixy Misa at 09:34 PM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 409 words, total size 3 kb.


Daily News Stuff 17 December 2019

Water Feature Edition

Tech News

  • If you have LXD running on a ZFS storage pool, you can adjust the storage parameters for each container individually.  So if you suspect that compression is slowing down your database container, you can just do a quick

        zfs set compress=off my_server/containers/db_container

    and future writes will not be compressed and existing compressed pages will be written uncompressed if and when they get updated.

    ZFS is one of my favourite things.

    It seems to have worked: I'm currently posting (and full-text-indexing) 80 messages per second on my test server (four core / 2.6 GHz / 8 GB RAM) with  less than 40% CPU usage.

  • Samsung has announced two new mid-range phones: The Galaxy A51 and A71.  (AnandTech)

    They are very similar devices, with the A51 having a quad Cortex A73 processor and a 6.5" screen, and the A71 having a dual Cortex A76 processor and a 6.7" screen.

    2400x1080 display, 6 GB RAM, 128 GB storage, microSD, headphone jack, 32 MP front camera, 12 MP ultrawide camera, 5 MP depth camera, and 5 MP macro camera on both.  The A51 has a 48 MP main camera and the A71 has a 64 MP.

    The A71 is also 7 grams heavier.

    A51 is around €310, if you live in Poland or Vietnam.

  • You can plug an iPad Pro into Apple's new Pro Display.  (WCCFTech)

    Even though the Pro Display is a Thunderbolt device and the iPad Pro lacks Thunderbolt, it actually works seamlessly.

  • A high-end Threadripper deserves a high-end Threadripper motherboard.  (Serve the Home)

    Sure, the 3970X is a $2000 chip, and the Asus ROG Zenith II Extreme is an $850 board, but they handily beat the Intel Xeon W-3275 which costs $4500 by itself.

    Also worth noting that it uses about 100W less than the last-generation  2990WX while delivering much better (and much more consistent) performance.

    And this isn't even the fastest Thirdripper.  That comes next month.

  • A new startup has raised $100 million to build a demonstration fusion reactor.  (The Telegraph)

    I looked around and everyone has exactly the same story, but someone at The Telegraph at least bothered to look up the company and add a quick paragraph on what makes this different from a Tokamak.

    General Fusion has a quick overview paper on their design.

    They've been at this for ten years already.  Fusion is easy, but practical, controlled fusion that produces net energy is not easy at all.

Disclaimer: Twitter is over capacity.  Please wait a few moments and then go somewhere else.

Posted by: Pixy Misa at 01:03 AM | Comments (4) | Add Comment | Trackbacks (Suck)
Post contains 427 words, total size 4 kb.

Monday, December 16


Daily News Stuff 16 December 2019

Dirty Creature Edition

Tech News

  • Lexar has a new PCIe 4.0 M.2 SSD on the way that should be able to hit 7GB per second.  (Tom's Hardware)

    At least on reads; 4GB per second on writes.

    Current PCIe 4.0 SSDs all use the same controller and are limited to only 5GB per second.

  • ICANN has put the .org TLD sale on hold until all the fuss dies down.  (ICANN)

    I mean, they don't explicitly say that, but...

  • Full-text search indexes and ACID databases don't mix: Adding a full-text index to your InnoDB table can dramatically slow down transactions.

    But if, rather than indexing the raw fields, you have a specific search content field and populate that once a second, you can speed things up by a couple of orders of magnitude.

    And because data is only flushed to disk at the end of the transaction, it achieves this performance while also drastically reducing disk I/O - important if you're on a cloud server and don't have direct access to a fast SSD.

    Having an explicit search field has a couple of other advantages: You can add computed values, so that if your post is tagged "news" you can search specifically for tag:news without having to fuss about with multiple indexes.  (Elasticsearch does this automatically.) 

    And if you only need to keep the last, say, 90 days of comments searchable you can just null that field after 90 days.  The comments work just as before, but disappear from search.  (Although MySQL also allows you do do search queries without an index if you really want to.)

    The big advantage of having search inside your relational database is that you can use search and relations at the same time - that is, you can search only posts by people or blogs you follow, without needing a huge custom search infrastructure to manage it.

  • Speaking of MySQL, if something is unspeakably slow and it's not a full-text index, it's probably a subquery.  Got rid of one of those as well today.

  • Vim 8.2 has killer sheep.  (GitHub)

    Someone will now point out that EMACS had killer sheep in 1982.

Video of the Day

Got up this morning to find that my air conditioner had had enough of the 100% humidity and had flooded my kitchen.  Not with nice clean water either, oh no.  Had this running through my head all day as a result. 

Disclaimer: Your kitchen floor should probably not be a big black lake.

Posted by: Pixy Misa at 11:46 PM | Comments (4) | Add Comment | Trackbacks (Suck)
Post contains 419 words, total size 3 kb.

<< Page 2 of 4 >>
101kb generated in CPU 0.0648, elapsed 1.1335 seconds.
58 queries taking 1.1153 seconds, 367 records returned.
Powered by Minx 1.1.6c-pink.
Using http / http://ai.mee.nu / 365