• Was testing deployment options for the new system today.  DigitalOcean running out of a RAM disk was about the same speed as Vultr running from SSD.
  
  Although....  That test is very write-heavy, and unless I start getting a million posts a day right away I'll be far more concerned with reads.  Vultr is still ahead there but not by so much that DO isn't an option.
  
  The advantage of going with DigitalOcean is that (a) they pool bandwidth, so if the database server comes with 4TB (which it would) that gets added to the pool to be used by the web servers, and (b) they have block and object storage in every location.
  
  Advantage of Vultr is that it's faster and it's easy to do a custom install running ZFS.  With DO I can easily add block storage and configure that with ZFS, but that is even slower than the standard storage.
  
  Third option is RamNode, who I haven't used before, but who have been around a while and are well-regarded.  They don't have object or block storage, but they do have fast NVMe servers like Vultr, and cheap disk-based storage servers with tons of available bandwidth.  (As low as $1 per TB transferred.)
  
  All three are viable so I'll just pick one and go with it.
  
  
• AMD's Lisa Su may be presenting Zen 3 at CES.  (WCCFTech)
  
  And/or the Ryzen 4000 APUs, which may or may not deliver 8 cores to mainstream laptops.
  
  Or possibly something else.
  
  
  
  
  
• Speaking of which, AMD is set to become TSMC's largest 7nm customer.  (Tom's Hardware)
  
  Only partly because Apple will be moving to 5nm later this year, and AMD isn't planning on that until 2021.
  
  
• Isaac Asimov's century.  (The Humanist)
  
  Today is the 100th anniversary of Isaac Asimov's birth, more or less (there aren't exact records).  TechCrunch reminded me of the fact, so thanks for that, but their article will just make you irritated so I won't link it here.
  
  Science Magazine also has a respectful article.
  
  
• Literature, films, and music created in 1924 have now entered the public domain.  (Hyperallergic)
  
  In the US; in many other countries they have been in the public domain for years.
  
  
• Your programming language is bad and you should feel bad.
  
  Mind you, this list would reject Python specifically, and pretty much everything other than Lisp for other reasons.
  
  
• Apple is suing a security company under the DMCA for creating an iOS security tool.  (iFixit)
  
  This is section 1201 of the DMCA, the anti-circumvention section, a.k.a the bad part.  Corellium - the company in question - allows you to create virtual iPhones in your browser to test security issues.  Apple is not happy about that and wants them dead.
  
  
• SAFe is an unholy incarnation of darkness.  (Medium has not yet blocked me this month)
  
  SAFe is Agile for Enterprise.  Agile is a lightweight team-driven software development methodology for smaller projects, so it is exactly what you don't want at enterprise scale.  Scroll down to the diagram in the linked article where they explain it all.
  
  
• A look at the Threadripper 3960X.  (Serve the Home)
  
  AMD's smaller Thirdripper is still impressive.  Intel's fastest workstation processor - costing twice as much - can beat it on, oh, on Passmark, and a couple of others, but in almost every case Intel's best is coming in third, and sometimes fourth.  And that's even before AMD releases their high-end parts.
  
  
• Python 2.7 has reached EOL.  (Bleeping Computer)
  
  There is one final update due in April, and then that is it.
  
  Unless you are using PyPy, which is both written in and compatible with Python 2.  Since dropping support for Python 2 would require rewriting the entire thing, they are planning to support it indefinitely.
  
  Still, libraries will slowly drop support for Python 2.  Web3, the Ethereum library, doesn't work with Python 2 at all.
  
  RedHat will also continue supporting Python 2.7 through to at least 2024.
  
  
• MacOS 8 and the Y2.02K problem.  (Six Colors)
  
  In 1997?  Seriously, Apple, in 1997 you did this?
  
  Like every other MacOS problem there is a third-party utility to fix it.
  
  
• The LG G8X ThinQ Dual Screen solves the folding screen problem by...  Not folding the screen.  (ZDNet)
  
  It has two screens (both 6.4" 2340x1080 OLED displays) and a hinge in between.
  
  
• One quarter of all the pigs in the world died in 2019 of swine fever.  (New York Times)
  
  Which sounds shocking - and I suppose it is - but pigs raised for market (which is almost all of them) do not live very long lives in the first place.
  
  
• In an apparent attempt to make cancer great again, the FDA has banned the production and sale of many flavoured vape products.  (Engadget)
  
  Sigh.
  
  
• A federal judge has blocked...  Wait, what's this?  A federal judge has blocked California's terrible "gig economy" law.  (CBS News)
  
  Apparently only as it applies to truck drivers though.  How it will play out for other groups if they succeed in their lawsuit will be interesting to watch.
  
  I am not against laws protecting workers from being taken advantage of by unscrupulous companies.  I am just against this law, because it is dumb.
  
  
• LG will be launching eight new 8K TVs at CES next week.  (9to5Mac)
  
  The models range from 65" to 85".  Prices are not mentioned and are likely still stratospheric, but prices for 4K came down pretty quickly once the second-tier players got onto the market, and it's likely to be the same only more so with 8K.
  
  

Other News

The newly announced official song of the United States Space Force.

Posted by: Rick C at Saturday, January 04 2020 12:45 AM (Iwkd4)

Posted by: Wonderduck at Saturday, January 04 2020 11:53 AM (cTMj+)

Hide Comments | Add Comment

• Intel's upcoming Core i9-10900K is reportedly up to 30% faster than the 9900K on multi-threaded tasks.  (WCCFTech)
  
  Unfortunately for Intel, the Ryzen 3900X is already 50% faster than the 9900K on those tasks.  (CPUBenchmark.net)  
  
  
• It's either that or elemental fluorine: How to tarnish platinum: Sell it as a Xeon 9200.  (AnandTech)
  
  Ouch.
  
  
• Running Python in the Linux kernel.  (Medium)
  
  Or not.  I'll go with not.
  
  
• The Chrome extension Shitcoin Wallet much to everyone's surprise steals your Ethereum coins.  (ZDNet)
  
  I mean, they all but called it "Ethereum Coin Stealer".
  
  
• India is heading back to the Moon.  (Reuters)
  
  Their previous lunar orbiter worked smoothly, but their Moon rover suffered a hard landing, as in, it was completely obliterated.
  
  
• Well don't do that then.
  
  
  
  
  
• One of Samsung's memory chip factories suffered a 60-second power outage.  (Tom's Hardware)
  
  It will take up to three days to get the plant back on line and all the chips on the production line might need to be scrapped.  I'm not sure if the queue is as long for DRAM, but 3d flash has a lot of processing steps so they might lose weeks of production.
  
  
• Is the light over my desk blinking on and off or am I having a seizure?
  
  (Looks directly at LED downlight.)
  
  Fuck, now I'm blind.  That thing is far too bright.  Oh, and three of the LEDs were indeed flickering on and off.
  
  
• Apple has renewed its licensing agreement with GPU design company Imagination.  (AnandTech)
  
  Apple's custom Arm chips used Imagination graphics for years, before announcing they were going to use their own designs and cancelling their deal with Imagination, which almost killed that company.
  
  
• Oh no.
  
  
  
  

Disclaimer: Don't try this at home without first checking the deductables and exclusions clause in your insurance policy.

Posted by: Rick C at Thursday, January 02 2020 11:58 PM (Iwkd4)

Posted by: Rick C at Friday, January 03 2020 12:20 AM (Iwkd4)

Posted by: Pixy Misa at Saturday, January 04 2020 12:25 AM (PiXy!)

Posted by: Rick C at Saturday, January 04 2020 12:35 AM (Iwkd4)

Hide Comments | Add Comment

• Keep your computer cool this summer (well, it's summer here!) by dunking the whole thing in a tank of coolant that costs $18 a litre.  (AnandTech)
  
  The kit costs $2450.  That does not include the computer.  The computer is extra.
  
  
• This is not technical itself, but a social psychology study of clustering effects in social networks is certainly of interest, and the clusterfuck that the researchers managed to generate doubly so.
  
  
  
  
  The summary notes that moderate conservative voices on Twitter - the Center Right - are overwhelmed by the Extreme Right.  (Medium)
  
  What it does not note is what the definition of the Extreme Right is that is used by this study.  And that is because there is no definition used by the study.
  
  What they did was label certain Twitter accounts as Extreme Right, and then those accounts that follow them get tagged as Extreme Right as well.
  
  And the first example - there are only two - the first example of an Extreme Right account they give is The Heritage Foundation which is middle-of-the-road classical liberalism.
  
  Part one and part two are available for download as PDFs.  Part two consists entirely of interviews with journalists, if you needed a clearer idea of where this was going.
  
  The only bright spot in this farrago of academic malfeasance is page 22 of part one which notes that everybody hates the New York Times.
  
  
• Speaking of Medium, it takes five clicks and a page reload to reply to a comment - starting from the individual post page.  This is the competition?
  
  
• The Big 5 tech companies are worth $5 trillion.  (Tech Crunch)
  
  If you're playing Fantasy Stockmarket, short Google.  The company is rotten to the core.
  
  
• The old internet died and we did nothing.  (BuzzFeed)
  
  Don't know about you, BuzzFeed, but my posts from 2003 are still intact.
  
  And - hah, just thought of something.  I have a backup of every tweet I posted before Twitter nuked my account.  I'm about to launch my own social network....
  
  
• Samsung TVs may upload screenshots for automatic content recognition.  (Hacker News)
  
  I'll likely get a 4K TV this year.  I have 4K monitors - plural - because I use them all day for work, but my TV is still 720p.
  
  It will be a cheap, no-name device as devoid of intelligence as I can find.  If it can't even automatically switch to the active input, I'm okay with that.

• An interesting who-knew-what-when piece on the FISA abuse scandal.  (The Epoch Times)
  
  Is the Epoch Times trustworthy?  I don't know.  They were implicated in a fake news ring on Facebook recently.  (The Verge)  But they were implicated by Facebook and I don't trust Facebook either.
  
  But this article links to the PDF of the related court order - on the .gov site, not their own copy - and provides page references for the claims it makes.  That is refreshing.

MMO by ~ スズキゴロウ

Posted by: Rick C at Thursday, January 02 2020 04:13 AM (Iwkd4)

Posted by: Rick C at Thursday, January 02 2020 04:17 AM (Iwkd4)

Posted by: Rick C at Thursday, January 02 2020 05:01 AM (Iwkd4)

Posted by: Pixy Misa at Thursday, January 02 2020 11:09 AM (PiXy!)

Posted by: Rick C at Thursday, January 02 2020 01:11 PM (Iwkd4)

Hide Comments | Add Comment

• Mark Papermaster of AMD says there's more where that came from.  (AnandTech)
  
  The interview is irritatingly vague because you only really want to as a C-suite exec about future products and they can't talk about that without inviting a visit from the SEC.
  
  
• Meanwhile, here's a fanless Epyc desktop.  (AnandTech)
  
  The heatsink alone weighs 2.5kg.
  
  
• Police are getting a clue and keeping a registry of swatting targets.  (TechDirt)
  
  It's another damn thing that emergency responders have to check on, but it will save lives, and hopefully some bastiches will end up having a lot of time to reflect on their life choices.
  
  
• Horrible company Uber has sued horrible state California over the horrible law that protects gig workers from exploitation by denying them their gigs.  (Bloomberg)
  
  California, could you do us all a favour and slide quietly beneath the Pacific?  Thanks.
  
  
• The Chinese scientist who created genetically modified human babies has been banned from creating any more genetically modified human babies.  (Bloomberg)
  
  And sent to prison for three years.
  
  
• France will monitor your social media posts to check that you are not under-reporting your income.  (Reuters)
  
  As Isaac Asimov said, Welcome to the goldfish bowl.
  
  
• Updating the template parser for the new system.  I wrote some code for this a few years ago but never got it into production.  Dug it out today, and turns out (a) it worked perfectly and (b) it runs cleanly under Python 3.6 which didn't even exist at the time I wrote the code.  So score one for past me.

• Intel's 10th generation Comet Lake desktop chips could use two different sockets.  (Tom's Hardware)
  
  That is, different to each other, and also different to the current socket.  Because Intel hates you.
  
  
• The EVGA SR-3 DARK is a motherboard for the Xeon W-3175X.  (WCCFTech)
  
  It supports six DIMM slots, six PCIe slots, dual 10GbE ports, two M.2 slots, two U.2 ports, ten SATA ports, 10 USB 5 ports, 2 USB 10 ports....  And costs $1800.
  
  
• This might be an interesting blog post about Google's monopolistic practices.  (Medium)
  
  But I guess we'll never know.
  
  
  
  Don't use Medium for your blog.  Don't use Google for anything if you can possibly avoid it.

Posted by: Rick C at Tuesday, December 31 2019 12:56 AM (Iwkd4)

Posted by: Rick C at Tuesday, December 31 2019 01:00 AM (Iwkd4)

Posted by: Pixy Misa at Thursday, January 02 2020 12:07 AM (PiXy!)

Posted by: Rick C at Thursday, January 02 2020 04:08 AM (Iwkd4)

Posted by: Pixy Misa at Thursday, January 02 2020 11:06 AM (PiXy!)

Hide Comments | Add Comment

Needs a little refinement, so I'm going to up the test dataset to 10 million and then give that a shot.  Where it will fall down is if you are following very few people or people who haven't posted recently, but I can set a threshold to only scan so many thousand records if that becomes a problem.

The stack solution is still the way to go long term, but having a fallback that is also reasonably fast is very much a good thing.

Update: Refinements done and working like a charm on 10 million messages.

What this query does is this:

• Find the most recent messages
• From your friends
• Who you haven't muted
• And are in channels that you have access to
• And get their user details
• And the channel details
• And the parent message if it's a reply
• And the shared content if it's a share (like a retweet)
• And whether you've liked it
• Or bookmarked it
• Or reacted to it in another way
• And whether the poster is following you
• And the details of when you followed them
• And whether either party is blocking the other (which in this system just prevents interaction, not viewing content)
• And also whether either party has the other muted
• And if it's a poll, whether you have voted and which option you voted for

• The Radeon 5600XT looks to be a Radeon 5700 (non-XT) with the memory bus cut from 256 bits to 192 unless it isn't.  (WCCFTech)
  
  ASRock leaked the details of an upcoming card, confirming clocks, shader counts, memory speeds and size, everything but the TDP and price, basically.
  
  
• FreeDB is shutting down.  (Hacker News)
  
  FreeDB was forked from CDDB when that went project commercial.  It was acquired by Magic in 2006, so they've been maintaining it for free for 13 years now.
  
  
• Wyze, a vendor of IoT-enabled cameras and door locks, suffered a data breach that exposed the details of 2.4 million customers.  (ZDNet)
  
  The culprit was - you guessed it - Elasticsearch.
  
  
• Google Stadia will add yet another Tomb Raider game in January.  (WCCFTech)
  
  At that point, 20% of all games available on Stadia will be Tomb Raider.
  
  
• LG will be showing off an automated vegetable garden at CES.  (The Verge)
  
  It is the 21st century.  I guess.
  
  
• I was going over some benchmark scores today, and the Ryzen 3900X offers a bigger performance boost over my current 1700 than the 1700 does over the old FX-8150 Bulldozer chips.  On single-threaded tasks, never mind multi-threaded.
  
  I'm still going to hold out for Ryzen 3.
  
  
• YouTube has been evil for a long time.  (Archive.org)
  
  Back in 2008 they were in discussions with the developer of a small site built using the YouTube APIs, and offered to showcase him as a featured developer.
  
  At exactly the same time they changed their TOS to make sure his site could never earn a penny.
  
  Don't ask why the scorpion stings you.

Doing API testing on the new system today. Request routing, logins, sessions, cookies, automatic compression, all that good stuff.

The query that I was worried would slow down as the database grows - the standard user timeline - does indeed slow down as the database grows. I built in an engine to take care of that, and today I wrote the necessary query to use that engine.

Since that new query currently takes the time to build a timeline from 0.03s to 0.00s I'm adding more data to my test system to measure it again.

Oh, and you can search within your timeline. Twitter lets you do that now too, though.

Update: Stack engine vs. timeline engine:

500 timeline requests in 69.248s, avg request 139.1ms
500 stack requests in 3.963s, avg request 7.9ms

With a small database the timeline query was running fine. But if the system had taken off it would have been Fail Whale Squared. (I think this type of query caused about 90% of Twitter's problems in the early days.)

Stack requests automatically remember the last N items in your timeline so they don't have to mess around finding them again.

The other major mode is the channel request, which are used for blogs and forums and things like that. Those have no problems:

500 channel requests in 3.042s, avg request 6.1ms

That's the API request time, by the way, not just the database request, though for the timeline the database request is the overwhelming majority of the time.

I knew about this before but hadn't done the optimisation, because having a standard query let me enforce privacy checks in a single central location. Now I have three versions of the query and have to make sure the privacy checks are applied to each one.

Now I'm wondering if I can fix up that timeline query to make it run faster, because that could be really useful...

Update: Hah! Yes, that works. If I need to rebuild a user's stack, I can find the IDs of the last thousand posts that should appear in their timeline and shove them into their stack in 60 milliseconds flat. Then database queries within the stack take about 4ms.

The idea there is that if you don't log in for a while the system will stop updating your stack to save resources, but when you do log in I want it brought up to date quickly enough that you don't really notice it. 60ms is fine.

The main message query has five joins and ten subselects, which is great when the optimisation is just right because it gets everything the API needs in one go. When the optimisation is not just right, though, things go south in a hurry.

The stack works great because it means the main query never has to sort - to get the top 20 messages it just reads the top 20 stack records in index order and does five one-to-one joins.

• Had to restart Minx just now, and then block yet another misbehaving web crawler.
  
  I have more sympathy for J. Jonah Jameson every day.
  
  
• Cloudflare's warrant canary is an ex-parrot.  (TechDirt)
  
  Three of the seven statements included in the canary at the beginning of the year are now gone.
  
  
• There will be a second season of The Mandalorian.  (Tech Crunch)
  
  Disney having destroyed the Star Wars franchise in the cinemas and de-canonised the Expanded Universe, it's all they have left right now.
  
  
• Zen 3 will have a 17% IPC gain and a 50% floating point gain over Zen 2 unless it doesn't.  (WCCFTech)
  
  That's a lot, but AMD has been saying for a while that Zen 3 will be a significant update.
  
  
• Sonos devices feature a recycling mode that lets you retire your old equipment in a safe and environmentally friendly way - by irrevocably destroying it.
  
  
  
  
  Sustainability is non-negotiable, says Sonos.  You can't have it.
  
  
• Oh look another horribly security hole in NPM.  (Snyk.io)
  
  The article is from September and I believe this has since been fixed and replaced with seven brand new security holes.
  
  
• My test environment for the new system is on a private container in a virtual server behind two firewalls.  The container is accessible via SSH from my home IP address on a non-standard port by forwarding the SSH connection over an internal SSH tunnel on the container host.
  
  Now that I'm testing the app and API (and not just the code they're built on) I need access to the web services, so I have an SSH tunnel from WSL - a Linux virtual server running on Windows - running over the SSH tunnel on the KVM VPS to connect localhost:8080 on my PC to localhost:8080 on the container.
  
  This actually works.
  
  
• MSI has announced a mini-LED laptop to be introduced at CES.  (ZDNet)
  
  To be clear, at 17" it's not mini at all, and it has an LCD display, not LED.  Apart from that, though, it will deliver HDR 1000 and 100% DCI-P3 at 4K, which is as good as it gets right now.
  
  
• Plant-based burgers will make men grow boobs - says Livestock News.  (Ars Technica)
  
  The story (which seems to be popping up everywhere) is that Burger King's Impossible Whopper contains 18 million times as much estrogen as a regular Whopper.
  
  In fact it contains zero estrogen, because it's plant-based and plants don't produce estrogen.  What it contains are isoflavones, plant molecular analogs of estrogen that can bind to estrogen receptors but as far as any actual research has been able to determine, do absolutely nothing in humans.
  
  
• RethinkDB 2.4 is out.  (RethinkDB)
  
  RethinkDB is an interesting MongoDB-style document database that offered very flexible changefeeds (a.k.a tail cursors or notification streams) years before anyone else had them.
  
  The company behind it couldn't get enough funding - MongoDB sucked all the air out of the room there - and was forced to shut down, but the open-source repositories have been handed over to a volunteer effort and the project is back on its feet.
  

Posted by: Rick C at Sunday, December 29 2019 10:29 AM (Iwkd4)

Posted by: Rick C at Sunday, December 29 2019 10:32 AM (Iwkd4)

Posted by: Pixy Misa at Sunday, December 29 2019 03:27 PM (PiXy!)

Hide Comments | Add Comment

Tech News

• China's latest CPUs have caught up with AMD's Excavator.  (Tom's Hardware)
  
  Current mobile chips are already faster than Excavator.
  
  
• Graphene batteries aren't coming in 2020, and Huawei's P40 Pro won't have one unless it does.  (WCCFTech)
  
  This is getting confusing, guys.
  
  
• Europe is making life difficult for internet companies with a corporate presence within the EU.  You can however just choose not to have a corporate presence there.
  
  India is planning to close that loophole.  (Tech Crunch)
  
  If you have more than five million users and want to operate in India, you have to provide hostages.  And deploy a Class 2 AI to ban speech the Indian government might object to, before they object to it.
  
  The article discusses Wikipedia because they have hundreds of millions of users and would clearly fall under the planned legislation, but operate on a - relatively - tiny budget.
  
  
• In letter to dinosaurs, YouTube describes the Chicxulub Impactor as "regrettable".  (CoinDesk)
  
  
• Sony may have had to retreat from some markets but their digital sensor division is going gangbusters.  (Bloomberg)
  
  They make the cameras for a lot of phones, and their factories are running 24x365 and still not keeping up.
  
  
• Set JavaScript on fire and set that damn chart on fire too.  (StateofJS)
  
  
• Statler and Waldorf discuss the latest UX trends.  (grumpy.website)
  
  Yes, the site is actually called grumpy.website, and it delivers.
  
  
• I wonder how fast uWSGI RPC is compared to web requests...
  
  Update: uWSGI RPC doesn't seem to work.  Not sure what I'm doing wrong, but I don't want to rely on something this fiddly.
  
  Swapped in ZMQ and not only did it work first time, but I'm getting <500µs for a 100-byte JSON request with a 10K JSON response, and <900µs with a 100K response.  That's with the standard JSON encoder, not the custom one with the fancy date/time support.
  
  Let's see....  The smart encoder can spit out 10K in 120µs and 100K in 1.3ms.  That depends a lot on the balance of fiddly stuff (dates, decimals) and easy stuff (text) but it's not bad; if someone uses the smart JSON option it's not going to make the system collapse.  Anyway, it's fast enough to flood a gigabit internet uplink with JSON, which is all I ask.

Posted by: Rick C at Saturday, December 28 2019 11:10 AM (Iwkd4)

Posted by: Rick C at Saturday, December 28 2019 11:13 AM (Iwkd4)

Hide Comments | Add Comment

I fussed around with manually building uWSGI, but it turns out you don't need to do that.  Just install PyPy 2 alongside your PyPy 3 and use pip install uwsgi.  That will give you uWSGI built with the PyPy plugin, which works with either version.

You then need to use --pypy-home to point to your PyPy 3 directory and --pypy-lib to point to your PyPy .so file.

It still won't actually work, though, because the setup script is written in Python 2.  So grab this alternate setup script and specify it with --pypy-setup.

And now it works.

There's a thread with all the details on GitHub.

It's a bit disappointing that a complete solution exists but hasn't been pulled into the codebase, but at least a complete solution exists.

Also, having done all that, it's slower than with PyPy 2.  About 1.6ms for a proxy request vs. 1.2ms under PyPy 2.

Also also, the startup command is now full of nonsense:

But we can fix that.  Link the uwsgi binary into /usr/local/bin, create an ini file:

    [uwsgi]

    http-socket = 127.0.0.1:8080

    stats = 127.0.0.1:8081

    master = true

    processes = 1

    threads = 100

    pypy-home = /opt/pypy3

    pypy-lib = /opt/pypy3/bin/libpypy3-c.so

    pypy-setup = tools/pypy_setup.py