• Mark Papermaster of AMD says there's more where that came from.  (AnandTech)
  
  The interview is irritatingly vague because you only really want to as a C-suite exec about future products and they can't talk about that without inviting a visit from the SEC.
  
  
• Meanwhile, here's a fanless Epyc desktop.  (AnandTech)
  
  The heatsink alone weighs 2.5kg.
  
  
• Police are getting a clue and keeping a registry of swatting targets.  (TechDirt)
  
  It's another damn thing that emergency responders have to check on, but it will save lives, and hopefully some bastiches will end up having a lot of time to reflect on their life choices.
  
  
• Horrible company Uber has sued horrible state California over the horrible law that protects gig workers from exploitation by denying them their gigs.  (Bloomberg)
  
  California, could you do us all a favour and slide quietly beneath the Pacific?  Thanks.
  
  
• The Chinese scientist who created genetically modified human babies has been banned from creating any more genetically modified human babies.  (Bloomberg)
  
  And sent to prison for three years.
  
  
• France will monitor your social media posts to check that you are not under-reporting your income.  (Reuters)
  
  As Isaac Asimov said, Welcome to the goldfish bowl.
  
  
• Updating the template parser for the new system.  I wrote some code for this a few years ago but never got it into production.  Dug it out today, and turns out (a) it worked perfectly and (b) it runs cleanly under Python 3.6 which didn't even exist at the time I wrote the code.  So score one for past me.

• Intel's 10th generation Comet Lake desktop chips could use two different sockets.  (Tom's Hardware)
  
  That is, different to each other, and also different to the current socket.  Because Intel hates you.
  
  
• The EVGA SR-3 DARK is a motherboard for the Xeon W-3175X.  (WCCFTech)
  
  It supports six DIMM slots, six PCIe slots, dual 10GbE ports, two M.2 slots, two U.2 ports, ten SATA ports, 10 USB 5 ports, 2 USB 10 ports....  And costs $1800.
  
  
• This might be an interesting blog post about Google's monopolistic practices.  (Medium)
  
  But I guess we'll never know.
  
  
  
  Don't use Medium for your blog.  Don't use Google for anything if you can possibly avoid it.

Posted by: Rick C at Tuesday, December 31 2019 12:56 AM (Iwkd4)

Posted by: Rick C at Tuesday, December 31 2019 01:00 AM (Iwkd4)

Posted by: Pixy Misa at Thursday, January 02 2020 12:07 AM (PiXy!)

Posted by: Rick C at Thursday, January 02 2020 04:08 AM (Iwkd4)

Posted by: Pixy Misa at Thursday, January 02 2020 11:06 AM (PiXy!)

Hide Comments | Add Comment

Needs a little refinement, so I'm going to up the test dataset to 10 million and then give that a shot.  Where it will fall down is if you are following very few people or people who haven't posted recently, but I can set a threshold to only scan so many thousand records if that becomes a problem.

The stack solution is still the way to go long term, but having a fallback that is also reasonably fast is very much a good thing.

Update: Refinements done and working like a charm on 10 million messages.

What this query does is this:

• Find the most recent messages
• From your friends
• Who you haven't muted
• And are in channels that you have access to
• And get their user details
• And the channel details
• And the parent message if it's a reply
• And the shared content if it's a share (like a retweet)
• And whether you've liked it
• Or bookmarked it
• Or reacted to it in another way
• And whether the poster is following you
• And the details of when you followed them
• And whether either party is blocking the other (which in this system just prevents interaction, not viewing content)
• And also whether either party has the other muted
• And if it's a poll, whether you have voted and which option you voted for

• The Radeon 5600XT looks to be a Radeon 5700 (non-XT) with the memory bus cut from 256 bits to 192 unless it isn't.  (WCCFTech)
  
  ASRock leaked the details of an upcoming card, confirming clocks, shader counts, memory speeds and size, everything but the TDP and price, basically.
  
  
• FreeDB is shutting down.  (Hacker News)
  
  FreeDB was forked from CDDB when that went project commercial.  It was acquired by Magic in 2006, so they've been maintaining it for free for 13 years now.
  
  
• Wyze, a vendor of IoT-enabled cameras and door locks, suffered a data breach that exposed the details of 2.4 million customers.  (ZDNet)
  
  The culprit was - you guessed it - Elasticsearch.
  
  
• Google Stadia will add yet another Tomb Raider game in January.  (WCCFTech)
  
  At that point, 20% of all games available on Stadia will be Tomb Raider.
  
  
• LG will be showing off an automated vegetable garden at CES.  (The Verge)
  
  It is the 21st century.  I guess.
  
  
• I was going over some benchmark scores today, and the Ryzen 3900X offers a bigger performance boost over my current 1700 than the 1700 does over the old FX-8150 Bulldozer chips.  On single-threaded tasks, never mind multi-threaded.
  
  I'm still going to hold out for Ryzen 3.
  
  
• YouTube has been evil for a long time.  (Archive.org)
  
  Back in 2008 they were in discussions with the developer of a small site built using the YouTube APIs, and offered to showcase him as a featured developer.
  
  At exactly the same time they changed their TOS to make sure his site could never earn a penny.
  
  Don't ask why the scorpion stings you.

Doing API testing on the new system today. Request routing, logins, sessions, cookies, automatic compression, all that good stuff.

The query that I was worried would slow down as the database grows - the standard user timeline - does indeed slow down as the database grows. I built in an engine to take care of that, and today I wrote the necessary query to use that engine.

Since that new query currently takes the time to build a timeline from 0.03s to 0.00s I'm adding more data to my test system to measure it again.

Oh, and you can search within your timeline. Twitter lets you do that now too, though.

Update: Stack engine vs. timeline engine:

500 timeline requests in 69.248s, avg request 139.1ms
500 stack requests in 3.963s, avg request 7.9ms

With a small database the timeline query was running fine. But if the system had taken off it would have been Fail Whale Squared. (I think this type of query caused about 90% of Twitter's problems in the early days.)

Stack requests automatically remember the last N items in your timeline so they don't have to mess around finding them again.

The other major mode is the channel request, which are used for blogs and forums and things like that. Those have no problems:

500 channel requests in 3.042s, avg request 6.1ms

That's the API request time, by the way, not just the database request, though for the timeline the database request is the overwhelming majority of the time.

I knew about this before but hadn't done the optimisation, because having a standard query let me enforce privacy checks in a single central location. Now I have three versions of the query and have to make sure the privacy checks are applied to each one.

Now I'm wondering if I can fix up that timeline query to make it run faster, because that could be really useful...

Update: Hah! Yes, that works. If I need to rebuild a user's stack, I can find the IDs of the last thousand posts that should appear in their timeline and shove them into their stack in 60 milliseconds flat. Then database queries within the stack take about 4ms.

The idea there is that if you don't log in for a while the system will stop updating your stack to save resources, but when you do log in I want it brought up to date quickly enough that you don't really notice it. 60ms is fine.

The main message query has five joins and ten subselects, which is great when the optimisation is just right because it gets everything the API needs in one go. When the optimisation is not just right, though, things go south in a hurry.

The stack works great because it means the main query never has to sort - to get the top 20 messages it just reads the top 20 stack records in index order and does five one-to-one joins.

• Had to restart Minx just now, and then block yet another misbehaving web crawler.
  
  I have more sympathy for J. Jonah Jameson every day.
  
  
• Cloudflare's warrant canary is an ex-parrot.  (TechDirt)
  
  Three of the seven statements included in the canary at the beginning of the year are now gone.
  
  
• There will be a second season of The Mandalorian.  (Tech Crunch)
  
  Disney having destroyed the Star Wars franchise in the cinemas and de-canonised the Expanded Universe, it's all they have left right now.
  
  
• Zen 3 will have a 17% IPC gain and a 50% floating point gain over Zen 2 unless it doesn't.  (WCCFTech)
  
  That's a lot, but AMD has been saying for a while that Zen 3 will be a significant update.
  
  
• Sonos devices feature a recycling mode that lets you retire your old equipment in a safe and environmentally friendly way - by irrevocably destroying it.
  
  
  
  
  Sustainability is non-negotiable, says Sonos.  You can't have it.
  
  
• Oh look another horribly security hole in NPM.  (Snyk.io)
  
  The article is from September and I believe this has since been fixed and replaced with seven brand new security holes.
  
  
• My test environment for the new system is on a private container in a virtual server behind two firewalls.  The container is accessible via SSH from my home IP address on a non-standard port by forwarding the SSH connection over an internal SSH tunnel on the container host.
  
  Now that I'm testing the app and API (and not just the code they're built on) I need access to the web services, so I have an SSH tunnel from WSL - a Linux virtual server running on Windows - running over the SSH tunnel on the KVM VPS to connect localhost:8080 on my PC to localhost:8080 on the container.
  
  This actually works.
  
  
• MSI has announced a mini-LED laptop to be introduced at CES.  (ZDNet)
  
  To be clear, at 17" it's not mini at all, and it has an LCD display, not LED.  Apart from that, though, it will deliver HDR 1000 and 100% DCI-P3 at 4K, which is as good as it gets right now.
  
  
• Plant-based burgers will make men grow boobs - says Livestock News.  (Ars Technica)
  
  The story (which seems to be popping up everywhere) is that Burger King's Impossible Whopper contains 18 million times as much estrogen as a regular Whopper.
  
  In fact it contains zero estrogen, because it's plant-based and plants don't produce estrogen.  What it contains are isoflavones, plant molecular analogs of estrogen that can bind to estrogen receptors but as far as any actual research has been able to determine, do absolutely nothing in humans.
  
  
• RethinkDB 2.4 is out.  (RethinkDB)
  
  RethinkDB is an interesting MongoDB-style document database that offered very flexible changefeeds (a.k.a tail cursors or notification streams) years before anyone else had them.
  
  The company behind it couldn't get enough funding - MongoDB sucked all the air out of the room there - and was forced to shut down, but the open-source repositories have been handed over to a volunteer effort and the project is back on its feet.
  

Posted by: Rick C at Sunday, December 29 2019 10:29 AM (Iwkd4)

Posted by: Rick C at Sunday, December 29 2019 10:32 AM (Iwkd4)

Posted by: Pixy Misa at Sunday, December 29 2019 03:27 PM (PiXy!)

Hide Comments | Add Comment

Tech News

• China's latest CPUs have caught up with AMD's Excavator.  (Tom's Hardware)
  
  Current mobile chips are already faster than Excavator.
  
  
• Graphene batteries aren't coming in 2020, and Huawei's P40 Pro won't have one unless it does.  (WCCFTech)
  
  This is getting confusing, guys.
  
  
• Europe is making life difficult for internet companies with a corporate presence within the EU.  You can however just choose not to have a corporate presence there.
  
  India is planning to close that loophole.  (Tech Crunch)
  
  If you have more than five million users and want to operate in India, you have to provide hostages.  And deploy a Class 2 AI to ban speech the Indian government might object to, before they object to it.
  
  The article discusses Wikipedia because they have hundreds of millions of users and would clearly fall under the planned legislation, but operate on a - relatively - tiny budget.
  
  
• In letter to dinosaurs, YouTube describes the Chicxulub Impactor as "regrettable".  (CoinDesk)
  
  
• Sony may have had to retreat from some markets but their digital sensor division is going gangbusters.  (Bloomberg)
  
  They make the cameras for a lot of phones, and their factories are running 24x365 and still not keeping up.
  
  
• Set JavaScript on fire and set that damn chart on fire too.  (StateofJS)
  
  
• Statler and Waldorf discuss the latest UX trends.  (grumpy.website)
  
  Yes, the site is actually called grumpy.website, and it delivers.
  
  
• I wonder how fast uWSGI RPC is compared to web requests...
  
  Update: uWSGI RPC doesn't seem to work.  Not sure what I'm doing wrong, but I don't want to rely on something this fiddly.
  
  Swapped in ZMQ and not only did it work first time, but I'm getting <500µs for a 100-byte JSON request with a 10K JSON response, and <900µs with a 100K response.  That's with the standard JSON encoder, not the custom one with the fancy date/time support.
  
  Let's see....  The smart encoder can spit out 10K in 120µs and 100K in 1.3ms.  That depends a lot on the balance of fiddly stuff (dates, decimals) and easy stuff (text) but it's not bad; if someone uses the smart JSON option it's not going to make the system collapse.  Anyway, it's fast enough to flood a gigabit internet uplink with JSON, which is all I ask.

Posted by: Rick C at Saturday, December 28 2019 11:10 AM (Iwkd4)

Posted by: Rick C at Saturday, December 28 2019 11:13 AM (Iwkd4)

Hide Comments | Add Comment

I fussed around with manually building uWSGI, but it turns out you don't need to do that.  Just install PyPy 2 alongside your PyPy 3 and use pip install uwsgi.  That will give you uWSGI built with the PyPy plugin, which works with either version.

You then need to use --pypy-home to point to your PyPy 3 directory and --pypy-lib to point to your PyPy .so file.

It still won't actually work, though, because the setup script is written in Python 2.  So grab this alternate setup script and specify it with --pypy-setup.

And now it works.

There's a thread with all the details on GitHub.

It's a bit disappointing that a complete solution exists but hasn't been pulled into the codebase, but at least a complete solution exists.

Also, having done all that, it's slower than with PyPy 2.  About 1.6ms for a proxy request vs. 1.2ms under PyPy 2.

Also also, the startup command is now full of nonsense:

But we can fix that.  Link the uwsgi binary into /usr/local/bin, create an ini file:

    [uwsgi]

    http-socket = 127.0.0.1:8080

    stats = 127.0.0.1:8081

    master = true

    processes = 1

    threads = 100

    pypy-home = /opt/pypy3

    pypy-lib = /opt/pypy3/bin/libpypy3-c.so

    pypy-setup = tools/pypy_setup.py

• The Threadripper 3970X is a 32 core 280W monster.  But how does it perform if you dial the power back a little?  (WCCFTech)
  
  Thirdripper has configurable TDP, the same as Ryzen 3000.  So this doesn't even require manual undervolting; just fire up Ryzen Master and pick your TDP target.
  
  At 180W it still delivers 90% of full multi-core performance, and at 140W it manages 82%.  That gives us a good idea of the performance of the upcoming 3990X - double the cores, double the power budget back up to 280W, and it will be 64% faster than the 3970X.  Of course, it's not as simple as that, because memory bandwidth won't double, but memory bus power consumption won't double either, and cache will double...
  
  Cut to 95W the 3970X is clearly struggling and achieves only 48% of its full potential.  At that point it is actually slower than the Intel 10980XE.
  
  
• Samsung is investing $116 billion into its chip manufacturing division.  (The Verge)
  
  That's quite a lot.  TSMC's ascendence has effectively been bankrolled by Apple, and Samsung are their only real competition at this point.
  
  
• Cloudflare has taken on the role of supporting CDNJS.  (Cloudflare)
  
  While Cloudflare were providing the CDN for CDNJS, the project was run until now entirely by volunteers.  When some automated scripts broke last month and the volunteers didn't have the resources to fix it right away, the future of CDNJS seemed in doubt.
  
  So this is, on the whole, a good thing.
  
  
• TokuDB doesn't support foreign key constraints.  Doesn't even save your definitions, it just drops them straight onto the floor.  That's not a critical issue here because I control the application code as well as the database definitions, but it's a bit limiting.
  
  
• YouTube's latest target is cat videos.  I'm not even kidding.
  
  

Movie of the Day

Posted by: Rick C at Friday, December 27 2019 12:40 AM (Iwkd4)

Posted by: Pixy Misa at Friday, December 27 2019 12:59 AM (PiXy!)

Posted by: J Greely at Saturday, December 28 2019 08:20 AM (ZlYZd)

Posted by: Pixy Misa at Sunday, December 29 2019 01:27 AM (PiXy!)

Hide Comments | Add Comment

• Ponte Vecchio, Rambo Cache, and Gelato: The crisis inside Intel's codename division.  (AnandTech)
  
  Or possibly an in-depth analysis of Intel's upcoming high-performance computing platform.  One or the other.
  
  
• A Twitter bug allowed hackers to match your account to your phone number.  (Tech Crunch)
  
  Of course, Twitter has made it a habit to lock existing accounts for no reason and demand your phone number to unlock them, just as Facebook demands photos of yourself.  Otherwise Twitter wouldn't have the phone numbers for hackers to steal in the first place.
  
  
• How not to migrate customer data.  (Increment)
  
  Given the size of the project - 2500 man-years - and the requirement for a hard cutover, it was pretty much guaranteed to go wrong.  Which it did.
  
  
• A handy Python cheatsheet.
  
  Print it out and use it as a pillow.
  
  
• Hmm.  WebNX have 2288G and 3800X servers with NVMe SSDs.  ReliableSite also have the 3800X and I already have an account with them, but they don't use ECC RAM in their Ryzen servers and WebNX do.
  
  Anyway, won't need one for another couple of months yet.
  
  Even the 3800X will deliver 50% better single-threaded and nearly 3x the multithreaded performance of our current servers.  (CPU Benchmark)
  
  And that's the new low end.
  
  The WebNX servers are available with Intel enterprise 1.2TB NVMe SSDs.  I was thinking "Isn't that expensive for a server that only costs $125 per month?"
  
  Turns out the answer is no, not especially.
  
  Not sure if that's the model they're using - it's a couple of years old now and no longer in production - but it could well be given that a company like that would be looking for hardware that is reliable and cheap, rather than new and shiny.
  
  And it delivers 2.6GB per second on reads and 30 Âµs random write latency, so I certainly wouldn't be complaining if that's what I got.
  
  
• YouTube's latest target is cryptocurrency videos.  (Daily Hodl)
  
  Now certainly a large percentage of cryptocurrency content is nonsense, but no more than for the rest of YouTube.  And purging informational videos en masse as "dangerous or harmful content" is just...  Exactly what we've come to expect from the idiots running YouTube.
  
  
  
  
  
  
  
• Speaking of terrible messes Ethereum 2.0 might be arriving early next year.  (CoinDesk)
  
  It should be faster than current Ethereum, though not as fast as planned.  It probably won't be able to interoperate with current Ethereum.  It won't be as fast as planned.  It will be more expensive to run complex contracts that read data from the blockchain.
  
  And it won't offer generic atomic transactions, which could well be a disaster.
  
  Basically, Ethereum 2.0 divides the network into shards - 64 of them - each representing a single atomic distributed database.  If your application is on one shard and you want to interact with an application on another shard, it's time for you to roll your own two-phase commit implementation. 
  
  In a language that is basically awful, has all sorts of arbitrary limitations, and costs you real money for every instruction you execute.
  
  
• Ruby 2.7 is out.  (Ruby-lang)
  
  I don't really use Ruby, but I've always liked it and would have felt quite at home if Python hadn't shown up first.
  
  
• Redis 6.0 is at RC1 (release candidate one).  (GitHub)
  
  They don't expect to release the final version for three months or more; they go through an extended public test cycle with every new release to shake loose every bug they can.
  
  This release notably adds support for local caching: Your Redis client can cache data itself, directly in memory in the right format for your programming language, and Redis will notify it of cache invalidations.  This can speed up cache lookups by 10x or more at the cost of a little extra RAM; no extra application code required.
  
  I'm really looking forward to this.  It's not every day someone hands you a free 10x speedup.
  
  
• Merry Christmas everyone!

Video of the Day

Posted by: StargazerA5 at Thursday, December 26 2019 09:30 AM (puJpj)

Posted by: Pixy Misa at Thursday, December 26 2019 10:24 AM (PiXy!)

Hide Comments | Add Comment

• Testing an internal proxy server today, a little thing I wrote to manage filtering, routing, caching, logging, and statistics for API requests and web pages.
  
  It seems to work pretty well - it has a 90th percentile latency of 2.5ms and a 99th percentile latency of 4.1ms.  Given that it's written in Python and uses a MySQL server for management, that's not a bad result at all.
  
  
• Is the Asus ROG Zenith II Extreme the right board for your next Thirdripper build?  (AnandTech)
  
  With quad channel DDR4-4733, 16-phase power, an internal 1.7" OLED display (which I cannot find any pictures of or information about), 10Gbase-T, WiFi 6, eight SATA ports, five PCIE 4.0 x4 M.2 slots, one USB 20 port, seven USB 10 ports, six USB 5 ports, and four PCIe 4.0 x16 slots arranged as x16/x8/x16/x8 it just might be.
  
  It does cost $850 though.
  
  
• Apple's new Mac Pro has zero SATA ports, zero drive bays, and zero M.2 slots.  What storage it does have is not replaceable or upgradable, and only runs at 3.4GB per second.
  
  Enter the OWC Accelsior 4M2.  (AnandTech)
  
  It's just a PCIe card with up to 8TB of M.2 SSD on it, with a PCIe x8 interface, a built-in PCIe switch, and performance up to 6.7GB per second - nearly twice as fast as the Mac Pro.
  
  Price with 8TB is $1600.  Price for 8TB of SSD from Apple is $2600.  Price for a 7.68TB Micron 5200 ECO is $1400.
  
  So while not exactly cheap, it's not overpriced either.
  
  
• The Sonnet M.2 4x4 is much the same thing, except it's PCIe 3.0 x16, has no switch chip (as far as I can tell) and runs 10% faster.  (9to5Mac)
  
  Not sure what the limiting factor is here; they tested it with four Samsung 970 EVO drives, which should, in aggregate, be able to deliver 10GB per second on writes and 14GB per second on reads, rather than the 6.8 and 7.2 they measured.
  
  
• Ampere will release their 80 core Arm server CPU in Q1 of 2020.  (AnandTech)
  
  It uses the same Arm-designed architecture as Amazon's Graviton 2 and is manufactured on the same TSMC 7nm process; the big difference is that it will be available to customers other than Amazon.  Oh, and it has 25% more cores.
  
  
• Got a Ring camera?  Never too late to burn down your house.  (TechDirt)
  
  
• It's also never too late to burn your fish.  (Tech Crunch)
  
  
• It's also never too late to burn your Lyft e-bike.  (Tech Crunch)
  
  Oh wait, that already happened.
  
  
• How to handle a million websockets with Go.
  
  Step One: You don't need to handle a million websockets.
  Step Two: Seriously.
  Step Three: Get a DigitalOcean account, request an increase in droplet limits, and run 100 $5 droplets handling 10,000 websockets each.
  
  
• Protocol Buffers are wrong.  (Reasonably Polymorphic)
  
  I looked at these a while back when I needed to handle 10,000 messages per second.  We found another solution.
  
  
• Twitter has banned animated PNG attachments because they don't know to fix bugs anymore.  (ZDNet)

Posted by: Rick C at Wednesday, December 25 2019 02:31 AM (Iwkd4)

Posted by: Rick C at Wednesday, December 25 2019 06:57 AM (Iwkd4)

Posted by: Pixy Misa at Wednesday, December 25 2019 12:40 PM (PiXy!)

Hide Comments | Add Comment