This matters. This is important. Why did you say six months?
Why did you say five minutes?
Wednesday, April 10
It's A Spreadsheet And Therefore Technically I'm Working Edition
- DungeonDelveXL is a Roguelike role playing game implemented as an Excel spreadsheet.
What do you mean, "why"?
- The Democrats' empty net neutrality bill is DOA. (Reuters)
I'm in favour of focused, specific net neutrality legislation. This is exactly not that.
- Canonical are busy beavering away on booting Ubuntu from ZFS. (Phoronix)
Or at least installing to ZFS; they might still require an EXT boot partition.
- Google has a new Cloud Strategy. (ZDNet)
They got it from Oracle. So as you would expect, the strategy is to be as annoying as possible and twice as expensive.
- Hackers are probing servers within a minute of them going online. (ZDNet)
The solution: IPv6. Assign a single /128 and no-one will be able to find you until you activate your DNS.
- DH Games managed to lock every single Android player out of their popular game Idle Heroes and they don't have an automated password reset process.
- So today I ventured out beyond the Crystal standard library and... Uh, yeah, things immediately got rather bumpy.
Trying to do a
shards installof crystal-lmdb failed until I had the bright idea of removing the version specification. Compiling it failed because it couldn't find liblmdb.a located in /usr/local/lib.
(LMDB was in /usr/local/lib because I had to build it manually because the version that comes with Ubuntu 18.04 crashes when you run the test script bundled with crystal-lmdb. Well, good for the test script.
Fixed that and
shards buildcan't find the library I just installed with
shards install. I have no idea why. It congratulates me on finding a compiler bug. Yay.
I can just drop crystal-lmdb into my source directory and build it - and I did and it works and I can read and write a million (small) database records per second. But the build tools and their error messages need, let's just say, a little TLC.
Error: you've found a bug in the Crystal compiler. Please open an issue, including source code that will allow us to reproduce the bug: https://github.com/crystal-lang/crystal/issues
Social Media News
- Singapore is going full Ministry of Truth. (TechDirt)
- Europe is going full Ministry of Peace. (TechDirt)
- Britain is going full Ministry of Love. (TechDirt)
And also full Ministry of Children which Orwell was saving for the much darker sequel.
- Australia is going full Ministry of Plenty. (TechDirt)
Brilliant idea: Fine foreign corporations 10% of their annual revenue for ill-defined infractions that are impossible to avoid. Can't possibly have massively destructive and completely predictable adverse consequences.
- I believe all these countries are signatories to the UN Declaration of Human Rights which these laws and proposed laws massively violate, and all those involved should be declared war criminals and imprisoned in a specially constructed penitentiary at the bottom of the Challenger Deep until the heat death of the Universe.
- The New York Times is fighting the government of Singapore for that Minitrue role. (New York Times)
Venezuelan political refugee: You get jailed there because of tweets.
New York Times: But Trump.
There's a YouTube livestream of the announcement of the first direct images of a supermassive black hole in the galaxy M87.
Tuesday, April 09
- Well, yeah, sure the First Amendment says "Congress shall make no law" but it doesn't say "this means you". (TechDirt)
Republicans this time, because the urge to control what people say is not just bipartisan but almost omnipartisan.
- Netgear's new Nighthawk AX4 range of WiFi-6 routers shows the current problem with consumer and SMB networking: All but one model has significantly more WiFi bandwidth than wired bandwidth. (AnandTech)
So in theory you can do 6Gbps of WiFi, but you only have 1Gbps upstream to the internet or corporate network.
- Dell's Precision 7000 workstation supports up to 56 cores, three Nvidia Quadro RTX graphics cards, 3TB RAM, 16TB NVMe SSD, and 120TB of local hard disk. (AnandTech)
I'll take two.
- Has Google stopped indexing old web pages?
Well, let's see. I conveniently have a 16-year archive of stuff I can search for right here.
It found the current live archive page and direct link, and also the original post from Blogspot before it got imported first into Movable Type and then into Minx.
- Crystal is fast, but it's not C++. Two years ago I experimented with building a runtime for a new programming language (because at the time Crystal wasn't really ready for production) and I was getting 500 million instructions per second on my iMac. That's not bad for a simple implementation of a register-to-register bytecode interpreter.
I did a (very) quick rewrite in Crystal and it looks like it's trending towards ~100M ops per second as I implement more opcodes. Now, I am running the tests on Tohru (3.7GHz Ryzen 1700) rather than Taiga (4.2GHz i7 6700k) which accounts for some of the difference, but that's 15%, not 5x.
On the other hand, that's plenty fast for an embedded scripting language that runs inside a statically compiled application.
It might mean I need to use an array of pointers to inline blocks or something like that. Let me see...
Update: No, I think what's going on is that everything is bounds-checked (except, as my latest benchmark run just showed, integer overflow) where of course C/C++ says "You're on your own, mate. Good luck."
I've worked with Ada previously, years ago, and saw similar 5x performance differences between that and C.
5x slower is a lot. But:
- Crystal makes it 10x easier to write this code, which means that it might actually get done. Code that exists is almost infinitely faster than code that doesn't.
- Crystal itself provides the fast language I need - more than twice as fast as the best case for my C++ based interpreter - so this is just an engine for run-time features like scripts and templates.
- I can trivially spit out the bytecodes as compilable Crystal code and it gives back that 5x performance and then some. I got around 900M ops in a test run just now. (And a faster CPU would easily exceed 1B.)
And that is still using the generic data structures of the bytecode engine. So if you find yourself with a big piece of code that you need to speed up you can compile it and link it in to your binary, and mix and match Crystal, statically compiled bytecode, and dynamic bytecode however you want.
Side note: If I translate my bytecode to use named variables rather than the bytecode engine's indexed register files, the Crystal compiler optimises it down to a constant and runs in zero time. And if I turn off the optimiser, the named variable version of the code runs slower than the optimised register file version.
So I'm not sure exactly how fast that would be in a general case. The unoptimised named variable code is 15x faster than the unoptimised register file code, but that ratio would be impossible for the optimised version; the Ryzen CPU core can't actually issue that many instructions per second.
- I'm installing Crystal on my iMac. A little after midnight, and I hear this sound. What is that? It's the fan in my iMac, because the Homebrew install for Crystal builds the entire LLVM compiler framework from source. That fan never, ever spins up. I've had this iMac for three years and I don't think I've heard it before.
- Crystal has a port of the Ruby Language Toolkit, which includes all the stuff you need to build lexers and parsers and compilers, and has a fully worked example of simple language somewhere between Ruby and Python in syntax. So I can leverage all of that and don't need to start from scratch.
- Oh, there's a thought. Crystal is built on LLVM, as my iMac has pointed out rather volubly. But apart from that, Crystal is written in Crystal (though the very early versions were written in Ruby) so Crystal needs an interface to LLVM, and it has one. A very extensive interface. And LLVM has a JIT compiler. Only problem is reading through a couple of thousand pages of documentation. Well, something for v2 if v1 ever sees light of day.
Monday, April 08
Big In Akkad Edition
- I mentioned yesterday that Python has a library for everything.
Need to systematise your noun declensions in Classical Akkadian? There's a library for that.
- Crystal, as I mentioned, is single-threaded and has no serious inter-process IPC. If you are building a server application that needs to scale beyond one CPU core that's a problem, unless you are talking over a socket as pretty much every server application does, and are running on Linux kernel 3.9 or later, as pretty much every server does. In that case:
- reuse_port to enable multiple processes to bind to the same port (
You can just set the socket and start up multiple independent processes, as many as you like, all listening on the same port number. And basically that's it. Client connections go to whichever process grabs them first. It just works.
Not on Unix sockets though. Which means it is merely incredibly useful rather than perfect. I wondered how the Amber web framework got its performance numbers - whether it was tested behind a load balancer of some kind. No. Or to put it another way, yes, but the load balancer is inside the Linux kernel.
- reuse_port to enable multiple processes to bind to the same port (
Speaking of Linux kernels, 5.1 is on its way. (Phoronix)
Optane DIMM pricing has leaked some more. It is mostly in "if you have to ask" territory. (Tom's Hardware)
To be fair, buying 24TB of DDR4 RAM is not cheap either.
- MIT Technology Review provides some uninvited woker-than-thou advice to Google.
Burn academia to the ground. No exceptions.
Social Media News
- Facebook are "morally bankrupt liars" says idiot New Zealand civil servant who still believes in the Russiagate Hoax. (The Guardian)
Sunday, April 07
Tweetle Beetle Bottle Battle Edition
- My adventures in Crystal land continue so far without the crushing realisation that it all, in fact, sucks. Uploading binaries sucks because I'm still stuck on ADSL, but I'll get Crystal installed on my dev server shortly to sort that out.
I wrote a little test app to compare performance of HTTP via the Amber framework, raw HTTP by the bundled server, TCP sockets, UDP sockets, web sockets, and Unix sockets. Over a Unix socket with a single client and server process on my dev server I can get 50,000 requests per second. That's mostly consumed by the kernel and not by the app itself, because the packets I'm sending are small and I'm just echoing them back again without any real processing load.
Despite the fact that Crystal is statically typed and compiled, I can write code like this, compile it and run it without a single warning:
def handle_client(client)while message = client.getsclient.puts messageendend
def serve_tcpserver = TCPServer.new("localhost", 3000)while client = server.accept?spawn handle_client(client)endend
def serve_unixserver = UNIXServer.new("/tmp/amble.sock")while client = server.accept?spawn handle_client(client)endend
So TCP and Unix socket servers can share the same handler code transparently, and I haven't needed to define the type of anything anywhere. It could easily be Ruby code, except that it's fast.
The one notable limitation of Crystal right now is that it's single-threaded. Python and Ruby are multi-threaded but bound by something called a Global Interpreter Lock (the infamous GIL), which means that only one thread can actually be executing Python or Ruby instructions at any time. This actually makes things faster because otherwise you have to have locks controlling access to datastructures shared by the threads.
Crystal (currently) just has one operating system thread (plus a second one for the garbage collector) and uses green threads a.k.a fibers for internal concurrency. You can see that in the above code - every time a connection comes in to the socket I
spawna new thread. Fibers only use 4K of RAM each so you can have trillions of them if you want.
Internally it's using an event loop, but unlike a trainwreck like Node.js you don't care about that. You spawn your little fibers and get on with your life. Got 10,000 concurrent connections? Who cares? That's 40MB of RAM; it costs about 20¢ per month. If you even stopped for a minute to think about it you just wasted more money than it is likely to cost you.
And given that it's ten times faster than Python or Ruby for many tasks, it's like being given a multi-threaded language and a 10-core server for free. Even so, improved concurrency is at the top of the development roadmap.
I'm liking Crystal a lot so far, with the caveat that library support is (unsurprisingly) far behind Ruby, let alone Python. Python has a library for everything. Except VelocyPack.
Update: Ooh, a brand new LMDB binding! LMDB is perfect here because it supports multiple independent processes directly mapping the same database. I did find a couple of earlier bindings but the projects had been dormant a couple of years while the Crystal language has been updated regularly.
LMDB is kind of magical. It's pretty good at handling concurrent writes (it's properly ACIDic), though it doesn't deliver the transaction performance of high-end database engines. But being memory-mapped and copy-free it is phenomenally fast at reads. I ran some tests in Python a while back reading 1GB per second on a small VPS.
With LMDB it doesn't matter that your language has a GIL or uses green threads, because you can just run multiple processes. It handles transactions and crash recovery for you, the database is a single file, and it comes with a hot backup utility where the backups are themselves LMDB databases. That is, restore time once you have a backup is zero.
This makes a project I've wanted to do for years suddenly very, very feasible.
TSMC has started risk production of 5nm devices. (WikiChip Fuse)
This extends the use of EUV throughout the production process and significantly shrinks the geometry, packing about 80% more transistors per square millimetre. Performance and power improvements are expected to be smaller, on the order of
But MongoDB is web scale.
A good roundup of terrible trends in web development.
- Browsers are increasingly disabling disabling click tracking. (Bleeping Computer)
Except for Firefox among the major players. Brave also still allows this, and it's likely that MoonPie and Vivaldi and other niche browsers do too.
- Does Apple's "Magic Keyboard" still suck? Yes. (ZDNet)
Video of the Day
Just wait until he starts explaining how to construct five dimensional polytopes out of four dimensional ones. He even has a model.
Bonus Video of the Day
Saturday, April 06
Crystal Ball Edition
- I bought and read Programming Crystal last night and sat down this afternoon and wrote the little app I needed.
A bit of fiddling and refactoring and reading the docs for the Crystal standard library (which could do with some fleshing out) and it works. The code is clean, it compiles quickly, the binary is portable, and it's only 6 megabytes.
Okay, that last part isn't so great; as soon as I included the HTTP client library it yanked in 3.3MB of extra code. But that's what static linking does for you.
Don't concatenate large strings in a loop, though. Python handles that very well because it reserves space at the end of each string and can reuse the memory. Most languages don't do that trick and allocate memory and copy the entire string every time, so appending one byte to a one megabyte string is contraindicated.
This is something you have to watch out for just going from CPython to PyPy with the same code, so it's something I check with every language.
Oh yes. As to the IDE question, while there is no convenient CrystalMine from Jetbrains, and the plugin that is available is meh, if you install VSCode and Windows Services for Linux and Ubuntu 18.04 under WSL and Crystal under Ubuntu 18.04 under WSL and the Crystal Language extension for VSCode (the one by Faustino Aguilar) and the Crystal Installer for Windows which does not install Crystal on Windows because Crystal doesn't run on Windows bu instead creates a .bat file pointing to the Crystal on Ubuntu on WSL and puts the .bat file in your path... It mostly works.
I also gave the Amber web framework a try. After some initial swearing when I was editing the wrong copy of my source file, it basically worked. I've looked at innumerable web frameworks over the last decade and this one is easy to pick up. It compiles down to a portable binary (if you want). But it does use the Code of Cancer. Ugh.
That crap aside, it's a lot faster than CherryPy, which I have been using for years, even CherryPy running under PyPy. I'd previously tested Swift with the Vapor framework and it wasn't much faster than CherryPy under PyPy, certainly not enough to justify a switch of programming languages.
Responding to 50,000 simple queries takes around 70 CPU seconds with CherryPy on CPython, 20 seconds on PyPy, and 1.5 seconds with Amber on Crystal 0.27. That's a big difference. (This is all on a small VPS with a single server process and default configuration.)
This bears some thinking about.
- PostgreSQL is an enormously powerful SQL database built by perfectionists and designed by idiots that tells you to go fuck yourself at every possible occasion.
- Optane DIMMs cost about half as much as RAM. (AnandTech)
A 128GB Optane DIMM costs around $900, the price of a 64GB DDR4 module, and a 256GB Optane DIMM at $2850 is about the price of a 128GB DDR4 module. (Higher density modules are proportionally more expensive.)
On the other hand, you can only use Optane DIMMs with very expensive CPUs, and they are significantly slower than DRAM (though still lightning fast compared to SSDs.) So the market for these at the moment seems small.
- "Ironically" too many video streaming services is leading to an increase in piracy. (TechDirt)
That's not irony, you idiots, that's arithmetic.
All told, there are more than 300 over-the-top video options in the U.S.
Lessons learned from porting 50,000 lines of code from Java to Go.
From the publisher of such titles as A Walking Tour of the Sewers of Major European Cities and Leaving Ethiopia for Somalia.
A one-day bug bounty program run by Dropbox found 264 vulnerabilities. (ZDNet)
A misconfigured Qt5 app and a malicious link can ruin your whole day. (Bleeping Computer)
A sneaky hacker slipping a backdoor into a popular library can ruin your whole day. (ZDNet)
Fortunately this one was caught and removed the same day it appeared.
Social Media News
- When all the news is bad maybe it's time to buy a newspaper. (Business Insider)
It almost worked for Jeff Bezos.
5K 360° Video of the Day
For those of you with iMacs and gigabit internet access.
Friday, April 05
All Day Does Not Include 6:49 AM Edition
- Intel does not want customers using Optane memory. (AnandTech)
The cheapest model in the new Xeon lineup is $213. It's kind of slow, but hey, $213.
The cheapest model supporting Optane memory is $9119. (Unless I'm misreading the price list, which is entirely possible.)
- Microsoft says okay, fine, don't update, see if we care. (PC Perspective)
They'll still force-feed you security patches and reboot your computer at an inconvenient time at least once a month, but you can at least delay the feature updates.
- Google shut down its new AI ethices advisory board after it turned out that Google's employees are mostly insane. (Tech Crunch)
- WCCFTech is 15 years old today unless it isn't. (WCCFTech)
- 93% of paint splatters are valid Perl programs.
- A dual socket AMD Rome system will have three and possibly four times the bandwidth of Intel's competing Platinum 9200 systems. (Serve the Home)
If your motherboard and devices support PCIe 4.0, anyway. Otherwise the gain is only 60-100%.
- This asshole is always wrong. (ZDNet)
I'll get back to that on the weekend.
- Pixy's Third Law of Software Development: Windows developer tools expand until they include a complete Linux installation. Sometimes two.
Thursday, April 04
Fuck This Shit Edition
- Carpe Diem is a lovely Apache privilege escalation bug that fortunately was fairly easy to patch.
- Intel has released its first Wi-Fi 6 cards. (AnandTech)
They're standard M.2 2230 devices so you should be able to drop them into any system that uses such - which is probably any computer you can open anyway.
- Of all the possible antitrust cases currently floating around like so many turtles in a giant bowl of tomato soup the DOJ picked up on Steven Spielberg's anti-Netflix rant. (TechDirt)
Are they all asleep over there?
- Anyone working with the Facebook API has recently been faced with a 30-page questionnaire that asks for details right down to the vendor of the hardware on which you are running your virtual machines that run the databases that store your data.
This is why. (Tech Crunch)
- Microsoft's news app had a configuration bug that made users think their computer is infected with malware. (Bleeping Computer)
- Which to be fair is probably true. (ZDNet)
- Be sure to make a good impression on your next 8:30 AM international developer conference call with Voidol, which can make you sound like a cute anime girl and/or the lead singer from Queen. (One Angry Gamer)
Only two settings, I'm afraid. Mikuru and Mercury.
- Nuitka's idea of a standalone binary is not quite the same as mine. Plus if you use the Requests library, which I do, rather a lot, it burns down, falls over, and sinks into the swamp.
I did get to spend half an hour with Crystal, and it does seem to work, though I'm not certain that the standalone executable I generated is truly standalone.
Social Media News
- There's a lot of social media news to get to, but it all sucks, so I'm leaving it for tomorrow. Or maybe the weekend, when I'll have time to work up a really good rant.
Four Dimensional Visualisation Trick of the Day
This video is cool because not only does it explain how Klein bottles work, it gives you a way to accurately visualise many kinds of four-dimensional structures just using familiar everyday concepts. Ten dimensional hyperspheres not so much; the trick only really lets you jump from 3D to 4D, not beyond.
Wednesday, April 03
Fifty-Six Is The New Forty-Eight Edition
- Intel announced a bunch of stuff at their innovation day. (AnandTech)
Mostly enterprise-focused, unless you're looking for a 224-core, 448-thread gaming laptop.
- 56 core Cascade Lake CPUs, previously hinted at 48. These are two 28 core dies on a package, with 12 memory channels, and use a rather hefty 400W.
No pricing announced for these, but the new Xeon Platinum 8280L with 28 cores will set you back $17,906. Further down the line, a 16 core Xeon Silver 4216 is a rather more reasonable $1002.
- New Optane memory modules can be used as main memory (though you need at least some regular DRAM in the system) and come in capacities up to 512GB. Oddly the top-of-the-line 56 core CPUs don't support them.
- Agilex (Intel seems to be plagued by dumb name syndrome) is a new range of Altera FPGAs with PCIe 5.0 and DDR5 support, ready for the next generation of really expensive networking equipment.
- For the consumer market, Intel is offering a small box of coffee beans. (AnandTech)
A quad-core CPU with Iris Plus graphics puts this well ahead of other NUCs except for the expensive
Devil'sSkull Canyon double-wide models.
- If the 4.60" x 4.40" x 2.01" size of a NUC is just too big for your apartment, the ECS Liva Q2 might be what you need. (Tech Powerup)
It weighs in at a svelte 2.75" x 2.75" x 1.3" though admittedly the Celeron N4000 and the eMMC storage will be noticeably slower than the Bean Canyon NUC's Core i7 and NVMe SSD.
- Lego's new Spike Prime robotics kit is aimed at improving STEAM* education. (Tom's Hardware)
* Science, Technology, Engineering, Worthless Piece of Paper That Will Doom You To A Life in the Service Industry, and Mathematics.
- Sites need to start installing the filters that are not required for compliance with Europe's catastrophic Article 17 (nee 13) right away. (TechDirt)
- FOSTA is doing by malicious incompetence what the CDA tried to do by malicious intent. (TechDirt)
- Singapore's submission to the global Ministry of Truth contest that seems to have sprung up looks promising. (Tech Crunch)
Internet content deemed to be false will be forcibly corrected, with fines of up to SG$1M (US$740K) applying to social networks that fail to comply. As with all other plots by the current crop of pig-ignorant power-crazed censorious lunatics, the plan applies to the entire world.
- Don't like the web browsers currently available? Why not build your own? There's plenty of open source projects available, from KHTML to WebKit to Chromium to Firefox.
The answer, according to Google, is because fuck you that's why.
Where's the DOJ? You can nail Google to the wall over this.
- Which Xeon is right for you? (Serve the Home)
A handy colour-coded chart will help you decide. The previously mentioned 4216 is the same price as last year's 4116 but adds four more cores, which seems like a good deal.
Tuesday, April 02
All The Bad Web Pages Are Gone Now Edition
- Google Hangouts is terrible.
- Cloudflare has always been at war with New Zealand. (Tom's Hardware)
I don't want to hand even more power to Cloudflare, but if you want a free VPN, most of the other products on offer are worse. Except that Cloudflare Warp is not actually available just yet. Minor detail.
- Ryzen 3000 chips and X570 motherboards have started showing up in benchmark databases. (Tom's Hardware)
Nothing particularly exciting yet, unfortunately; still engineering samples and low-end parts.
- Facebook says please don't throw me into that anticompetitive regulatory briar patch. (TechDirt)
- Researchers say they have found a new way to hack into Intel-based computers. (ZDNet)
Intel says "nuh-uh".
Sofa Optimisation Documentary of the Day
Also, if you pack 1024 10-dimensional hyperspherical unit-radius sofas into a length 4 10-dimensional hypercubic moving van (which fits perfectly) the space left over between the sofas is larger than the van.
57 queries taking 0.2581 seconds, 354 records returned.
Powered by Minx 1.1.6c-pink.