Saturday, April 13
Tech News
- Flink 1.8.0 is out.
What is Flink? I don't know. I read "What is Apache Flink?" and I still don't know.
- Intel has announced - sort of - the Xeon Gold U series. (Serve the Home)
This includes a 20 core server processor for around $1000 - the 6209U - which is amazing value compared to other Intel server parts, but only supports single-socket motherboards.
It replaces traditional configs with two ten core CPUs for much less money - the same configurations AMD has targeted with its 24 core Epyc 7401P at $1075. Intel's advantage is that it has higher boost clock speeds - 3.9GHz vs. 3.0GHz on the Epyc. AMD's advantage is it offers vastly more I/O - 128 PCIe lanes vs. just 40 on the Xeon.
Supermicro are ready and waiting with their X11SPA-TF motherboard. It's focused more at workstations but has onboard BMC so it's fine for servers as well. Single socket LGA3467, 12 DIMM slots (up to 3TB RAM), 4 M.2 slots, 8 SATA ports, gigabit and 10G Ethernet (both RJ45), plus another port dedicated to IPMI.
- Doing what the DOJ won't: Apple may be breaking up iTunes. (Six Colors)
Much of iTunes' terribleness comes down to it saving all your metadata in a single huge XML file rather than using a database the way any even vaguely sane person would.
- Intel's H10 SSD is a piece of garbage that no-one should buy. (ZDNet)
It puts up to 32GB of Optane cache together with up to 1TB of QLC flash. So far so good. A small amount of high-performance, high-endurance Optane backed by lots of cheap QLC flash. What's not to like?
What's not to like is that this is not a single device. It is two separate devices - a tiny useless Optane drive and a large, slow QLC flash drive with no integrated pseudo-SLC cache. It only actually works if you have an 8th or 9th generation Intel CPU and RST support.
Another two generations and they might actually get it right; for now just get the Intel 660p for fast bulk storage (~$100 per TB) or the Samsung 870 EVO Plus for high-speed storage (~$250 per TB).
Social Media News
- Facebook accidentally outs Big Brother. (Business Insider)
Oops.
- Trust no-one. Block everyone. (TechDirt)
The truth isn't out there.
Crystal Ballroom
- Crystal has an issue that prevents you from building static binaries on MacOS. Or so it seemed. Turns out that MacOS has an issue that prevents you from building static libraries on MacOS: Some system libraries have no static version, and can only be dynamically linked.
You can however build a portable Mac binary with Crystal that includes everything except the system libraries, and will run on any sufficiently recent version of the operating system.
So that lets me run on any version of Linux (3.9 kernel or later for full functionality), and on Mac. Currently I have a problem even on WSL let alone directly on Windows, but we'll see about that.
Posted by: Pixy Misa at
04:59 PM
| Comments (8)
| Add Comment
| Trackbacks (Suck)
Post contains 517 words, total size 5 kb.
Friday, April 12
One Law Part Two Edition
- Rust 1.34.0 is out.
- Kotlin 1.3.30 is out.
- Crystal 0.28.0 is... It's dated "2019-04-??"? Okay, maybe not quite out.
- Sharp has a new 8K 120Hz monitor. (AnandTech)
They already have an 8K 60Hz model.
- Sony says pfft, demonstrates 783" 16K monitor. (AnandTech)
That is... Fairly large.
- There's an exploit for WPA3 and it's barely even out yet. (PC Perspective)
If you're using WPA3, use a password with 10 or more characters. If you're not using WPA3, use something even longer.
Social Media News
- Mike Masnick wants to make it totally clear that there is no social media bias against conservatives. (TechDirt)
- Google just banned the Dissenter extension from the Chrome store for "hate speech". (One Angry Gamer)
Of course, there is no hate speech in the extension. It just delivers content from the internet.
One law for left and right alike, that forbids them equally from saying "learn to code" and assuming someone's gender.
- Google totally doesn't shadow ban sites as you can tell by the lack of results if you search for "google shadow ban" on Google. (One Angry Gamer)
- Zero tolerance policies are recursive. (Ars Technica)
Rules against drug use become rules against discussing drug use become rules against discussing literature discussing drug use, ad infinitum.
Pro Tip: Print your books on hemp paper.
Video of the Day

Posted by: Pixy Misa at
04:57 PM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 239 words, total size 4 kb.
Ftruncate Does Not Truncate Mmap'd Files Edition
Tech News
- HP's new Xeon workstations also go up to 56 cores and 6TB of Optane memory. (AnandTech)
Prices start at $2372 and continue up past $100,000.
- The EU has identified a great amount of terrorist content hosted on archive.org including Project Gutenberg, a collection of Grateful Dead bootleg recordings, and CSPAN. (TechDirt)
Under the EU's new censorship law - no, a different one - archive.org would have one hour to remove that content or face criminal prosecution.
- Firefox has removed Dissenter from their add-on gallery. (One Angry Gamer)
Mozilla were quite clear on this: If your extension is used by people to say things Mozilla does not like, it will be removed.
Web browsers have to be neutral to retain users' trust. Without that they are dead. I don't think any of the major browser companies remember that fundamental rule.
- Ethereum continues to be a pain in the ass.
- Oh joy, an API method that returns success when the value is unknown. Good work guys.
Crystal Ballroom
- LMDB doesn't work properly on WSL, which is also a pain in the ass. There are a couple of known bugs in WSL's mmap that give you happy little segmentation faults in place of working code. That also applies to Docker on Windows.
I was looking at building an app in Crystal that you could run on Windows either using Docker or WSL, but this blows that up. The other thing I could do is write a wrapper that emulates LMDB with SQLite, which has the advantage of running with 100% reliability on everything from supercomputers to vacuum cleaners. Or just use SQLite as SQLite. I know, radical idea.
And static linking on MacOS doesn't work yet either.
- On the other hand, when it does work, I can read and write 300,000 native Crystal records per second sequentially, single-threaded. Or, and this is important, 180,000 random reads per second. That's converting everything to JSON and back again, unpacking nested data structures, all that stuff.
Random access time is around 2-2.5µs, with the same going for JSON decoding. Throughput should scale linearly with core count but I just tried it and it doesn't, which is kind of annoying.
- Hmm. How fast is SQLite then?
Read 2000000 records in 00:00:01.68
Okay, that should do. (Those are smaller records, so it's probably about the same speed as LMDB.)

Posted by: Pixy Misa at
12:15 AM
| Comments (4)
| Add Comment
| Trackbacks (Suck)
Post contains 425 words, total size 4 kb.
Wednesday, April 10
It's A Spreadsheet And Therefore Technically I'm Working Edition
Tech News
- DungeonDelveXL is a Roguelike role playing game implemented as an Excel spreadsheet.
What do you mean, "why"?
- The Democrats' empty net neutrality bill is DOA. (Reuters)
I'm in favour of focused, specific net neutrality legislation. This is exactly not that.
- Canonical are busy beavering away on booting Ubuntu from ZFS. (Phoronix)
Or at least installing to ZFS; they might still require an EXT boot partition.
- Google has a new Cloud Strategy. (ZDNet)
They got it from Oracle. So as you would expect, the strategy is to be as annoying as possible and twice as expensive.
- Hackers are probing servers within a minute of them going online. (ZDNet)
The solution: IPv6. Assign a single /128 and no-one will be able to find you until you activate your DNS.
- DH Games managed to lock every single Android player out of their popular game Idle Heroes and they don't have an automated password reset process.
Ruby Ballroom
- So today I ventured out beyond the Crystal standard library and... Uh, yeah, things immediately got rather bumpy.
Trying to do ashards install
of crystal-lmdb failed until I had the bright idea of removing the version specification. Compiling it failed because it couldn't find liblmdb.a located in /usr/local/lib.
(LMDB was in /usr/local/lib because I had to build it manually because the version that comes with Ubuntu 18.04 crashes when you run the test script bundled with crystal-lmdb. Well, good for the test script.
Fixed that andshards build
can't find the library I just installed withshards install
. I have no idea why. It congratulates me on finding a compiler bug. Yay.
I can just drop crystal-lmdb into my source directory and build it - and I did and it works and I can read and write a million (small) database records per second. But the build tools and their error messages need, let's just say, a little TLC.Error: you've found a bug in the Crystal compiler. Please open an issue, including source code that will allow us to reproduce the bug: https://github.com/crystal-lang/crystal/issues
Social Media News
- Singapore is going full Ministry of Truth. (TechDirt)
- Europe is going full Ministry of Peace. (TechDirt)
- Britain is going full Ministry of Love. (TechDirt)
And also full Ministry of Children which Orwell was saving for the much darker sequel.
- Australia is going full Ministry of Plenty. (TechDirt)
Brilliant idea: Fine foreign corporations 10% of their annual revenue for ill-defined infractions that are impossible to avoid. Can't possibly have massively destructive and completely predictable adverse consequences.
- I believe all these countries are signatories to the UN Declaration of Human Rights which these laws and proposed laws massively violate, and all those involved should be declared war criminals and imprisoned in a specially constructed penitentiary at the bottom of the Challenger Deep until the heat death of the Universe.
- The New York Times is fighting the government of Singapore for that Minitrue role. (New York Times)
Venezuelan political refugee: You get jailed there because of tweets.
New York Times: But Trump.
Posted by: Pixy Misa at
11:59 PM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 525 words, total size 6 kb.
There's a YouTube livestream of the announcement of the first direct images of a supermassive black hole in the galaxy M87.
Posted by: Pixy Misa at
11:10 PM
| Comments (1)
| Add Comment
| Trackbacks (Suck)
Post contains 28 words, total size 1 kb.
Posted by: Pixy Misa at
10:33 PM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 72 words, total size 1 kb.
Tuesday, April 09
Tech News
- Well, yeah, sure the First Amendment says "Congress shall make no law" but it doesn't say "this means you". (TechDirt)
Republicans this time, because the urge to control what people say is not just bipartisan but almost omnipartisan.
- Netgear's new Nighthawk AX4 range of WiFi-6 routers shows the current problem with consumer and SMB networking: All but one model has significantly more WiFi bandwidth than wired bandwidth. (AnandTech)
So in theory you can do 6Gbps of WiFi, but you only have 1Gbps upstream to the internet or corporate network.
- Dell's Precision 7000 workstation supports up to 56 cores, three Nvidia Quadro RTX graphics cards, 3TB RAM, 16TB NVMe SSD, and 120TB of local hard disk. (AnandTech)
I'll take two.
- Has Google stopped indexing old web pages?
Well, let's see. I conveniently have a 16-year archive of stuff I can search for right here.
It found the current live archive page and direct link, and also the original post from Blogspot before it got imported first into Movable Type and then into Minx.
Analysis: False.
Crystal Ballroom
- Crystal is fast, but it's not C++. Two years ago I experimented with building a runtime for a new programming language (because at the time Crystal wasn't really ready for production) and I was getting 500 million instructions per second on my iMac. That's not bad for a simple implementation of a register-to-register bytecode interpreter.
I did a (very) quick rewrite in Crystal and it looks like it's trending towards ~100M ops per second as I implement more opcodes. Now, I am running the tests on Tohru (3.7GHz Ryzen 1700) rather than Taiga (4.2GHz i7 6700k) which accounts for some of the difference, but that's 15%, not 5x.
On the other hand, that's plenty fast for an embedded scripting language that runs inside a statically compiled application.
It might mean I need to use an array of pointers to inline blocks or something like that. Let me see...
Update: No, I think what's going on is that everything is bounds-checked (except, as my latest benchmark run just showed, integer overflow) where of course C/C++ says "You're on your own, mate. Good luck."
I've worked with Ada previously, years ago, and saw similar 5x performance differences between that and C.
5x slower is a lot. But:
- Crystal makes it 10x easier to write this code, which means that it might actually get done. Code that exists is almost infinitely faster than code that doesn't.
- Crystal itself provides the fast language I need - more than twice as fast as the best case for my C++ based interpreter - so this is just an engine for run-time features like scripts and templates.
- I can trivially spit out the bytecodes as compilable Crystal code and it gives back that 5x performance and then some. I got around 900M ops in a test run just now. (And a faster CPU would easily exceed 1B.)
And that is still using the generic data structures of the bytecode engine. So if you find yourself with a big piece of code that you need to speed up you can compile it and link it in to your binary, and mix and match Crystal, statically compiled bytecode, and dynamic bytecode however you want.
Side note: If I translate my bytecode to use named variables rather than the bytecode engine's indexed register files, the Crystal compiler optimises it down to a constant and runs in zero time. And if I turn off the optimiser, the named variable version of the code runs slower than the optimised register file version.
So I'm not sure exactly how fast that would be in a general case. The unoptimised named variable code is 15x faster than the unoptimised register file code, but that ratio would be impossible for the optimised version; the Ryzen CPU core can't actually issue that many instructions per second.
- I'm installing Crystal on my iMac. A little after midnight, and I hear this sound. What is that? It's the fan in my iMac, because the Homebrew install for Crystal builds the entire LLVM compiler framework from source. That fan never, ever spins up. I've had this iMac for three years and I don't think I've heard it before.
- Crystal has a port of the Ruby Language Toolkit, which includes all the stuff you need to build lexers and parsers and compilers, and has a fully worked example of simple language somewhere between Ruby and Python in syntax. So I can leverage all of that and don't need to start from scratch.
- Oh, there's a thought. Crystal is built on LLVM, as my iMac has pointed out rather volubly. But apart from that, Crystal is written in Crystal (though the very early versions were written in Ruby) so Crystal needs an interface to LLVM, and it has one. A very extensive interface. And LLVM has a JIT compiler. Only problem is reading through a couple of thousand pages of documentation. Well, something for v2 if v1 ever sees light of day.
Posted by: Pixy Misa at
11:26 PM
| Comments (5)
| Add Comment
| Trackbacks (Suck)
Post contains 855 words, total size 7 kb.
Monday, April 08
Big In Akkad Edition
Tech News
- I mentioned yesterday that Python has a library for everything.
Need to systematise your noun declensions in Classical Akkadian? There's a library for that.
- Crystal, as I mentioned, is single-threaded and has no serious inter-process IPC. If you are building a server application that needs to scale beyond one CPU core that's a problem, unless you are talking over a socket as pretty much every server application does, and are running on Linux kernel 3.9 or later, as pretty much every server does. In that case:
- reuse_port to enable multiple processes to bind to the same port (
SO_REUSEPORT
).
You can just set the socket and start up multiple independent processes, as many as you like, all listening on the same port number. And basically that's it. Client connections go to whichever process grabs them first. It just works.
Not on Unix sockets though. Which means it is merely incredibly useful rather than perfect. I wondered how the Amber web framework got its performance numbers - whether it was tested behind a load balancer of some kind. No. Or to put it another way, yes, but the load balancer is inside the Linux kernel.
- reuse_port to enable multiple processes to bind to the same port (
-
Optane DIMM pricing has leaked some more. It is mostly in "if you have to ask" territory. (Tom's Hardware)
To be fair, buying 24TB of DDR4 RAM is not cheap either.
- MIT Technology Review provides some uninvited woker-than-thou advice to Google.
Burn academia to the ground. No exceptions.
Social Media News
- Facebook are "morally bankrupt liars" says idiot New Zealand civil servant who still believes in the Russiagate Hoax. (The Guardian)
Posted by: Pixy Misa at
07:23 PM
| Comments (2)
| Add Comment
| Trackbacks (Suck)
Post contains 296 words, total size 3 kb.
Sunday, April 07
Tweetle Beetle Bottle Battle Edition
Tech News
- My adventures in Crystal land continue so far without the crushing realisation that it all, in fact, sucks. Uploading binaries sucks because I'm still stuck on ADSL, but I'll get Crystal installed on my dev server shortly to sort that out.
I wrote a little test app to compare performance of HTTP via the Amber framework, raw HTTP by the bundled server, TCP sockets, UDP sockets, web sockets, and Unix sockets. Over a Unix socket with a single client and server process on my dev server I can get 50,000 requests per second. That's mostly consumed by the kernel and not by the app itself, because the packets I'm sending are small and I'm just echoing them back again without any real processing load.
Despite the fact that Crystal is statically typed and compiled, I can write code like this, compile it and run it without a single warning:
def handle_client(client)while message = client.getsclient.puts messageendend
def serve_tcpserver = TCPServer.new("localhost", 3000)while client = server.accept?spawn handle_client(client)endend
def serve_unixserver = UNIXServer.new("/tmp/amble.sock")while client = server.accept?spawn handle_client(client)endend
So TCP and Unix socket servers can share the same handler code transparently, and I haven't needed to define the type of anything anywhere. It could easily be Ruby code, except that it's fast.
The one notable limitation of Crystal right now is that it's single-threaded. Python and Ruby are multi-threaded but bound by something called a Global Interpreter Lock (the infamous GIL), which means that only one thread can actually be executing Python or Ruby instructions at any time. This actually makes things faster because otherwise you have to have locks controlling access to datastructures shared by the threads.
Crystal (currently) just has one operating system thread (plus a second one for the garbage collector) and uses green threads a.k.a fibers for internal concurrency. You can see that in the above code - every time a connection comes in to the socket Ispawn
a new thread. Fibers only use 4K of RAM each so you can have trillions of them if you want.
Internally it's using an event loop, but unlike a trainwreck like Node.js you don't care about that. You spawn your little fibers and get on with your life. Got 10,000 concurrent connections? Who cares? That's 40MB of RAM; it costs about 20¢ per month. If you even stopped for a minute to think about it you just wasted more money than it is likely to cost you.
And given that it's ten times faster than Python or Ruby for many tasks, it's like being given a multi-threaded language and a 10-core server for free. Even so, improved concurrency is at the top of the development roadmap.
I'm liking Crystal a lot so far, with the caveat that library support is (unsurprisingly) far behind Ruby, let alone Python. Python has a library for everything. Except VelocyPack.
Update: Ooh, a brand new LMDB binding! LMDB is perfect here because it supports multiple independent processes directly mapping the same database. I did find a couple of earlier bindings but the projects had been dormant a couple of years while the Crystal language has been updated regularly.
LMDB is kind of magical. It's pretty good at handling concurrent writes (it's properly ACIDic), though it doesn't deliver the transaction performance of high-end database engines. But being memory-mapped and copy-free it is phenomenally fast at reads. I ran some tests in Python a while back reading 1GB per second on a small VPS.
With LMDB it doesn't matter that your language has a GIL or uses green threads, because you can just run multiple processes. It handles transactions and crash recovery for you, the database is a single file, and it comes with a hot backup utility where the backups are themselves LMDB databases. That is, restore time once you have a backup is zero.
This makes a project I've wanted to do for years suddenly very, very feasible.
-
TSMC has started risk production of 5nm devices. (WikiChip Fuse)
This extends the use of EUV throughout the production process and significantly shrinks the geometry, packing about 80% more transistors per square millimetre. Performance and power improvements are expected to be smaller, on the order of
10%.
- Browsers are increasingly disabling disabling click tracking. (Bleeping Computer)
Except for Firefox among the major players. Brave also still allows this, and it's likely that MoonPie and Vivaldi and other niche browsers do too.
I didn't know that the ping attribute was at thing. That's really useful and much better than using JavaScript to achieve the same end, but users should be able to turn it off.
- Does Apple's "Magic Keyboard" still suck? Yes. (ZDNet)
Video of the Day
Just wait until he starts explaining how to construct five dimensional polytopes out of four dimensional ones. He even has a model.
Bonus Video of the Day
Posted by: Pixy Misa at
11:42 PM
| Comments (7)
| Add Comment
| Trackbacks (Suck)
Post contains 880 words, total size 8 kb.
Saturday, April 06
Crystal Ball Edition
Tech News
- I bought and read Programming Crystal last night and sat down this afternoon and wrote the little app I needed.
A bit of fiddling and refactoring and reading the docs for the Crystal standard library (which could do with some fleshing out) and it works. The code is clean, it compiles quickly, the binary is portable, and it's only 6 megabytes.
Okay, that last part isn't so great; as soon as I included the HTTP client library it yanked in 3.3MB of extra code. But that's what static linking does for you.
Don't concatenate large strings in a loop, though. Python handles that very well because it reserves space at the end of each string and can reuse the memory. Most languages don't do that trick and allocate memory and copy the entire string every time, so appending one byte to a one megabyte string is contraindicated.
This is something you have to watch out for just going from CPython to PyPy with the same code, so it's something I check with every language.
Oh yes. As to the IDE question, while there is no convenient CrystalMine from Jetbrains, and the plugin that is available is meh, if you install VSCode and Windows Services for Linux and Ubuntu 18.04 under WSL and Crystal under Ubuntu 18.04 under WSL and the Crystal Language extension for VSCode (the one by Faustino Aguilar) and the Crystal Installer for Windows which does not install Crystal on Windows because Crystal doesn't run on Windows bu instead creates a .bat file pointing to the Crystal on Ubuntu on WSL and puts the .bat file in your path... It mostly works.
I also gave the Amber web framework a try. After some initial swearing when I was editing the wrong copy of my source file, it basically worked. I've looked at innumerable web frameworks over the last decade and this one is easy to pick up. It compiles down to a portable binary (if you want). But it does use the Code of Cancer. Ugh.
That crap aside, it's a lot faster than CherryPy, which I have been using for years, even CherryPy running under PyPy. I'd previously tested Swift with the Vapor framework and it wasn't much faster than CherryPy under PyPy, certainly not enough to justify a switch of programming languages.
Responding to 50,000 simple queries takes around 70 CPU seconds with CherryPy on CPython, 20 seconds on PyPy, and 1.5 seconds with Amber on Crystal 0.27. That's a big difference. (This is all on a small VPS with a single server process and default configuration.)
This bears some thinking about.
- PostgreSQL is an enormously powerful SQL database built by perfectionists and designed by idiots that tells you to go fuck yourself at every possible occasion.
- Optane DIMMs cost about half as much as RAM. (AnandTech)
A 128GB Optane DIMM costs around $900, the price of a 64GB DDR4 module, and a 256GB Optane DIMM at $2850 is about the price of a 128GB DDR4 module. (Higher density modules are proportionally more expensive.)
On the other hand, you can only use Optane DIMMs with very expensive CPUs, and they are significantly slower than DRAM (though still lightning fast compared to SSDs.) So the market for these at the moment seems small.
- "Ironically" too many video streaming services is leading to an increase in piracy. (TechDirt)
That's not irony, you idiots, that's arithmetic.All told, there are more than 300 over-the-top video options in the U.S.
-
Lessons learned from porting 50,000 lines of code from Java to Go.
From the publisher of such titles as A Walking Tour of the Sewers of Major European Cities and Leaving Ethiopia for Somalia.
-
A one-day bug bounty program run by Dropbox found 264 vulnerabilities. (ZDNet)
Uh, what?
-
A misconfigured Qt5 app and a malicious link can ruin your whole day. (Bleeping Computer)
-
A sneaky hacker slipping a backdoor into a popular library can ruin your whole day. (ZDNet)
Fortunately this one was caught and removed the same day it appeared.
Social Media News
- When all the news is bad maybe it's time to buy a newspaper. (Business Insider)
It almost worked for Jeff Bezos.
5K 360° Video of the Day
For those of you with iMacs and gigabit internet access.
Posted by: Pixy Misa at
08:08 PM
| Comments (4)
| Add Comment
| Trackbacks (Suck)
Post contains 741 words, total size 7 kb.
57 queries taking 0.3064 seconds, 399 records returned.
Powered by Minx 1.1.6c-pink.