Wednesday, April 06
Daily News Stuff 6 April 2022
The Load Average Is Over 9000 Edition
The Load Average Is Over 9000 Edition
Top Story
- Waiting for house news.
Update: There's a buyer for this one as well so I've upped their offer by $25k. I'll be cross if this one gets away because I've already set up a shopping list at Ikea.
Update: And now I'm probably going to be cross. The other buyers lost their house in the floods, and they've already paid the deposit, so unless they can't exchange contracts tomorrow the owners are going to take their offer.
Also, apologies are due to unknown real-estate agent #1 who I thought was playing games. That property - which had been on the market for quite a while - has sold.
- Had a report that one of my (virtual) servers was port scanning someone. I'm not sure if it was legit, and a scan of the server showed nothing untoward, but I have to take it seriously.
First step was to block unwanted outbound traffic at the firewall. The first thing I do with any server is to block unwanted inbound traffic, but everyone* does that these days so all the major exploits sneak unwanted data in on trusted connection. - like the huge Log4j mess a few weeks ago.
Easy peasy because if you screw up the firewall settings on a virtual server you can get in via the virtual console. I've done this on physical servers on the other side of the planet, with complex networking arrangements and no console, and that is nerve-wracking.
Next step was to rebuild that server entirely since all it runs is a proxy server (Caddy) and it's needed an update for a while. Before doing that I checked on the backups to make sure if anything went wrong I could easily restore and, well, f*ck.
The backup drive on the backup server is not responding at all. The syslog is full of ZFS deadman events. And the load average is a personal record, and I've worked on some pretty big servers.
That is not a happy 10-core bunny.
The backups run A->B->C where A is the active server and C is a an archive server that can collect lots of daily backups and B is broken. Which means that for, uh, ten weeks, no backups were going off site from that origin server to anywhere.
So, I arranged three off-site backups and then rebuilt the proxy server with the latest software and the new firewall rules and basically tweaked the config file at random and kept restarting it until it all worked. I don't know why that was necessary, but it was. At least I could flick the routes back and forth internally and didn't need to wait for DNS to update.
Then back to the backup server. Look at the I/O stats. It's written how much data? 18 petabytes? No wonder it's not working, the SSD must be fried.
Wait... Insert commas manually. 18 terabytes. That's not much at all.
Kill the stuck processes (all 24,000 of them). Load average goes up. Kill the parent process... Load goes up even more.
Guess it's reboot time.
And... It sits there with ping working but no other sign of life for ten minutes. (This one doesn't have a remote console either. I grabbed it during the datacenter fire last year and took what I could get.)
It does have a button to remotely power cycle the server but the page asks you not to do that if the server can ping because they'd rather have a technician look at it before the evidence of whatever the problem was disappears.
So I start writing a tech support ticket and just as I'm about to submit it the server comes back up and is working perfectly as if there was never anything wrong.
Okay.
How was your evening?
* Not everyone.
- Twitter is adding an edit button, maybe. (The Verge)
"Protecting the integrity of that public conversation" is of course Newspeak for eradicating dissent.
Tech News
- Over $50 billion worth of video cards were sold in 2021. (Tom's Hardware)
That's up 30% by value and 20% from 2020, and with 50 million cards sold last year that means the average retail price was $1000.
Which is far too much.
- The US government - which plans to replace its vehicle fleet with electric models - has nearly 1% of the charging stations it needs to do that. (Tech Crunch)
Sounds about right.
- Elon Musk is joining the board of directors of Twitter. (CNBC)
That's going to put the honey badger among the stool pigeons.
- Fire Toolbox turns your Kindle Fire into an Android tablet. (Liliputing)
Ish. Some things remain locked down - like the app launcher, which can't be entirely replaced - but it provides a lot of customisation options.
Disclaimer: No, we are not having fun yet.
Posted by: Pixy Misa at
02:03 PM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 814 words, total size 7 kb.
53kb generated in CPU 0.013, elapsed 0.0984 seconds.
56 queries taking 0.0902 seconds, 338 records returned.
Powered by Minx 1.1.6c-pink.
56 queries taking 0.0902 seconds, 338 records returned.
Powered by Minx 1.1.6c-pink.
