Wednesday, April 06


Daily News Stuff 6 April 2022

The Load Average Is Over 9000 Edition

Top Story

  • Waiting for house news. 

    Update: There's a buyer for this one as well so I've upped their offer by $25k.  I'll be cross if this one gets away because I've already set up a shopping list at Ikea.

    Update: And now I'm probably going to be cross.  The other buyers lost their house in the floods, and they've already paid the deposit, so unless they can't exchange contracts tomorrow the owners are going to take their offer.

    Also, apologies are due to unknown real-estate agent #1 who I thought was playing games.  That property - which had been on the market for quite a while - has sold.

  • Had a report that one of my (virtual) servers was port scanning someone.  I'm not sure if it was legit, and a scan of the server showed nothing untoward, but I have to take it seriously.

    First step was to block unwanted outbound traffic at the firewall.  The first thing I do with any server is to block unwanted inbound traffic, but everyone* does that these days so all the major exploits sneak unwanted data in on trusted connection. - like the huge Log4j mess a few weeks ago.

    Easy peasy because if you screw up the firewall settings on a virtual server you can get in via the virtual console.  I've done this on physical servers on the other side of the planet, with complex networking arrangements and no console, and that is nerve-wracking.

    Next step was to rebuild that server entirely since all it runs is a proxy server (Caddy) and it's needed an update for a while.  Before doing that I checked on the backups to make sure if anything went wrong I could easily restore and, well, f*ck.

    The backup drive on the backup server is not responding at all.  The syslog is full of ZFS deadman events.  And the load average is a personal record, and I've worked on some pretty big servers.

    That is not a happy 10-core bunny.

    The backups run A->B->C where A is the active server and C is a an archive server that can collect lots of daily backups and B is broken.  Which means that for, uh, ten weeks, no backups were going off site from that origin server to anywhere.

    So, I arranged three off-site backups and then rebuilt the proxy server with the latest software and the new firewall rules and basically tweaked the config file at random and kept restarting it until it all worked.  I don't know why that was necessary, but it was.  At least I could flick the routes back and forth internally and didn't need to wait for DNS to update.

    Then back to the backup server.  Look at the I/O stats.  It's written how much data?  18 petabytes?  No wonder it's not working, the SSD must be fried.

    Wait...  Insert commas manually.  18 terabytes.  That's not much at all.

    Kill the stuck processes (all 24,000 of them).  Load average goes up.  Kill the parent process...  Load goes up even more.

    Guess it's reboot time.

    And...  It sits there with ping working but no other sign of life for ten minutes.  (This one doesn't have a remote console either.  I grabbed it during the datacenter fire last year and took what I could get.)

    It does have a button to remotely power cycle the server but the page asks you not to do that if the server can ping because they'd rather have a technician look at it before the evidence of whatever the problem was disappears.

    So I start writing a tech support ticket and just as I'm about to submit it the server comes back up and is working perfectly as if there was never anything wrong.


    How was your evening?

    * Not everyone.

  • Twitter is adding an edit button, maybe.  (The Verge)

    "Protecting the integrity of that public conversation" is of course Newspeak for eradicating dissent.

Tech News

Disclaimer: No, we are not having fun yet.

Posted by: Pixy Misa at 02:03 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 814 words, total size 7 kb.

Apple pies are delicious. But never mind apple pies. What colour is a green orange?

53kb generated in CPU 0.0212, elapsed 0.8122 seconds.
56 queries taking 0.8027 seconds, 340 records returned.
Powered by Minx 1.1.6c-pink.