Wednesday, December 18

Geek

Daily News Stuff 18 December 2024

Orders Of Magnitude Edition

Top Story

  • What happened to ChatGPT the other day?  It wasn't DNS, but it wasn't not DNS either.  (Try Parity)

    OpenAI deployed a new system that had been running happily on their test servers for some time.  About twenty minutes in, things started going horribly wrong, with the monitoring system that reports on all things production taking over and then taking out the production servers, because it had a hidden scaling problem that never showed up on the smaller test environment.

    And with the monitoring process eating up all the bandwidth on the control network, the simple changes they needed to make to fix the problem couldn't be done because OpenAI's internal DNS was down.

    Lesson of the day: Hard code all your IP addresses in your software.*


Tech News



Disclaimer: * Do not actually do this.

Posted by: Pixy Misa at 06:08 PM | Comments (4) | Add Comment | Trackbacks (Suck)
Post contains 347 words, total size 3 kb.

1 At one of my previous jobs, we had all the server names and their IP addresses in a database to avoid DNS issues. And then we found that in reality, 99% of our DNS issues prevented us from reaching the database.

Posted by: David Eastman at Thursday, December 19 2024 02:45 AM (aAyxl)

2 (obligatory "web-scale" video link)

Posted by: J Greely at Thursday, December 19 2024 03:35 AM (oJgNG)

3 Place I used to work at had one last test site that was full production sized, just to prevent this type of thing from happening.

Posted by: Frank at Thursday, December 19 2024 04:15 AM (+i6Xr)

4 A software change providing a speed increase doesn't surprise me. I've worked with processors that had programmable internal system clocks, as well as processors with configurable memory interfaces that included a programmable number of wait states added to memory accesses. 
As an extreme example of companies deliberately limiting performance, back in the early 1980s, I used to work for a company that made dedicated word processors with Motorola 6800-based terminals and Intel 8086-based central servers. These were sold with several available speeds, and when a customer ordered a speed upgrade, a technician would go to the customer site and remove one or more boards that inserted wait states onto the memory bus.

Posted by: wheels at Thursday, December 19 2024 07:31 AM (IB9dr)

Hide Comments | Add Comment




Apple pies are delicious. But never mind apple pies. What colour is a green orange?




52kb generated in CPU 0.026, elapsed 0.1284 seconds.
58 queries taking 0.1192 seconds, 351 records returned.
Powered by Minx 1.1.6c-pink.