Monday, May 27
We're Back
Well, at least it wasn't a fire.
Well, at least it wasn't a fire.
At roughly 6:20am EST the facility where [name of hosting company] operates its DAL1 data center ... lost utility power. Redundant power sources, UPS and generators, did not operate as designed thus causing the entire facility to lose power.A few hours later:
All [name of hosting company] servers and infrastructure now have power fully restored. If you are experiencing any issues with your server(s), please open a support ticket so that we can troubleshoot the problem.This server needed some manual intervention before the blogs were accessible again.
Posted by: Pixy Misa at
08:21 AM
| Comments (2)
| Add Comment
| Trackbacks (Suck)
Post contains 86 words, total size 1 kb.
1
It's amazing how frequently data centers and hosting companies are failing at what are supposed to be their competencies the last few years. Redundant backups not actually being redundant or even not actually existing. Power outages taking down facilities that supposedly had multiple redundant power sources. Configuration errors that are propagated simultaneously to primary and secondary redundant sites taking them both down...
My company is still in the process of transitioning from a combo of locally hosted and off site hosted but self managed servers to the Microsoft cloud. In the roughly two years since we spun up our first apps, we've been hit by three total outages, one of which took three days to recover from. Our IT director had to stand in front of the board and defend continuing the Azure migration, and his defense basically boiled down to "our current hosting company is going to cease operations next quarter, and as bad as Azure is proving to be, it's more reliable than any of the other options we've got available."
My company is still in the process of transitioning from a combo of locally hosted and off site hosted but self managed servers to the Microsoft cloud. In the roughly two years since we spun up our first apps, we've been hit by three total outages, one of which took three days to recover from. Our IT director had to stand in front of the board and defend continuing the Azure migration, and his defense basically boiled down to "our current hosting company is going to cease operations next quarter, and as bad as Azure is proving to be, it's more reliable than any of the other options we've got available."
Posted by: David Eastman at Monday, May 27 2024 02:32 PM (rmrII)
2
Data centers. Boeing. Etc.
I'm about ninety percent sure that a lot of seemingly unusual patterns of industrial safety/reliability issues are downstream of shutting down as much of the economy as could be shut down.
Industrial quality, safety, and reliability processes are kinda on top of having your existing process going. There's a baseline you need, that you lose when you are not operating, or when you fire or drive off or alienate or demoralize the people that make up your institutional memory. There's some margin that you need, above the bare minimums, to avoid the combinations of oversights that cause incidents.
There is also diversity hiring, hiring managemetn from universities, and the like. Which is probably weaker. Yes, some of those people are crazier and less constrained than they were before. But, we have had that sort of crazy all along, but had more functional manpower willing and able to route around.
'Essential' means screwing over everyone, but the geniuses at the universities think that they are safe where the first order effects are concerned. This is mostly hirign and promoting idiots, because there are non-specialist thinkers at universities who can manage to tie their own shoes, and see that the idiot leadership have obviously screwed up.
Some things are also outright malice. (The Russians were planning some degree of the Ukraine war for a while. (It was not something that Biden forced them into. Biden was a weak idiot, and provided the opportunity. The Russians were stupid and crazy enough to think that they could not pass up the opportunity. Now they feel like they were outsmarted.) The pipeline hack was the Russian government. It was a strategic operation in support of a war in Europe. The lock down, our idiots, gave them the opportunity for the hack.)
I'm about ninety percent sure that a lot of seemingly unusual patterns of industrial safety/reliability issues are downstream of shutting down as much of the economy as could be shut down.
Industrial quality, safety, and reliability processes are kinda on top of having your existing process going. There's a baseline you need, that you lose when you are not operating, or when you fire or drive off or alienate or demoralize the people that make up your institutional memory. There's some margin that you need, above the bare minimums, to avoid the combinations of oversights that cause incidents.
There is also diversity hiring, hiring managemetn from universities, and the like. Which is probably weaker. Yes, some of those people are crazier and less constrained than they were before. But, we have had that sort of crazy all along, but had more functional manpower willing and able to route around.
'Essential' means screwing over everyone, but the geniuses at the universities think that they are safe where the first order effects are concerned. This is mostly hirign and promoting idiots, because there are non-specialist thinkers at universities who can manage to tie their own shoes, and see that the idiot leadership have obviously screwed up.
Some things are also outright malice. (The Russians were planning some degree of the Ukraine war for a while. (It was not something that Biden forced them into. Biden was a weak idiot, and provided the opportunity. The Russians were stupid and crazy enough to think that they could not pass up the opportunity. Now they feel like they were outsmarted.) The pipeline hack was the Russian government. It was a strategic operation in support of a war in Europe. The lock down, our idiots, gave them the opportunity for the hack.)
Posted by: PatBuckman at Monday, May 27 2024 04:36 PM (rcPLc)
50kb generated in CPU 0.0206, elapsed 0.4503 seconds.
58 queries taking 0.4433 seconds, 347 records returned.
Powered by Minx 1.1.6c-pink.
58 queries taking 0.4433 seconds, 347 records returned.
Powered by Minx 1.1.6c-pink.