It was a bad day. A lot of bad stuff happened. And I'd love to forget it all. But I don't. Not ever. Because this is what I do. Every time, every day, every second, this: On five, we're bringing down the government.
Friday, July 03
And I've Got WTFF News
Akane, the new super-server that hosts everything mu.nu and mee.nu in over a dozen virtual machine, just died for no apparent reason.
Yeah, we're back. Because I was already awake at 4:30AM because I'd been up all night fixing other problems.
Had to do an index repair on the main posts table; other than that we seem to be okay.
Update: Guess what the cause of the crash was? Go on, guess! You'll never guess!
Oh, you guessed.
Update: It gets better. Apparently I need to back up all my data, rebuild the array, and restore everything again. Why the hell do I have a RAID controller in the first place?
Update: Okay, I think I've got it. The system locked up and crashed, apaprently due to a bad disk. Now it has inconsistent data between the data drives and the parity drive for some part of the secondary volume. That may or may not be a serious problem, and there's no way to tell unless you run the fix process. The fix process may involve completely wiping and rebuilding the array.
So no matter what, I need a complete copy of everything on another server. Which is not a bad thing to have anyway, but it is sixteen million files.
Update: 700,000 down, 15.3 million to go. I have to back up sixteen complete virtual machines, plus the backup directory, plus the mee.nu filestore.
Update: 2,700,000 done. There are 5.7 million files across the virtual machines, and another 9.7 million in the backup volume. But the backup volume doesn't *have* to be backed up, because it's a backup.
So, halfway.
I'll schedule the RAID repair for this weekend, so that if it does end up taking us out, I have the maximum time to fix things.
Akane, the new super-server that hosts everything mu.nu and mee.nu in over a dozen virtual machine, just died for no apparent reason.
Yeah, we're back. Because I was already awake at 4:30AM because I'd been up all night fixing other problems.
Had to do an index repair on the main posts table; other than that we seem to be okay.
Update: Guess what the cause of the crash was? Go on, guess! You'll never guess!
Oh, you guessed.
Update: It gets better. Apparently I need to back up all my data, rebuild the array, and restore everything again. Why the hell do I have a RAID controller in the first place?
Update: Okay, I think I've got it. The system locked up and crashed, apaprently due to a bad disk. Now it has inconsistent data between the data drives and the parity drive for some part of the secondary volume. That may or may not be a serious problem, and there's no way to tell unless you run the fix process. The fix process may involve completely wiping and rebuilding the array.
So no matter what, I need a complete copy of everything on another server. Which is not a bad thing to have anyway, but it is sixteen million files.
Update: 700,000 down, 15.3 million to go. I have to back up sixteen complete virtual machines, plus the backup directory, plus the mee.nu filestore.
Update: 2,700,000 done. There are 5.7 million files across the virtual machines, and another 9.7 million in the backup volume. But the backup volume doesn't *have* to be backed up, because it's a backup.
So, halfway.
I'll schedule the RAID repair for this weekend, so that if it does end up taking us out, I have the maximum time to fix things.
Posted by: Pixy Misa at
05:10 AM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 329 words, total size 2 kb.
Monday, June 29
I've Got WTF News, And I've Got Burned Too Many Times News
The other serverâ„¢ blew up again. I was meaning to transfer the two remaining sites on that server to the new system over the weekend, but I ended up having to work Friday night and Saturday, and then I was too tired to focus on server stuff and watched Gurren Lagann instead.
So, naturally, it died the very next day.
That's the WTF part.
On the burned too many times side of things, however, I have a full backup from an hour before the server died.
In fact, I have two full backups from an hour before the server died. Plus a full backup from the previous day, and another from the day before that, and one from last week, and from the previous week, and... Yeah. Burned too many times.
Update: Drive failure. That makes two in what, three months? In a server that only has two drives in the first place.
They're 500GB Western Digital drives. I don't have any 500GB WD drives... Oh, wait, I do have one, and it sucks, but that's the external enclosure, not the drive itself. But I have, um, 14 other Western Digital drives in use at home (is that a lot?) bought over the last three years, and not one of them has failed. So... Bad batch? Insufficient airflow? Cosmic rays? Evil leprechauns? Karma? Or just the hard drive destruction bunny spreading the joy around? I dunno.
The other serverâ„¢ blew up again. I was meaning to transfer the two remaining sites on that server to the new system over the weekend, but I ended up having to work Friday night and Saturday, and then I was too tired to focus on server stuff and watched Gurren Lagann instead.
So, naturally, it died the very next day.
That's the WTF part.
On the burned too many times side of things, however, I have a full backup from an hour before the server died.
In fact, I have two full backups from an hour before the server died. Plus a full backup from the previous day, and another from the day before that, and one from last week, and from the previous week, and... Yeah. Burned too many times.
Update: Drive failure. That makes two in what, three months? In a server that only has two drives in the first place.
They're 500GB Western Digital drives. I don't have any 500GB WD drives... Oh, wait, I do have one, and it sucks, but that's the external enclosure, not the drive itself. But I have, um, 14 other Western Digital drives in use at home (is that a lot?) bought over the last three years, and not one of them has failed. So... Bad batch? Insufficient airflow? Cosmic rays? Evil leprechauns? Karma? Or just the hard drive destruction bunny spreading the joy around? I dunno.
Posted by: Pixy Misa at
09:41 PM
| Comments (6)
| Add Comment
| Trackbacks (Suck)
Post contains 248 words, total size 2 kb.
Saturday, April 04
No Poo
Server status: PE_INVENTORYCHECK_FAILURE
Guess I won't have my new server today.
Hmm. Hang on. It also says it's already configured with 2 CPUs, 12 DIMMS, 3 disks and 1 SSD. So what inventory check did it fail?
Update: Okay, it was just some software inventory issue. The automated hardware burn-in test is running now.
Update: So I sent an email off to sales to check up on a couple of things, and they managed to sound like they answered all of my questions without actually answering any of them. Yeah, the natural way to read the response is that the answer is yes, but it doesn't actually say that, it answers subtly different and completely unhelpful questions instead.
It's annoying and kind of embarassing to have to ask the same questions again to try to elicit an apropos response, but if I don't sort it out now it could be a major pain in the future.
Server status: PE_INVENTORYCHECK_FAILURE
Guess I won't have my new server today.
Hmm. Hang on. It also says it's already configured with 2 CPUs, 12 DIMMS, 3 disks and 1 SSD. So what inventory check did it fail?
Update: Okay, it was just some software inventory issue. The automated hardware burn-in test is running now.
Update: So I sent an email off to sales to check up on a couple of things, and they managed to sound like they answered all of my questions without actually answering any of them. Yeah, the natural way to read the response is that the answer is yes, but it doesn't actually say that, it answers subtly different and completely unhelpful questions instead.
It's annoying and kind of embarassing to have to ask the same questions again to try to elicit an apropos response, but if I don't sort it out now it could be a major pain in the future.
Posted by: Pixy Misa at
11:38 AM
| Comments (2)
| Add Comment
| Trackbacks (Suck)
Post contains 159 words, total size 1 kb.
What The Hell, Microsoft?
I was just installing the drivers for my colour laser (nice internet connectivity there, HP - 7kb/sec?) so I could print out the receipt for my shiny new server and my E drive disappeared. Not in Explorer, not in Storage Manager, just gone.
Out of all of my drives, that would be best one to lose, but that doesn't mean I want to lose it. But if something weird happens when you install a driver, 99% of the time it will be Windows or the driver at fault, and so it was.
But still - how the hell can a printer driver disable a disk drive, Microsoft?
I was just installing the drivers for my colour laser (nice internet connectivity there, HP - 7kb/sec?) so I could print out the receipt for my shiny new server and my E drive disappeared. Not in Explorer, not in Storage Manager, just gone.
Out of all of my drives, that would be best one to lose, but that doesn't mean I want to lose it. But if something weird happens when you install a driver, 99% of the time it will be Windows or the driver at fault, and so it was.
But still - how the hell can a printer driver disable a disk drive, Microsoft?
Posted by: Pixy Misa at
11:19 AM
| Comments (1)
| Add Comment
| Trackbacks (Suck)
Post contains 111 words, total size 1 kb.
Tuesday, March 31
Well, Poop, Part 37 Or Something
The new Xeon 5500 dual-processor Nehalem has been announced.
So too - if you look hard enough - the single-processor Xeon 3500.
SuperMicro (and SoftLayer exclusively uses SuperMicro) have announced over 30 new Xeon 5500 motherboards.
And no new Xeon 3500 boards. Meaning that all they have are their existing Core i7 boards, none of which support registered memory, so they're limited to 12GB.
Which means that to go beyond 12GB I'll still need to pay the dual-processor-motherboard tax of at least $100 per month per server - and then pay for the extra memory.
And the dual-processor CPUs themselves are substantially more expensive than the single-processor equivalents.
Dammit, SuperMicro, pull your finger out. Even Gigabyte managed to get this one right, and they hardly have a presence in the server market at all.
The new Xeon 5500 dual-processor Nehalem has been announced.
So too - if you look hard enough - the single-processor Xeon 3500.
SuperMicro (and SoftLayer exclusively uses SuperMicro) have announced over 30 new Xeon 5500 motherboards.
And no new Xeon 3500 boards. Meaning that all they have are their existing Core i7 boards, none of which support registered memory, so they're limited to 12GB.
Which means that to go beyond 12GB I'll still need to pay the dual-processor-motherboard tax of at least $100 per month per server - and then pay for the extra memory.
And the dual-processor CPUs themselves are substantially more expensive than the single-processor equivalents.
Dammit, SuperMicro, pull your finger out. Even Gigabyte managed to get this one right, and they hardly have a presence in the server market at all.
Posted by: Pixy Misa at
08:44 AM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 140 words, total size 1 kb.
Friday, March 20
History Is Accelerating: Now The First Time Is Farce
Or, This Post No Links
So, our present and bountifully incompetent federal government is bent on censoring this internet thingy, though it appears that they have yet to work out quite what it is.
The ACMA (which apparently stands for "petty-minded bureaucrats") has a list of a thousand-odd sites which are banned in Australia.
This list is secret.
Also, it's just a list. The sites are banned, and viewing them is illegal, and linking to them incurs an $11,000-a-day fine, but no-one is allowed to know what is on the list, and the banned sites aren't actually blocked in any way.
So when, for example, popular Australian technical forum and news site Whirlpool found itself featuring a comment linking to a banned site, it also found itself facing a huge fine, and understandably removed the comment.
This even though the site in question has not been shown to be in breach of any law or regulation - except, perhaps, for the wonderfully vague clause other material dealing with intense adult themes, something that would be struck down as unconstituionally broad inside of thirty seconds by the US Supreme Court, bless 'em - and that the details of the site were made public by the original complainant, which is also not a breach of any law or regulation.
The secrecy of the list must be maintained, regardless. And for that reason I can't link to Wikipedia any more either, because the Wikipedia article regarding the ACMA itself contains a link to a banned website. (And was the subject of a 24-hour revert war as the inevitable result.)
We know some of what is on the list, of course, because after Wikileaks published similar secret censorship lists from other countries (Denmark, Finland, and Thailand, to be precise) Wikileaks was itself banned, and so was linking to the relevant pages on Wikileaks. So I can't link to them.
Indeed, now we know all of what is on the list, because the list has been leaked to the banned Wikileaks. Which, as I say, I cannot link to.
It gets better, though.
The leaked list (which, of course, I haven't seen, and won't reproduce, because to do so is to face ten years in jail, never mind the fines) reportedly contains over two thousand banned sites, some of which are reportedly legitimately illegitimate; others including tour operators, religious sites, online gambling sites, and one unfortunate dentist. None of which, of course, can I link to.
I gets better still.
The Minister for Communication, the estimable Stephen Conroy, claims that the leaking of the list is irresponsible, illegal, and inaccurate - that it is not the real thing at all.
Wikileaks is still banned, of course, and it is still illegal for me to link to or reproduce the not-the-real-list.
So I won't.
I'll just link to the ABC, The Courier-Mail, The Brisbane Times, IT News, news.com.au, The Australian, Computerworld (twice), The Sydney Morning Herald, The Register, Slashdot, Forbes, Wired, ZDnet, Gizmodo, CNet, Ars Technica, and Google News.
Or, This Post No Links
So, our present and bountifully incompetent federal government is bent on censoring this internet thingy, though it appears that they have yet to work out quite what it is.
The ACMA (which apparently stands for "petty-minded bureaucrats") has a list of a thousand-odd sites which are banned in Australia.
This list is secret.
Also, it's just a list. The sites are banned, and viewing them is illegal, and linking to them incurs an $11,000-a-day fine, but no-one is allowed to know what is on the list, and the banned sites aren't actually blocked in any way.
So when, for example, popular Australian technical forum and news site Whirlpool found itself featuring a comment linking to a banned site, it also found itself facing a huge fine, and understandably removed the comment.
This even though the site in question has not been shown to be in breach of any law or regulation - except, perhaps, for the wonderfully vague clause other material dealing with intense adult themes, something that would be struck down as unconstituionally broad inside of thirty seconds by the US Supreme Court, bless 'em - and that the details of the site were made public by the original complainant, which is also not a breach of any law or regulation.
The secrecy of the list must be maintained, regardless. And for that reason I can't link to Wikipedia any more either, because the Wikipedia article regarding the ACMA itself contains a link to a banned website. (And was the subject of a 24-hour revert war as the inevitable result.)
We know some of what is on the list, of course, because after Wikileaks published similar secret censorship lists from other countries (Denmark, Finland, and Thailand, to be precise) Wikileaks was itself banned, and so was linking to the relevant pages on Wikileaks. So I can't link to them.
Indeed, now we know all of what is on the list, because the list has been leaked to the banned Wikileaks. Which, as I say, I cannot link to.
It gets better, though.
The leaked list (which, of course, I haven't seen, and won't reproduce, because to do so is to face ten years in jail, never mind the fines) reportedly contains over two thousand banned sites, some of which are reportedly legitimately illegitimate; others including tour operators, religious sites, online gambling sites, and one unfortunate dentist. None of which, of course, can I link to.
I gets better still.
The Minister for Communication, the estimable Stephen Conroy, claims that the leaking of the list is irresponsible, illegal, and inaccurate - that it is not the real thing at all.
Wikileaks is still banned, of course, and it is still illegal for me to link to or reproduce the not-the-real-list.
So I won't.
I'll just link to the ABC, The Courier-Mail, The Brisbane Times, IT News, news.com.au, The Australian, Computerworld (twice), The Sydney Morning Herald, The Register, Slashdot, Forbes, Wired, ZDnet, Gizmodo, CNet, Ars Technica, and Google News.
Posted by: Pixy Misa at
12:10 AM
| Comments (7)
| Add Comment
| Trackbacks (Suck)
Post contains 512 words, total size 7 kb.
Tuesday, March 10
Ugh
Ugh.
Ugh.
Posted by: Pixy Misa at
09:09 PM
| Comments (2)
| Add Comment
| Trackbacks (Suck)
Post contains 2 words, total size 1 kb.
Tuesday, February 10
Impeach Obama
They bumped House for an address from the Campaigner-in-Chief?
They bumped House for an address from the Campaigner-in-Chief?
Posted by: Pixy Misa at
08:19 PM
| Comments (6)
| Add Comment
| Trackbacks (Suck)
Post contains 11 words, total size 1 kb.
Saturday, January 31
Stupid Apache
If you're going to die with a fatal error, at least print out the damn error message!!
Nearly six hours of downtime over a problem I could have fixed in thirty seconds if it had just told me what was wrong.
If you're going to die with a fatal error, at least print out the damn error message!!
Nearly six hours of downtime over a problem I could have fixed in thirty seconds if it had just told me what was wrong.
Posted by: Pixy Misa at
05:39 AM
| Comments (11)
| Add Comment
| Trackbacks (Suck)
Post contains 44 words, total size 1 kb.
Sunday, January 25
Western Digital, You Dickheads!
Okay, it's not as bad as Seagate's brick-in-a-box trick, but it's plenty annoying.
Western Digital have three lines of desktop drives: Black, the performance model; Blue, the everday model; and Green, the low-power model.
The Green range run at 5400RPM.
Western Digital not only do not advertise this fact, they go to great lengths to hide it. The speed of the Green range of drives is not stated anywhere on their website.
That leads to retailers assuming that the drives are, in fact, 7200RPM.
That leads to people buying them as replacements for actual 7200RPM drives, installing them, building them into RAID arrays, and then having to manually unpick their whole filesystem/logical volume/volume group/physical volume/RAID volume structure so that the 5400RPM drive isn't mirrored with the 7200RPM drive for the database partition.
I sent them a nastygram, you bet.
And another thing: It's 2008. Sun had a graphical tool to manage all that crap in 1998. Why doesn't Linux? (Update: Maybe YAST can do it. If I can just get YAST to work...)
That said, all the RAID/LVM stuff is at least robust. I pulled it apart and put it back together using the command line tools, and it all worked. Didn't need to change mdadm.conf or fstab at all. Mind you, that's because I got it right, but if you get it right it works. Unlike certain things I could mention...
Okay, it's not as bad as Seagate's brick-in-a-box trick, but it's plenty annoying.
Western Digital have three lines of desktop drives: Black, the performance model; Blue, the everday model; and Green, the low-power model.
The Green range run at 5400RPM.
Western Digital not only do not advertise this fact, they go to great lengths to hide it. The speed of the Green range of drives is not stated anywhere on their website.
That leads to retailers assuming that the drives are, in fact, 7200RPM.
That leads to people buying them as replacements for actual 7200RPM drives, installing them, building them into RAID arrays, and then having to manually unpick their whole filesystem/logical volume/volume group/physical volume/RAID volume structure so that the 5400RPM drive isn't mirrored with the 7200RPM drive for the database partition.
I sent them a nastygram, you bet.
And another thing: It's 2008. Sun had a graphical tool to manage all that crap in 1998. Why doesn't Linux? (Update: Maybe YAST can do it. If I can just get YAST to work...)
That said, all the RAID/LVM stuff is at least robust. I pulled it apart and put it back together using the command line tools, and it all worked. Didn't need to change mdadm.conf or fstab at all. Mind you, that's because I got it right, but if you get it right it works. Unlike certain things I could mention...
Posted by: Pixy Misa at
08:28 PM
| Comments (4)
| Add Comment
| Trackbacks (Suck)
Post contains 236 words, total size 2 kb.
77kb generated in CPU 0.0681, elapsed 0.175 seconds.
57 queries taking 0.1595 seconds, 262 records returned.
Powered by Minx 1.1.6c-pink.
57 queries taking 0.1595 seconds, 262 records returned.
Powered by Minx 1.1.6c-pink.
