No boom today. Boom tomorrow. There's always a boom tomorrow.
Sunday, May 31
I Think It May Just Be Time For A Reboot
18:25:45 up 518 days, 12:28
I've had servers run that long before, or nearly so, but that's not bad. But it is starting to show some odd errors - lost database connections, errors starting threads, that sort of thing - that just haven't happened before.
If it gets any worse I'll reboot the server, but first I'll see about getting everything set up on Akane.
Posted by: Pixy Misa at
04:28 AM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 75 words, total size 1 kb.
Microsoft have a program called BizSpark that offers free development tools and server software to small web startups.
I'm a small web startup. I'm about as small a web startup as you can find. And while I'm not sure I want to move away from Python, I could use IronPython and have access to all the other Visual Studio languages - so I could easily import modules written in C# or IronRuby or whatever.
And SQL Server 2008 has some attractive advantages over MySQL, and can be licensed quite cheaply for web applications.
But I wasn't going to spend a lot of time, effort, and money testing out the Microsoft platform when open source was working pretty well. Microsoft knows this, so they set up BizSpark to remove one of those factors completely.
SoftLayer is now a BizSpark partner, which made it really easy for me to apply. So I did. And today my application was approved.
Now I just have about three metric tons of stuff to download. They are very generous. I currently have Visual Studio Express installed - it's the free edition, but it gives you access to a pretty decent set of tools for basic development. Now I have access to so many different versions of their enterprise-level development tools that I don't know where to start.
Whether and how I use the software remains to be seen. I do want to write a desktop client as a companion to Minx, so that seems to be the place to start. But in any case, I have to thank Microsoft for giving me access to all this stuff.
Bug? On this post of mine, 5 comments seem to have vanished into the ether. I added a new one, and it shows up. The comment total says 6 even though that last one is the only one you can see.
Posted by: Steven Den Beste at Sunday, May 31 2009 05:22 AM (+rSRq)
There was a brief period of error 500, and now everything is back to normal. (Reboot?)
Posted by: Steven Den Beste at Sunday, May 31 2009 05:29 AM (+rSRq)
11
Yeah, sorry about that. I saw the comments go missing, and I figured that it had to be index corruption. A scan confirmed this, so I ran a fix immediately, but that more-or-less locks out the application while it's running, hence the 500 errors.
Posted by: Pixy Misa at Sunday, May 31 2009 05:38 AM (PiXy!)
We're virtualised now. I'll never have to do another messy server move. When we get the next server (which could be quite a while, since Akane has a much better upgrade path than our previous servers), all I need to do is set up OpenVZ (or Virtuozzo) on the new system, copy the container across, and then on the day of the move, stop the old container, do an rsync update, and start it up on the new server.
A few things broke because of CPanel overwriting configuration files, or version changes of underlying software, or moving from 32-bit to 64-bit. Next time, none of that will happen.
I've reached the quiet spot between frantic preparation and even more frantic bug-fixing, where the files are doing their final update. Back to panic mode in 3... 2...
All finished! Not too much pain involved, either. Just waiting for the DNS changes to roll out now.
The server encountered an internal error or
misconfiguration and was unable to complete
your request.
Please contact the server administrator,
webmaster@blog.mu.nu and inform them of the time the error occurred,
and anything you might have done that may have
caused the error.
More information about this error may be available
in the server error log.
Additionally, a 404 Not Found
error was encountered while trying to use an ErrorDocument to handle the request.
Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.8e-fips-rhel5
mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635
PHP/5.2.9 Server at blog.mu.nu Port 80
Posted by: Susie at Saturday, May 23 2009 09:08 PM (SFNt4)
Hi Pixy, I have a question about the server you described over at Ace of Spades HQ —
“a dual-processor quad-core hyper-threaded Xeon E5520 with 24GB of RAM, a 4TB RAID-5 array (using an Adaptec 5805 controller), a 32GB Intel X25-E SSD, redundant power supplies, and a gigabit internet connection, running CentOS 5.3 and OpenVZ.”
— Does this server host all of your mee.nu sites, or just Ace?
Thanks!
Posted by: CTN at Saturday, May 23 2009 11:52 PM (hJ2AD)
4
It's to host all of mu.nu and mee.nu, replacing three smaller servers. Ace is just one of the biggest sites and was going to be down a little longer because I had to set up the comments server separately, so I posted an announcement there.
There's a second, smaller server in the new arrangement - a quad-core Xeon 3230 with 2GB of RAM - but that's mainly for test and development. I got that because it came as a special deal with 4TB of bandwidth that I can pool with the main server - for half the price of the bandwidth alone.
Posted by: Pixy Misa at Sunday, May 24 2009 12:00 AM (PiXy!)
5
My next question isn’t fair, primarily because it’s none of my business . . . . but . . . how do you make money hosting sites for others, for free, when it requires you to maintain oodles of hardware and software, with no ostensible returns?
Posted by: CTN at Sunday, May 24 2009 12:19 AM (tyVJj)
Posted by: Pixy Misa at Sunday, May 24 2009 12:33 AM (PiXy!)
7
Clearly, I should have opened my reader before my site this morning. Thanks for all your hard work!!
Posted by: pam at Sunday, May 24 2009 01:29 AM (l6NIn)
8
Thanks, Pixy! Main menu good--blacklist not so good: "An error occurred: Byte order is not compatible at blib/lib/Storable.pm
(autosplit into blib/lib/auto/Storable/thaw.al) line 415, at
lib/MT/PluginData.pm line 28"
Sorry to be such a pest!
Posted by: Susie at Sunday, May 24 2009 02:51 PM (SFNt4)
It actually works four times out of five. But reading the log files it creates will give you the impression that not only has it failed, it's crashed your server and the two adjacent servers and set the rack on fire.
Update: It installs 2.6GB of stuff. Or in one case, 3.1GB. I'm reinstalling that one.
My favourite part of the whole install process is this:
!! If you want to create a support ticket with cPanel regarding this please reference 'BuildAP Report Id': '2035297' !! Stopping portmap: [FAILED] inetd: no process killed Shutting down kernel logger: [PASSED] sendmail: no process killed open3: exec of /etc/rc.d/init.d/exim stop failed at ./install line 1049 open3: exec of /etc/rc.d/init.d/exim start failed at ./install line 1049 open3: exec of /etc/rc.d/init.d/proftpd start failed at ./install line 1049 If you are getting an "undefined catalog error", please run: Found hostname to be azusa.mu.nu, which resolves to 174.37.121.36 initfpsuexec: using apache 2.x support error reading information on service proftpd: No such file or directory error reading information on service exim: No such file or directory error reading information on service cpanel: No such file or directory error reading information on service bandmin: No such file or directory Starting sshd: [ OK ] Cannot open /etc/chkserv.d at /scripts/restartsrv_exim line 99.
What this means is that the main install has completed succsessfully and it's now configuring Apache and PHP.
But you knew that already, right? After all, if you wanted to indicate success, a string of neon-green boldface messages that use the word "failed" four times and "error" five times would be your first choice.
Absolutely! Error messages are red. Green means everything is fine. Donchaknow?
(Bug? As part of making that joke, I tried to make the word "red" be red by using the editor gizmo. The code it generated put a "font" tag on that string, but it's like this:
<font face="">red</< SPAN>font>
Which doesn't do anything. Meanwhile, after entering that quote, now my formatting really is loused up. This text looks blue.
Posted by: Steven Den Beste at Sunday, May 17 2009 02:54 PM (+rSRq)
2
No, that didn't come through right. Oh, well. You're busy with other things right now anyway.
Posted by: Steven Den Beste at Sunday, May 17 2009 02:54 PM (+rSRq)
Highlight the text you want to color, click the A button, editor inserts “span style="color: rgb(x,y,z );"” tag ahead and “/ span” tag behind.
Also tried it the "other" way (no text selected, click A, select color, type text, click A, select black), and it still generates span style tags (but they don't get closed as elegantly).
Posted by: Old Grouch at Sunday, May 17 2009 04:56 PM (FZKGS)
Posted by: Steven Den Beste at Sunday, May 17 2009 11:18 PM (+rSRq)
6
Didn't mean to imply you didn't know how, was just curious if that was how you tried to do it.
Because it appears that color can also be set using the "styles & formatting" button, which (1) I've never used except to specify fonts, and (2) I see only in the post editor, not the comment editor. (I gather what displays for the editors is somewhat browser dependent: When mee.nu first came online, I didn't have any editing buttons for comments (although I did for posts). The comment buttons appeared a couple of months later when Pixy upgraded some of the tools/widgets.)
Posted by: Old Grouch at Monday, May 18 2009 03:52 AM (N7blS)
Posted by: Steven Den Beste at Tuesday, May 19 2009 02:08 PM (+rSRq)
9
Something I noticed that may or may not be a bug.
When I go to browse my templates, at the bottom I get the << Page 1 >> widget showing up. I can click the right arrow, get to Page 2, and it's all the same stuff. Go to Page 3, it's all the same again. It looks like I can just keep clicking forever and I see the same thing over and over again.
Posted by: Will at Thursday, May 21 2009 04:11 PM (RMeRQ)
I just bought 5GB of content for Bryce, the terrain-modelling and 3D-rendering program. I've hardly touched Bryce since I bought it on sale late in 2007* but I do have a nice new quad-core system with a teraflop graphics card to run it on if I ever find the time.
So, you ask, if I never use the program, why did I buy all that content?
* Or indeed, when I first bought it way back in... Whenever version 4 came out.
** Individual list price is over $700, discounted price is $50, Platinum Club member price is $35, and I had my monthly $5 voucher, so only $30 in the end.
Posted by: Pixy Misa at
12:11 AM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 121 words, total size 1 kb.
Saturday, May 16
Eating Mice
Anyway, I'm going to try a new recipe tonight - teriyaki beef with snow peas on jasmine rice. Was going to use lamb, but there was no appealing-looking diced lamb at the supermarket this evening.
In theory, we're scheduled to move to a new server tomorrow.
That might still happen, though it's looking doubtful. The reason it might still happen is that if it doesn't, I'll be up for hundreds of dollars in fees to keep the old servers around for another month.
And the reason it's doubtful is that we had trouble with the systems at work every day last week. Every day, US time. So between 3AM and 5AM my time. And every time, I got woken up and had to fix it.
One hardware fault, one software bug, one software bug in an error-handling routine triggered by another site going haywire, one software bug in normal processing triggered by another site going haywire, and one instance of things just not working right for no apparent reason.
Then I finally got back to setting up the new server. I found some issues with my configuration of the containers for the CPanel sites - insufficent space allocated for the kernel structures, insufficient datagram buffers (which is hardly critical, but I fixed it while I was there) and a couple of other things were we were close to the limit.
So I fixed all of that, recreated the containers, and reinstalled CPanel.
Whereupon I came unstuck, because CPanel would not install.
Turned out - after considerable cursing and deleting and recreating of containers - to be a firewall issue. CPanel's installer couldn't access CPAN (no relation) because CPAN couldn't access the server.
Fucking FTP. Passive mode is there for a reason - though it seems to work about as consistently as secondary DNS servers.
Anyway, sorted that out, but noticed that 6GB of RAM had gone walkies without leaving a forwarding address. It was physically there, but unaccounted for inside Linux. A reboot sorted that out, but then I had a slight accident with a command run on the hardware node instead of inside a container.
So I decided to do a clean reinstall of Linux and load everything again. Since I've been experimenting on the box for two weeks now, this is not overall a bad idea.
The reinstall of CentOS went smoothly. The I went to load OpenVZ - and the OpenVZ repository, and indeed the site - and the list of mirrors - was down.
Gah.
It's back now, but I've pretty much lost a day.
Plan now is to move mee.nu (and the mu.nu Minx sites), and prepare to move mu.nu. That way I can cancel at least one of the servers.
Akari is a quad-core system with 6GB of RAM; Azusa and Kodachi are dual-core systems with 4GB of RAM. All running CentOS 5.3 64-bit and CPanel. They'll be taking over duties from Midori and Sakura.
Next up are Nabiki and Kasumi. Nabiki is a quad-core system with 4GB of RAM and a dedicated SSD; it's our new database server. Kasumi is our replication server, to couple our data extra-double-safe. It's a one-and-a-half core system with 1GB of RAM.
Yeah.
As that last might have alerted you, these are not real, physical, stuff-you-can-kick servers, they're virtual servers under OpenVZ. It takes about seven seconds to create a new server, but it took about five hours to get all the configuration settings correct* and install CPanel.**
Took a little reading to get up to speed on all the commands and options, but I'm saving about $2500 a year by not paying for the pretty user interface of Virtuozzo, and for $2500 I'm willing to do a little reading.
* At one point I had Akari, Azusa, and Kodachi running with 30% and 20% of a CPU - rather than 30% and 20% of the total number of CPUs. That was a leetle slow.
** CPanel's installation takes a while at the best of times. It takes about five whiles if you've accidentally restricted your server to only use one-fifth of one processor.
Some of the utilities to do that don't want to install on 64-bit Linux. Which is retarded, because that's probably 90% of the people who would be using them.
Posted by: Pixy Misa at Monday, May 11 2009 10:27 AM (PiXy!)
3
Aha. You can indeed do it without the missing tools, thought it's a bit circuitous.
You need to copy the directory containing the VPS you want to clone to a new directory with the ID number of the new VPS. Then you need to likewise copy the config file, and assign it the appropriate hostname and IP address.
Then... Um, then you're done. Not all that circuitous after all.
Posted by: Pixy Misa at Monday, May 11 2009 05:19 PM (PiXy!)
1
Do you mean "Crap, the drive died" blew up or "Umm, there are pieces of what used to be a HDD in what used to be a server" blew up?
Posted by: ReallyBored at Thursday, May 07 2009 01:33 AM (aGZBd)
2
I mean, "I'm pretty sure the data centre was here yesterday" blew up.
...
Well, actually, it just spontaneously developed about thirty million bad sectors. In the middle of restructuring our production database. Had to drag everything back from the replica.
Posted by: Pixy Misa at Thursday, May 07 2009 01:52 AM (PiXy!)
3
I blame it on the lack of sun spots... we had something along those lines happen with one of our production servers recently. Scared the crap out of my boss. heh. (thank heaven I wasn't in Chicago or it might've been me there for the spectacular failure *grin*)
Posted by: Teresa at Friday, May 08 2009 02:20 PM (ybEr8)
4
Ah, I see you were visited by Charming Monkey Girl too. Personally, I found her way too talky and downright pushy.
Posted by: The Brickmuppet at Friday, May 08 2009 05:40 PM (xqpbD)
5
Yeah, banning her now. Just had an exploding server at work and a DMCA complain against one of the forums I host, what feels like a mild case of food poisoning, and about a six-month sleep deficit slowing me down...
Posted by: Pixy Misa at Saturday, May 09 2009 12:18 AM (PiXy!)
6
Four sightings at my place (unusual!) at 0617, 0625, 0633, 0641 GMT. Will just delete.
Can't be said enough: REALLY appreciate what you do!
Posted by: Old Grouch at Saturday, May 09 2009 12:25 AM (8vPcQ)