Thursday, December 28
With new servers coming soon, and lots of new users with them, I've been looking for a good monitoring system.
Simple Server Monitor is what it says. It monitors your servers - on ping, HTTP, SMTP and/or POP - and screams like a stuck parrot if anything goes down.* It also plots little graphs of response times. Not fancy, but it fills the immediate need.
I also tried out Spiceworks, which is kind of neat, but really designed for monitoring small office networks. It's written in Ruby (using Rails), and runs in the background, accessed through your browser. For a Web app, it's pretty neat, though for a monitoring system it gives me a great deal of information I don't want. Doesn't seem to like monitoring the web servers, either.
Zenoss is probably quite good if you work with SNMP 7 days a week. Otherwise, avoid. The simplest way to set it up involves downloading a pre-packaged virtual machine and running it under VMWare.
PortSensor is another simple monitoring tool, but a very flexible one since you can have it execute custom shell scripts remotely via SSH and process the results using regular expressions. Naturally I'd rather not do any of that, and it doesn't have any particularly significant functionality built in. It's more a sort of monitoring toolkit.
Simple Server Monitor I'm keeping until I find something better; the others have hit the bit bucket.
* Well, that last is due to the alert sound I chose.
Tuesday, December 26
MySQL supports large files if your operating system does. So there's no need to worry about your application dropping dead if a particular MyISAM table exceeds 4GB.
Except that the default table pointer size is 32 bits.
And it seems that changing it from 32 to 40 bits slows database writes by about 10%.
Not that I will necessarily be using MyISAM. What I was doing when my table hit 4GB was finding out whether to use MyISAM or InnoDB.
Well, off we go again. About 4500 inserts per second right now (at one record per query, which is far from the fastest method, but is after all what the application will be doing).
Apache and Exim ganged up on me again. I caught it early this time, when it was only 1GB into swap. Load average peaked at over 1000.
Monday, December 25
IronPython is a version of Python written in C# for the .Net platform. It's intended as an embedded scripting language for .Net applications, but can also be used to quickly assemble applications from components written in other .Net languages. Since I'm currently working in both Python and .Net, this is of interest to me, so I downloaded IronPython* and ran my little benchmark.
And then I went and had dinner, and read a book, and read some blogs, and had some ice cream, and then came and posted this. Because IronPython has a leetle performance problem relative to CPython**; or rather, it lacks at least one very important optimisation from CPython. See if you can spot what it is:
|Amelia||Core Duo||1.66GHz||2.5 (Win)||1.243||3.158||1.033||5.434|
|Amelia||Core Duo||1.66GHz||2.5 (Win)+Psyco||0.037||0.483||1.190||1.710|
|Amelia||Core Duo||1.66GHz||IronPython 1.01||0.698||2236.319||2.045||2239.062|
Source code for the benchmark can be found in my earlier post.
* Again; I had it on my old notebook, but that got reformatted and reinstalled at least four times, and in any case I no longer have it.
** The standard version of Python, which is written in C. more...
Analysis by Peter Gutmann.
Sunday, December 24
Just rescued Akane from a load average of over 800, without rebooting.
Spam flood. Double spam flood, in fact - blog comments and incoming email.
Fluffy wasn't running, and Apache took over the machine, and then exim took it the rest of the way. By the time I logged in it was using nearly 100% of swap.
Saturday, December 23
Half a petabyte a month? Eek.
For some time I've been running a mix of SpamAssassin on the server and Thunderbird junk mail controls and message filters to keep my email under control. I had SpamAssassin just marking email, not deleting it, and then went into my Junk folder every two or three days to see if there was anything that had been misdirected that I needed to attend to.
Over the last few months, the frequency of my visits to the Junk folder have increased to every day, then to two or three times a day, then every two or three hours. Even though I have filters set to auto-delete the obvious crap, I'm getting something like 2000 emails a day, 99% of them spam.
Today I set SpamAssassin to auto-delete at 9, and set the value for BAYES_99 to 7. (I'm tempted to bump that up to 8.8.)
Unfortunately, since I monitor email for all of mu.nu - bounces, abuse reports, that sort of thing - I need to receive things that set off a lot of normal spam flags. I don't want to, but if I don't keep an eye on things then sooner or later I'll be hearing from our hosting company. (The PHP spam incident I refer to below was one that I caught and fixed without anyone at SoftLayer being troubled because I saw the sudden increase in bounce messages.)
Fortunately the Bayesian filtering in SpamAssassin seems to be very reliable, and a 99% rating plus two points worth of other spammy traits will now relieve me of ever seeing the message. I've seen valid messages get Bayes ratings of 40 or so, but that won't get zapped unless you're doing a lot of other things wrong.
Seems to be working. I'm still getting the occasional spam through to Thunderbird - at a setting of 9, SpamAssassin doesn't catch those blasted PHYA pump-and-dump spams - but most of it is then gets eaten by my filter rules.
And a blessed peace descended upon Pixy Central...
Thursday, December 21
One of the best things about Windows networking is that when a Windows PC loses a network connection - say, you unplug the cable - it can no longer see itself.
This matters to me more than most because I run VMWare Server on my notebook, and when I'm not connected to my wireless network I can't connect to my virtual Linux server either. Even though it's running on the same box.
So what I did was this: I installed the Microsoft Loopback Adaptor, and bridged it to my wireless network. So now, with or without wireless, I can still reach my server.
The only problem is, Windows now tells me that my wireless is disconnected. And if you go into the wireless networks screen, right below where it says "Not connected", it has a Disconnect button to disconnect the unconnected network.
Apart from that, it works fine.
I was having trouble mounting the Samba share from the virtual Linux server back to Windows, but that was because Plesk doesn't (or at least, didn't) automatically apply its firewall rules after a reboot.* Quite possible something I did wrong, but certainly confusing.
Plesk also has a nicely confusing way of managing your Samba shares; it adds an include file to the standard smb.conf, puts the shares it's managing in there, and ignores what's defined in the main file. That might actually be the best way to handle things, but it ain't obvious when you run into it for the first time.
I also set up a PPTP VPN to SoftLayer from my home network. That lets me manage the servers even if the regular public network at the hosting company is down (the VPN goes through a dedicated Cogent link); it lets me run backups without touching by 4TB/month bandwidth allowance; and it lets me do things like check on CPU temperatures and voltage levels in the servers. And do both soft and hard resets, if I should feel the urge. Without even opening a browser window!
* Yes, I have a complete web hosting management system installed on a virtual machine on my notebook.
You mean to say you don't?
Wednesday, December 20
Installing the new servers at PPoE.
I brought the FC5_x64 DVD with me instead of FC5_i386, but what the heck, the machines are 64 bit and it's officially supported by the software I want to install, so roll with it.
So I install FC5, and then I install the software I want to install, and it gets allll the way through downloading and installing itself and falls over with three dozen version conflicts in various RPMs.
So now I'm reinstalling CentOS 4.4, which is the only other Linux I have on me, but which I know definitely works. Unfortunately, all I have is the CDs, so I have to sit here swapping them back and forth between the machines.
The best part is when, two minutes into CD #4, it asks for CD #1 again. When CD #1 is in another machine.
57 queries taking 0.248 seconds, 308 records returned.
Powered by Minx 1.1.6c-pink.