CAN I BE OF ASSISTANCE?

I'm Glad That Wasn't Connected To My Pager

Still getting "Minx Failure Alerts" from my screw-up earlier.  We were down for about 15 minutes.  Each Minx process will automatically attempt a restart every 3 seconds, and there are five processes (each with 25 threads).  That's... carry the twelve... 1500 emails.

That's not even counting the panics and double panics, which I get whenever a page fails.  There's an automatic retry on page failure, so when there's a problem that leaves Minx broken but not actually dead, every page loaded delivers two emails to my inbox.

Posted by: Pixy Misa at 08:37 PM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 95 words, total size 1 kb.

PEBKAC

Okay, well that seemed to work fine, now I'll just remove the test database and

NO DON'T DO IT YOU'RE ON THE LIVE SYSTEM!!!!!!

Ctrl-C Ctrl-C Ctrl-C Ctrl-C Ctrl-C Ctrl-C Ctrl-C Ctrl-C Ctrl-C Ctrl-C Ctrl-C

Sorry about that, folks. 

Posts and comments and such are intact, but new users who registered in the last six hours might have to sign up again.

Posted by: Pixy Misa at 01:15 PM | Comments (8) | Add Comment | Trackbacks (Suck)
Post contains 64 words, total size 1 kb.

Damn Shinies

Ooh, shiny! @Mail, the nicest web mail client I've ever seen (and made right here in Australia) is now offered as part of Plesk, a fairly nice control panel that's the major competitor to CPanel.

If I run Virtuozzo on the new servers, I can get Plesk, and by extension @Mail, cheap.  Virtuozzo itself, however, is not cheap.

But shiny.

But not cheap.

But shiny.

Posted by: Pixy Misa at 11:26 PM | Comments (1) | Add Comment | Trackbacks (Suck)
Post contains 67 words, total size 1 kb.

Good Thing It Ain't Tokyo

The main server cluster (at SoftLayer), the off-site backup/spare server (at TailorMadeServers) and the new test cluster (at Rackspace) are all located in Dallas.

NOBODY NUKE DALLAS!

Should be safe from giant radiation-breathing aquatic lizards, at least.

Posted by: Pixy Misa at 10:23 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 42 words, total size 1 kb.

Well, Why Didn't You Say So?!

Mosso has two products of interest: Cloud Servers, which are virtual servers except that...  Well, except nothing, they're virtual servers, and Cloud Files, which is a CDN except that...  Uh, yeah, CDN.  Right.

Problem is, their CDN doesn't have an Australian POP.  The nearest one is Singapore, which is close in air miles but might as well be in Iceland as far as ping times go.  (...  Okay, no, ping time to Singapore is about 250ms, to Iceland about 350ms ...)

With SoftLayer's CDNLayer, I get about 15ms ping time to the nearest POP.  With Mosso's Cloud Files, I get about...

[root@yurie work]# ping cdn.cloudfiles.mosso.com
PING rackspace.vo.llnwd.net (117.121.253.28) 56(84) bytes of data.
64 bytes from cds20.syd.llnw.net (117.121.253.28): icmp_seq=1 ttl=58 time=14.7 ms
64 bytes from cds20.syd.llnw.net (117.121.253.28): icmp_seq=2 ttl=58 time=14.9 ms
64 bytes from cds20.syd.llnw.net (117.121.253.28): icmp_seq=3 ttl=58 time=14.4 ms
64 bytes from cds20.syd.llnw.net (117.121.253.28): icmp_seq=4 ttl=58 time=14.1 ms
64 bytes from cds20.syd.llnw.net (117.121.253.28): icmp_seq=5 ttl=58 time=14.4 ms

Uh.  Well, why didn't you say so?

Posted by: Pixy Misa at 10:03 PM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 167 words, total size 1 kb.

New Servers!

I got a new server for mee.nu today.  Say hello, Chiriri!

Hewwo.

Chiriri is hosted at Mosso, a division of Rackspace.  Rackspace is well-regarded in the industry as far as support and reliability goes; less well regarded as far as pricing goes.  To say they are expensive is to say that durians can be a bit whiffy.  So how much have you cost me so far, Chiriri?

'Bout ten cents.

Chiriri is a virtual server, with 256MB of memory and 10GB of disk space.  That's not much at all by today's standards, and if you tried to run Apache on there you'd be in for a world of hurt, but Minx uses Pound and CherryPy.  Where an Apache process on Midori uses 10 to 20MB of memory (after some ferocious tuning to reduce memory consumption), the Minx processes on Aoi use about 40MB each - to provide 25 threads.  So Minx uses about one tenth the memory of even a well-tuned Apache/PHP application, which means that it can actually run fairly well on a 256MB machine.

Which costs me 1.5¢ per hour.  (Plus bandwidth charges.)

So if I want to, say, try out a different database, or test a replication or clustering strategy, and need four small servers for a couple of days, I can do that for the cost of a Sausage and Egg McMuffin.  If I needed 512MB per server, it would run me as much as a KFC Works Burger (not the meal, just the burger).

Very very convenient.

The bandwidth charges are a bit high - 22¢ per GB where SoftLayer provide 2TB with a standard server and charge 10¢ per GB excess.  But for a small site that has a moderate volume, but that requires a custom configuration - say, separate database and application servers - very handy indeed. 

Right, Chiriri?

Right!

Posted by: Pixy Misa at 08:09 PM | Comments (7) | Add Comment | Trackbacks (Suck)
Post contains 304 words, total size 2 kb.

It's a Good Thing

That I have a quad-core machine, because Safari's using two of them.

Posted by: Pixy Misa at 05:49 PM | Comments (10) | Add Comment | Trackbacks (Suck)
Post contains 16 words, total size 1 kb.

The Other Nextwave*

* No, still not the Warren Ellis comic.  Just go and buy the darn thing.  It's got drop bears in it.  What more can you ask?

I had to run a little cleanup script at mee.nu yesterday.  I'd done a bulk zap of CJKV spam via phpMyAdmin, and that left the comment counts out of sync (because they're maintained by the application, rather than by a trigger).  The script took 84 seconds to scan 2.7 million posts and comments across 480,000 threads, and fix everything up nice and neat.

Which isn't bad at all, but for those 84 seconds, mee.nu was locked up.  You couldn't load a page or post a comment or pretty much anything at all.  Some of the admin functions probably worked, and that's about it.

That's a problem of MySQL's MyISAM tables.  They're fast, they're compact, they're easy to implement, and they have all the robust concurrency of a unicycle parade going backwards through a broken turnstile.

Uphill, in a thunderstorm, at night.

So I'm looking at ways to fix that.  If the cleanup script took twice as long - but didn't lock anything up while it was running - then that would make life a lot easier for me.

One way is to use InnoDB tables.  These are less compact and slower than MyISAM, but provide better concurrency.  They don't provide full-text indexing, but at the moment we're not using full-text indexing.  We do full-text searches, but using brute-force scans.  Because, for our purposes, it's more efficient.

But MySQL isn't the only kitten in the sea.  There's PostgreSQL, which is open source and more powerful and flexible than MySQL.  I didn't use it for Minx originally because full-text indexing in PostgreSQL is hideous, but then I didn't end up using full-text indexing.

There's Firebird, the open source version of Borland Interbase.  I don't know much about Firebird, and it doesn't seem to be used much, and the documentation is somewhat lacking, so I'm not inclined towards it at all, really.

There's Ingres.  Yep, Ingres is open source these days.  I'm not sure of the power or the feature set, since I've never used it, but I'm downloading it right now.  (Actually, it would appear that I'm not, since their download server has fallen over.  Ah, there we go.  Documentation is still downloadable; just the software download server is down.)

There's Sybase SQLAnywhere.  It's not open source, but it's free for web applications.  It's your typical solid workgroup database.  Not seriously enterprisey, but without all the random limitations of MySQL.

And there's DB2.  Unfortunately, the free version, DB2 Express-C, is limited to 2 CPUs and 2GB of memory, which is plenty for now (the rest of the memory can go to filesystem cache, which works well enough).  But the next step up - 4 CPUs and 4GB of memory - costs $3000 a year.  So I'm looking at DB2, but it's not my first choice.

Oh, and MaxDB, formerly SAPDB, a database system developed by SAP for running, um, SAP.  It's open source.  Doesn't have full-text indexing at all, but then, we're not using full-text indexing.

I want to get those Ingres docs, darn it.  Open source and professionally developed sounds like a good combination to me.

Update: Ingres has no full-text indexing.  That's a little disappointing, but as I say, we currently don't use full-text indexing because a brute-force search within a site is more efficient.  And I'm looking at dedicated full-text indexing solutions like Xapian in any case; they're a lot more sophisticated than what you typically get in an RDBMS.

It does have R-Trees.  That's a bit of a surprise, and potentially useful.

It appears to have INTERSECT and EXCEPT.  Those are really useful, and can dramatically simplify and accelerate certain queries.  I say appears to, because they're in the keyword list but not described in any detail that I've found so far.

Does come with a Python DB-API compliant interface.  Kind of important, that!

Posted by: Pixy Misa at 04:27 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 664 words, total size 4 kb.

Shorter Nextwave*

* Still not the Warren Ellis comic.  Sorry.  See below for longer Nextwave.

Coming in April: Akane and Mikan (crimson and orange) to replace Aoi and Midori (blue and green).

New hardware:

Xeon 3500 2.66GHz (40% to 80% faster)
12GB of memory (up from 8GB)
1.5TB RAID-5
Adaptec RAID controller with 256MB battery-backed cache
32GB Intel X25-E SSD (25x to 250x as fast as a disk)

New software:

64-bit Linux (we're currently on 32-bit)
Virtualisation of some sort (OpenVZ, Virtuozzo, or Citrix XenServer)

Posted by: Pixy Misa at 12:24 PM | Comments (7) | Add Comment | Trackbacks (Suck)
Post contains 85 words, total size 1 kb.

Nextwave*

* Not the Warren Ellis comic.  Sorry.

The current mu.nu / mee.nu server conglomeration consists of Aoi and Midori (blue and green), two Xeon 3220 systems with 8GB of memory and, well, a bunch of disk drives, and Sakura (pink), a Core 2 Duo 6500 system with 8GB of memory and a bunch of disk drives.

When Sakura went wobbly with a sprained disk a couple of weeks ago, I had to move some sites back to Midori, which then also went wobbly, albeit not because it was out of memory or CPU, but because it was out of Apache threads.  That caused me a few sleepless nights until I could get Sakura back in action.

There's plenty of free resources on Aoi, the mee.nu server, because Minx is so memory and CPU-efficient, but I can't easily drop random sites on there because the whole server is configured to run Minx.

That's why I got Sakura in the first place, or one of the reasons, the other being off-site backup in case something horrible happens.

Particularly around election time, I've wished that I could just upgrade the memory in Midori and install OpenVZ; that would solve a lot of problems.  And while I could install OpenVZ, upgrading the memory is not so easy: The motherboard supports a maximum of 8GB, which is kind of what we have already.

So I've been waiting, more-or-less patiently, for Nehalem-based servers to become available.  You see, the current Nehalem processors (Bloomfield and Gainestown) support three memory channels, and at least two DIMMs per channel, which means a single-processor Nehalem system can go to at least 12GB of memory.

And if the motherboard supports registered memory, you can get three DIMMs per channel and 4GB DIMMs, for a total of 36GB.

So at least 12, maybe 24 or even 36.  Any of which is an improvement on 8.

The server version of Nehalem - the Xeon 3500 (single processor) and 5500 (dual processor) ranges - are reportedly due out in exactly a week - March 29.  Intel haven't confirmed this so far as I know, but given that Apple is already shipping them in the new Mac Pro, the launch can't be too far off.

Apart from the improved memory support (up to 4.5x the capacity and 3x the bandwidth) over our current processors, the Nehalem delivers a sizeable performance boost.  On SpecInt a 2.66GHz Nehalem scores 28.6 vs 20.3 for the 2.4GHz 3220, a 40% increase, and on SpecIntRate, the multi-threaded version of the benchmark, it delivers 109 vs 59.2, more than an 80% increase.

The Nehalem supports hyperthreading, just like he good old Pentium 4, only this time it provides a much bigger performance boost.  A single 2.66GHz Nehalem scores about the same in multi-threaded benchmarks as a dual 2.33GHz Xeon of the previous model.  (On multi-threaded floating-point it does even better, about the equivalent of a dual 3GHz previous-model Xeon.  But I'm less interested in floating-point.)

So, that's memory and CPU covered.  What's next?

Well, last November we got an ioDrive in for testing at my day job.  These things are incredible; they deliver 120,000 random IOs per second.  Unfortunately, they're also incredibly expensive; the cheapest card costs $3000.

But SoftLayer now sells Intel's X25-E SSDs.  They're not as fast as an ioDrive, delivering "only" 30,000 read IOPS and 3,000 write IOPS, but that's still 25 to 250 times as fast as the disks we have right now.  We have one in our development server at my day job - I just finished setting it up - and it really seems to work as advertised.

Best part?  Only $50 per month.

So, 2.66GHz Nehalem, 12GB of RAM, 32GB X25-E.  What's next?

Next is RAID.  We're not running RAID at the moment, because when I set up the current servers I simply couldn't afford it - I have to pay for the RAID controller itself, and for twice the number of disks.  For each server.  Instead I have the servers doing a daily rsync to each other - which we'd still need to do, but with RAID we'd only need that for operator or software error, not for simple drive failures.

The good thing there is that SoftLayer's disk pricing is effectively half what it was when I set up the current servers, so I only have to pay for the controller.  So, hardware RAID it is, probably either 1TB RAID-1 or 1.5TB RAID-5.  RAID-5 is no good for databases, but the databases will be on the SSD, so it doesn't matter.

The one big variable is virtualisation.  I'd like to go Virtuozzo, because it's efficient and elegant (it's based on OpenVZ, but has some extra features and a nice control panel), and it integrates cleanly with Plesk.  (And offers substantial discounts on Plesk licenses - about 66%.)

The problem is, a 10-VM Virtuozzo license, the smallest practical size,
costs $100 per month.  Per server.  For that much money I could use Citrix XenServer (which is now free), add 6GB of memory to each server (to make up for Xen's less memory-efficient virtualisation model), and still save enough to pay for the more expensive Plesk license if I wanted it.

So, meh.  We'll see.

There's some other things I'm looking at - hardware firewalls (we sort of have one, but it's just there as a freebie and I don't have access to configure it); hardware load balancers (we use a software load balancer at mee.nu, which works very nicely as long as the server itself doesn't hiccup, but nothing at mu.nu right now); a content delivery network (which I already have, but need to configure); backups to iSCSI or CIFS storage (why don't they offer NFS? ).  They all cost money, though, so we'll see.

As to when?  First week of April.

Posted by: Pixy Misa at 11:29 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 971 words, total size 6 kb.