No boom today. Boom tomorrow. There's always a boom tomorrow.

Well, Poop, Part 37 Or Something

The new Xeon 5500 dual-processor Nehalem has been announced.

So too - if you look hard enough - the single-processor Xeon 3500.

SuperMicro (and SoftLayer exclusively uses SuperMicro) have announced over 30 new Xeon 5500 motherboards.

And no new Xeon 3500 boards.  Meaning that all they have are their existing Core i7 boards, none of which support registered memory, so they're limited to 12GB.

Which means that to go beyond 12GB I'll still need to pay the dual-processor-motherboard tax of at least $100 per month per server - and then pay for the extra memory.

And the dual-processor CPUs themselves are substantially more expensive than the single-processor equivalents.

Dammit, SuperMicro, pull your finger out.  Even Gigabyte managed to get this one right, and they hardly have a presence in the server market at all.

Posted by: Pixy Misa at 08:44 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 140 words, total size 1 kb.

Sydney Power Out, Hardly Any Zombies

See Twitter for updates.

Posted by: Pixy Misa at 05:40 PM | Comments (1) | Add Comment | Trackbacks (Suck)
Post contains 10 words, total size 1 kb.

I'm Glad That Wasn't Connected To My Pager

Still getting "Minx Failure Alerts" from my screw-up earlier.  We were down for about 15 minutes.  Each Minx process will automatically attempt a restart every 3 seconds, and there are five processes (each with 25 threads).  That's... carry the twelve... 1500 emails.

That's not even counting the panics and double panics, which I get whenever a page fails.  There's an automatic retry on page failure, so when there's a problem that leaves Minx broken but not actually dead, every page loaded delivers two emails to my inbox.

Posted by: Pixy Misa at 08:37 PM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 95 words, total size 1 kb.

PEBKAC

Okay, well that seemed to work fine, now I'll just remove the test database and

NO DON'T DO IT YOU'RE ON THE LIVE SYSTEM!!!!!!

Ctrl-C Ctrl-C Ctrl-C Ctrl-C Ctrl-C Ctrl-C Ctrl-C Ctrl-C Ctrl-C Ctrl-C Ctrl-C

Sorry about that, folks. 

Posts and comments and such are intact, but new users who registered in the last six hours might have to sign up again.

Posted by: Pixy Misa at 01:15 PM | Comments (8) | Add Comment | Trackbacks (Suck)
Post contains 64 words, total size 1 kb.

Damn Shinies

Ooh, shiny! @Mail, the nicest web mail client I've ever seen (and made right here in Australia) is now offered as part of Plesk, a fairly nice control panel that's the major competitor to CPanel.

If I run Virtuozzo on the new servers, I can get Plesk, and by extension @Mail, cheap.  Virtuozzo itself, however, is not cheap.

But shiny.

But not cheap.

But shiny.

Posted by: Pixy Misa at 11:26 PM | Comments (1) | Add Comment | Trackbacks (Suck)
Post contains 67 words, total size 1 kb.

Good Thing It Ain't Tokyo

The main server cluster (at SoftLayer), the off-site backup/spare server (at TailorMadeServers) and the new test cluster (at Rackspace) are all located in Dallas.

NOBODY NUKE DALLAS!

Should be safe from giant radiation-breathing aquatic lizards, at least.

Posted by: Pixy Misa at 10:23 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 42 words, total size 1 kb.

Well, Why Didn't You Say So?!

Mosso has two products of interest: Cloud Servers, which are virtual servers except that...  Well, except nothing, they're virtual servers, and Cloud Files, which is a CDN except that...  Uh, yeah, CDN.  Right.

Problem is, their CDN doesn't have an Australian POP.  The nearest one is Singapore, which is close in air miles but might as well be in Iceland as far as ping times go.  (...  Okay, no, ping time to Singapore is about 250ms, to Iceland about 350ms ...)

With SoftLayer's CDNLayer, I get about 15ms ping time to the nearest POP.  With Mosso's Cloud Files, I get about...

[root@yurie work]# ping cdn.cloudfiles.mosso.com
PING rackspace.vo.llnwd.net (117.121.253.28) 56(84) bytes of data.
64 bytes from cds20.syd.llnw.net (117.121.253.28): icmp_seq=1 ttl=58 time=14.7 ms
64 bytes from cds20.syd.llnw.net (117.121.253.28): icmp_seq=2 ttl=58 time=14.9 ms
64 bytes from cds20.syd.llnw.net (117.121.253.28): icmp_seq=3 ttl=58 time=14.4 ms
64 bytes from cds20.syd.llnw.net (117.121.253.28): icmp_seq=4 ttl=58 time=14.1 ms
64 bytes from cds20.syd.llnw.net (117.121.253.28): icmp_seq=5 ttl=58 time=14.4 ms

Uh.  Well, why didn't you say so?

Posted by: Pixy Misa at 10:03 PM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 167 words, total size 1 kb.

New Servers!

I got a new server for mee.nu today.  Say hello, Chiriri!

Hewwo.

Chiriri is hosted at Mosso, a division of Rackspace.  Rackspace is well-regarded in the industry as far as support and reliability goes; less well regarded as far as pricing goes.  To say they are expensive is to say that durians can be a bit whiffy.  So how much have you cost me so far, Chiriri?

'Bout ten cents.

Chiriri is a virtual server, with 256MB of memory and 10GB of disk space.  That's not much at all by today's standards, and if you tried to run Apache on there you'd be in for a world of hurt, but Minx uses Pound and CherryPy.  Where an Apache process on Midori uses 10 to 20MB of memory (after some ferocious tuning to reduce memory consumption), the Minx processes on Aoi use about 40MB each - to provide 25 threads.  So Minx uses about one tenth the memory of even a well-tuned Apache/PHP application, which means that it can actually run fairly well on a 256MB machine.

Which costs me 1.5¢ per hour.  (Plus bandwidth charges.)

So if I want to, say, try out a different database, or test a replication or clustering strategy, and need four small servers for a couple of days, I can do that for the cost of a Sausage and Egg McMuffin.  If I needed 512MB per server, it would run me as much as a KFC Works Burger (not the meal, just the burger).

Very very convenient.

The bandwidth charges are a bit high - 22¢ per GB where SoftLayer provide 2TB with a standard server and charge 10¢ per GB excess.  But for a small site that has a moderate volume, but that requires a custom configuration - say, separate database and application servers - very handy indeed. 

Right, Chiriri?

Right!

Posted by: Pixy Misa at 08:09 PM | Comments (7) | Add Comment | Trackbacks (Suck)
Post contains 304 words, total size 2 kb.

It's a Good Thing

That I have a quad-core machine, because Safari's using two of them.

Posted by: Pixy Misa at 05:49 PM | Comments (10) | Add Comment | Trackbacks (Suck)
Post contains 16 words, total size 1 kb.

The Other Nextwave*

* No, still not the Warren Ellis comic.  Just go and buy the darn thing.  It's got drop bears in it.  What more can you ask?

I had to run a little cleanup script at mee.nu yesterday.  I'd done a bulk zap of CJKV spam via phpMyAdmin, and that left the comment counts out of sync (because they're maintained by the application, rather than by a trigger).  The script took 84 seconds to scan 2.7 million posts and comments across 480,000 threads, and fix everything up nice and neat.

Which isn't bad at all, but for those 84 seconds, mee.nu was locked up.  You couldn't load a page or post a comment or pretty much anything at all.  Some of the admin functions probably worked, and that's about it.

That's a problem of MySQL's MyISAM tables.  They're fast, they're compact, they're easy to implement, and they have all the robust concurrency of a unicycle parade going backwards through a broken turnstile.

Uphill, in a thunderstorm, at night.

So I'm looking at ways to fix that.  If the cleanup script took twice as long - but didn't lock anything up while it was running - then that would make life a lot easier for me.

One way is to use InnoDB tables.  These are less compact and slower than MyISAM, but provide better concurrency.  They don't provide full-text indexing, but at the moment we're not using full-text indexing.  We do full-text searches, but using brute-force scans.  Because, for our purposes, it's more efficient.

But MySQL isn't the only kitten in the sea.  There's PostgreSQL, which is open source and more powerful and flexible than MySQL.  I didn't use it for Minx originally because full-text indexing in PostgreSQL is hideous, but then I didn't end up using full-text indexing.

There's Firebird, the open source version of Borland Interbase.  I don't know much about Firebird, and it doesn't seem to be used much, and the documentation is somewhat lacking, so I'm not inclined towards it at all, really.

There's Ingres.  Yep, Ingres is open source these days.  I'm not sure of the power or the feature set, since I've never used it, but I'm downloading it right now.  (Actually, it would appear that I'm not, since their download server has fallen over.  Ah, there we go.  Documentation is still downloadable; just the software download server is down.)

There's Sybase SQLAnywhere.  It's not open source, but it's free for web applications.  It's your typical solid workgroup database.  Not seriously enterprisey, but without all the random limitations of MySQL.

And there's DB2.  Unfortunately, the free version, DB2 Express-C, is limited to 2 CPUs and 2GB of memory, which is plenty for now (the rest of the memory can go to filesystem cache, which works well enough).  But the next step up - 4 CPUs and 4GB of memory - costs $3000 a year.  So I'm looking at DB2, but it's not my first choice.

Oh, and MaxDB, formerly SAPDB, a database system developed by SAP for running, um, SAP.  It's open source.  Doesn't have full-text indexing at all, but then, we're not using full-text indexing.

I want to get those Ingres docs, darn it.  Open source and professionally developed sounds like a good combination to me.

Update: Ingres has no full-text indexing.  That's a little disappointing, but as I say, we currently don't use full-text indexing because a brute-force search within a site is more efficient.  And I'm looking at dedicated full-text indexing solutions like Xapian in any case; they're a lot more sophisticated than what you typically get in an RDBMS.

It does have R-Trees.  That's a bit of a surprise, and potentially useful.

It appears to have INTERSECT and EXCEPT.  Those are really useful, and can dramatically simplify and accelerate certain queries.  I say appears to, because they're in the keyword list but not described in any detail that I've found so far.

Does come with a Python DB-API compliant interface.  Kind of important, that!

Posted by: Pixy Misa at 04:27 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 664 words, total size 4 kb.

Shorter Nextwave*

* Still not the Warren Ellis comic.  Sorry.  See below for longer Nextwave.

Coming in April: Akane and Mikan (crimson and orange) to replace Aoi and Midori (blue and green).

New hardware:

Xeon 3500 2.66GHz (40% to 80% faster)
12GB of memory (up from 8GB)
1.5TB RAID-5
Adaptec RAID controller with 256MB battery-backed cache
32GB Intel X25-E SSD (25x to 250x as fast as a disk)

New software:

64-bit Linux (we're currently on 32-bit)
Virtualisation of some sort (OpenVZ, Virtuozzo, or Citrix XenServer)

Posted by: Pixy Misa at 12:24 PM | Comments (7) | Add Comment | Trackbacks (Suck)
Post contains 85 words, total size 1 kb.

Nextwave*

* Not the Warren Ellis comic.  Sorry.

The current mu.nu / mee.nu server conglomeration consists of Aoi and Midori (blue and green), two Xeon 3220 systems with 8GB of memory and, well, a bunch of disk drives, and Sakura (pink), a Core 2 Duo 6500 system with 8GB of memory and a bunch of disk drives.

When Sakura went wobbly with a sprained disk a couple of weeks ago, I had to move some sites back to Midori, which then also went wobbly, albeit not because it was out of memory or CPU, but because it was out of Apache threads.  That caused me a few sleepless nights until I could get Sakura back in action.

There's plenty of free resources on Aoi, the mee.nu server, because Minx is so memory and CPU-efficient, but I can't easily drop random sites on there because the whole server is configured to run Minx.

That's why I got Sakura in the first place, or one of the reasons, the other being off-site backup in case something horrible happens.

Particularly around election time, I've wished that I could just upgrade the memory in Midori and install OpenVZ; that would solve a lot of problems.  And while I could install OpenVZ, upgrading the memory is not so easy: The motherboard supports a maximum of 8GB, which is kind of what we have already.

So I've been waiting, more-or-less patiently, for Nehalem-based servers to become available.  You see, the current Nehalem processors (Bloomfield and Gainestown) support three memory channels, and at least two DIMMs per channel, which means a single-processor Nehalem system can go to at least 12GB of memory.

And if the motherboard supports registered memory, you can get three DIMMs per channel and 4GB DIMMs, for a total of 36GB.

So at least 12, maybe 24 or even 36.  Any of which is an improvement on 8.

The server version of Nehalem - the Xeon 3500 (single processor) and 5500 (dual processor) ranges - are reportedly due out in exactly a week - March 29.  Intel haven't confirmed this so far as I know, but given that Apple is already shipping them in the new Mac Pro, the launch can't be too far off.

Apart from the improved memory support (up to 4.5x the capacity and 3x the bandwidth) over our current processors, the Nehalem delivers a sizeable performance boost.  On SpecInt a 2.66GHz Nehalem scores 28.6 vs 20.3 for the 2.4GHz 3220, a 40% increase, and on SpecIntRate, the multi-threaded version of the benchmark, it delivers 109 vs 59.2, more than an 80% increase.

The Nehalem supports hyperthreading, just like he good old Pentium 4, only this time it provides a much bigger performance boost.  A single 2.66GHz Nehalem scores about the same in multi-threaded benchmarks as a dual 2.33GHz Xeon of the previous model.  (On multi-threaded floating-point it does even better, about the equivalent of a dual 3GHz previous-model Xeon.  But I'm less interested in floating-point.)

So, that's memory and CPU covered.  What's next?

Well, last November we got an ioDrive in for testing at my day job.  These things are incredible; they deliver 120,000 random IOs per second.  Unfortunately, they're also incredibly expensive; the cheapest card costs $3000.

But SoftLayer now sells Intel's X25-E SSDs.  They're not as fast as an ioDrive, delivering "only" 30,000 read IOPS and 3,000 write IOPS, but that's still 25 to 250 times as fast as the disks we have right now.  We have one in our development server at my day job - I just finished setting it up - and it really seems to work as advertised.

Best part?  Only $50 per month.

So, 2.66GHz Nehalem, 12GB of RAM, 32GB X25-E.  What's next?

Next is RAID.  We're not running RAID at the moment, because when I set up the current servers I simply couldn't afford it - I have to pay for the RAID controller itself, and for twice the number of disks.  For each server.  Instead I have the servers doing a daily rsync to each other - which we'd still need to do, but with RAID we'd only need that for operator or software error, not for simple drive failures.

The good thing there is that SoftLayer's disk pricing is effectively half what it was when I set up the current servers, so I only have to pay for the controller.  So, hardware RAID it is, probably either 1TB RAID-1 or 1.5TB RAID-5.  RAID-5 is no good for databases, but the databases will be on the SSD, so it doesn't matter.

The one big variable is virtualisation.  I'd like to go Virtuozzo, because it's efficient and elegant (it's based on OpenVZ, but has some extra features and a nice control panel), and it integrates cleanly with Plesk.  (And offers substantial discounts on Plesk licenses - about 66%.)

The problem is, a 10-VM Virtuozzo license, the smallest practical size,
costs $100 per month.  Per server.  For that much money I could use Citrix XenServer (which is now free), add 6GB of memory to each server (to make up for Xen's less memory-efficient virtualisation model), and still save enough to pay for the more expensive Plesk license if I wanted it.

So, meh.  We'll see.

There's some other things I'm looking at - hardware firewalls (we sort of have one, but it's just there as a freebie and I don't have access to configure it); hardware load balancers (we use a software load balancer at mee.nu, which works very nicely as long as the server itself doesn't hiccup, but nothing at mu.nu right now); a content delivery network (which I already have, but need to configure); backups to iSCSI or CIFS storage (why don't they offer NFS? ).  They all cost money, though, so we'll see.

As to when?  First week of April.

Posted by: Pixy Misa at 11:29 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 971 words, total size 6 kb.

DIMM, I'm A Registered DIMM

And I dance dance dance, and I dance dance dance...

Okay, what's Pixy going on about now?

This puppy.

No, I'm not planning to buy one.  But in the details, you'll find that it's a single processor Nehalem workstation board, and supports both unbuffered and registered memory.

And what that means is that Akane and Mikan won't be limited to 12GB of memory, but will be able to reach at least 24GB and probably 36GB.

More on that in a moment.

Posted by: Pixy Misa at 10:57 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 86 words, total size 1 kb.

Ovbvious URL Is Obvious

http://downforeveryoneorjustme.com

Posted by: Pixy Misa at 04:47 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 5 words, total size 1 kb.

Delicious Meltiness

Both Order of the Stick and Erfworld are busy delivering crowning moments of awesome at the moment.  Unfortunately, that means that the GiantITP server has gone all melty.

Once you do get through, though, you'll find It's worth the wait.

Posted by: Pixy Misa at 06:36 PM | Comments (9) | Add Comment | Trackbacks (Suck)
Post contains 42 words, total size 1 kb.

Half A Yay

Good: Acid Pro 7.0b is out, which fixes the bug I found - it wouldn't let you produce AAC files, even though AAC is officially supported and documented.

Not quite so good: If you double-click on the resulting file to play it in Winamp - and Winamp isn't already running - it will crash.  If Winamp is already running, though, it will play just fine.  Works in iTunes and Quicktime, though, so I can't really blame that on Sony Creative.  I can test it by loading a WAV into iTunes and converting it there, and see how Winamp handles that.

Meh: Still can't output to Real Audio.  Not sure I care; I still have AAC, AC3, MP3, OGG, WMA, WAV and FLAC.

Posted by: Pixy Misa at 06:09 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 125 words, total size 1 kb.

History Is Accelerating: Now The First Time Is Farce

Or, This Post No Links

So, our present and bountifully incompetent federal government is bent on censoring this internet thingy, though it appears that they have yet to work out quite what it is.

The ACMA (which apparently stands for "petty-minded bureaucrats") has a list of a thousand-odd sites which are banned in Australia.

This list is secret.

Also, it's just a list.  The sites are banned, and viewing them is illegal, and linking to them incurs an $11,000-a-day fine, but no-one is allowed to know what is on the list, and the banned sites aren't actually blocked in any way.

So when, for example, popular Australian technical forum and news site Whirlpool found itself featuring a comment linking to a banned site, it also found itself facing a huge fine, and understandably removed the comment.

This even though the site in question has not been shown to be in breach of any law or regulation - except, perhaps, for the wonderfully vague clause other material dealing with intense adult themes, something that would be struck down as unconstituionally broad inside of thirty seconds by the US Supreme Court, bless 'em - and that the details of the site were made public by the original complainant, which is also not a breach of any law or regulation.

The secrecy of the list must be maintained, regardless.  And for that reason I can't link to Wikipedia any more either, because the Wikipedia article regarding the ACMA itself contains a link to a banned website.  (And was the subject of a 24-hour revert war as the inevitable result.)

We know some of what is on the list, of course, because after Wikileaks published similar secret censorship lists from other countries (Denmark, Finland, and Thailand, to be precise) Wikileaks was itself banned, and so was linking to the relevant pages on Wikileaks.  So I can't link to them.

Indeed, now we know all of what is on the list, because the list has been leaked to the banned Wikileaks.  Which, as I say, I cannot link to.

It gets better, though.

The leaked list (which, of course, I haven't seen, and won't reproduce, because to do so is to face ten years in jail, never mind the fines) reportedly contains over two thousand banned sites, some of which are reportedly legitimately illegitimate; others including tour operators, religious sites, online gambling sites, and one unfortunate dentist.  None of which, of course, can I link to.

I gets better still.

The Minister for Communication, the estimable Stephen Conroy, claims that the leaking of the list is irresponsible, illegal, and inaccurate - that it is not the real thing at all.

Wikileaks is still banned, of course, and it is still illegal for me to link to or reproduce the not-the-real-list.

So I won't.

I'll just link to the ABC, The Courier-Mail, The Brisbane Times, IT News, news.com.au, The Australian, Computerworld (twice), The Sydney Morning Herald, The Register, Slashdot, Forbes, Wired, ZDnet, Gizmodo, CNet,  Ars Technica, and Google News.

Posted by: Pixy Misa at 12:10 AM | Comments (7) | Add Comment | Trackbacks (Suck)
Post contains 512 words, total size 7 kb.

Truth Or Consequences

Two-valued (Boolean) logic is easy.  We recognise the basics of it from our everyday lives, and even if we can't immediately grasp some complex logical expression, we can work up to it from basic principles.

But when you start introducing more values, things get weird.

Take the simple if - then - else statement:

if a=b then:
  print "A = B".
else:
  print "A <> B".
end.

With two-valued logic, it couldn't be simpler.  If the expression is true, then you do the first bit, else you do the second bit.  But introduce null values, and hence logical nulls as a result of logical comparisons, and it starts to go pear-shaped.  If A is 1 and B is null, we find that this:

if a=b then:
  print "A = B".
else:
  print "A <> B".
end.

Prints A <> B, which is true from a two-valued perspective.  But this:

if a<>b then:
  print "A <> B".
else:
  print "A = B".
end.

Prints A = B, which is not true from a two-valued perspective.  What's going on?

What's going on is that the answer to the question, does A equal B, is neither true nor false.  It's null.  And so is the answer to the question, does A not equal B.  Since the then clause is taken if the if is true, either way, we take the else clause instead.

The problem is that if - then - else is implicitly two-valued.  What we need is a three-valued branch:

if A = B then:
  print "A = B".
else:
  print "A <> B".
otherwise:
  print "You have a null value somewhere".
end.

Yes, an if - then - else - otherwise structure will make any moderately experienced programmer choke on his cornflakes, but it's the logical consequence of the database null.

Well, that's ternary logic dealt with.  But what about yesterday's topic, quaternary logic?  Now we have four states: true, false, any, and none.  How should our if - then - else behave now?

Well, we can clarify that by looking at yet another variation on the names for the quaternary values: true, false, both and neither.  So the clear way to handle neither is not to take either of the branches - neither then nor else.

And the way to handle both is, well, to take both.

I can hear the anguished screams from here.

Posted by: Pixy Misa at 07:35 PM | Comments (8) | Add Comment | Trackbacks (Suck)
Post contains 391 words, total size 3 kb.

Small Wonder

Programmers have a lot of trouble with the concept of null, as found in relational databases.

There's a good reason for that. In regular boolean logic, there are 16 binary operators - the usual suspects like AND, OR, XOR, and implication and its converse; their negations; and the six degenerate forms that aren't truly binary at all - P, Q, their negations, and T and F (tautology and contradiction).

Okay?

There are 256 ternary boolean operators, as any hardcore Amiga programmer would know. (The Amiga's blitter allowed you to process three image sources into a destination, and you could program it to perform any one of the 256 possible operators.)

If you throw in null, so that you have three-valued, or ternary, logic, you now have 19,683 binary operators. (That is, binary ternary operators.  And 7,625,597,484,987 ternary ternary operators, which is why no-one made a three-valued Amiga.)

In 1990, Codd suggested that four-valued rather than three-valued logic was better for the relational model, dividing the concept of null into Missing but Applicable (A-values) and Missing but Inapplicable (I-values). I call them don't know and don't care, or none and any. (Because if you don't know and it matters, no value will give you the right answer, while if you don't care, any value will do.)

This does give you a more precise way of handling missing data. It also gives you 4,294,967,296 binary operators. That's probably why it hasn't really caught on yet.

Posted by: Pixy Misa at 12:30 AM | Comments (3) | Add Comment | Trackbacks (Suck)
Post contains 245 words, total size 2 kb.

Oh. Is That It?

So that's how R-Trees work.

Sure, it's useful, and generalises well, but they're just trees.

Posted by: Pixy Misa at 02:19 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 19 words, total size 1 kb.

Guck

I forgot why I went with MySQL rather than PostgreSQL for Minx, when PostgreSQL is so much more flexible and robust.

Full-text indexing in PostgreSQL is fucking hideous.

Posted by: Pixy Misa at 04:40 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 29 words, total size 1 kb.

Time To Die...

The latest bizarre failure cascade (a hiccup in a single MySQL table on one server leading to a completely different server becoming unbootable) made me think of this:

I've seen Sun monitors on fire off the side of the multimedia lab.  I've seen NTU lights glitter in the dark near the Mail Gate.  All these things will be lost in time, like the root partition last week.

Time to die...

Coda: And here's me going aargh, now I can't even post to my own blog on my own software on my own server...  Oh, yeah, 7PM exactly, the backup just kicked in.  One minute later - fine.

Posted by: Pixy Misa at 06:03 PM | Comments (3) | Add Comment | Trackbacks (Suck)
Post contains 109 words, total size 1 kb.

Offsite Backups

After our recent brush with disaster, I've been paying more attention to backups, including properly setting up the offsite backup server to do offsite backups.

There are, at last count, 5,172,047 user files on the main mu.nu server.

Thanks go to Movable Type and maildir.  I hate maildir.

Posted by: Pixy Misa at 10:34 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 50 words, total size 1 kb.

Ah, Bind Mounts

Of course.

The new mu.nu and mee.nu servers - probably arriving next month, depending on Intel and the weather - are going to be running on a virtualised platform.  Current contenders are Xen and XenServer (which provide better isolation between virtual nodes) and OpenVZ and Virtuozzo (which provide better efficiency).*

The way you set up either platform is pretty similar: You allocate a big bucket of disk space (which had better be RAID, or you risk losing everything at once), and then you create your virtual environments in that bucket, granting them certain amounts of disk, memory, and CPU resources.

Which is easy to configure and works fine for a basic setup.  But one of the other new things about the new servers is that I'm going to be installing SSDs - Intel X25-E SLC drives, to be precise, which deliver 3,000 write IOPS and 30,000 read IOPS, which is a whole bunch faster than anything we have at the moment.

The SSDs will be used only for databases; they're far too expensive for general storage.  But if the general storage for the virtual nodes is allocated from the big storage bucket, how do I point databases at the SSDs?

The answer - at least for OpenVZ and Virtuozzo - is something called bind mounts.  This is a new Linux kernel trick which allows you to mount any existing directory as a filesystem elsewhere on the server.  With OpenVZ, that lets me mount a particular directory on the SSD as a filesystem within a particular virtual node - exactly what I need.

So I can, as needed, split off a particular blog (or group of blogs) into its (their) own virtual server with its own specific configuration of Linux and Apache and MySQL and so on. 

The only catch is that CPanel costs $12 a month per virtual server.**  I can run as many sites as I like under each virtual server, but each server that needs CPanel is another $12.

Minx doesn't need CPanel, of course; it doesn't even use Apache.  Because the mee.nu server is specifically set up for Minx, though, I couldn't put Protein Wisdom on there while the other server was being fixed, so it ended up overloading the main mu.nu server.  Having virtual nodes will make it hugely easier for me to move things around like that.

Virtuozzo is the commercial version of OpenVZ, and it offers some nice extra features, including integration with the Plesk control panel (a competitor to CPanel, and a pretty good one).  The problem is, OpenVZ is free, while Virtuozzo is licensed per virtual server per month.  A three-VPS*** license runs $60 a month; a ten-VPS license $100, which is more reasonable per node, but not exactly cheap.

A 100-user VPS license for Plesk is $10 per month, compared to $30 per month for a hardware server license.  But only if the VPS is running on Virtuozzo, whereas the CPanel license is the same regardless of what virtualisation platform you're on.  And while Virtuozzo is nice and offers a control panel integrated with Plesk, I don't really have any users who need that.

So right now it looks like it'll be OpenVZ.  This weekend I'll be setting up a test server to play around with it; there are some issues with both RedHat 5 (the kernel is fairly old) and Fedora 10 (some libraries are too new) which caused me problems when I first tried it a couple of months ago.  I need to get all that sorted out quickly so that we can move forward into our shiny virtual future.

* That's a trade-off.  If you want to say, this 2GB of memory belongs to this virtual node and no-one else can use it, then that means that node is protected from memory contention from other nodes.  But if it only uses 1GB of memory, the other gig is wasted.  Xen is oriented more toward isolation, OpenVZ towards efficiency.

** Including Fantastico.

*** Virtual private server.

Posted by: Pixy Misa at 01:10 AM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 670 words, total size 4 kb.

Okay, Let's Give This A Whirl

Parsing time, only different:

mp3 - check!
ac3 - check!
wav - check!
wma - check!
flac - check!
ogg - bzzt

avi - check!
mkv - check!
mpeg - check!
mov - check!
mp4 - check!
ogm - bzzt

I'm beginning to sense a pattern here.  But if I can get the ogg handling sorted out (and it's possible my test files are crappy), kaa.metadata will prove very useful.

Update: Got the latest version from SVN, and everything works except one ogg file I generated myself (my own composition, from Acid Pro).  That file does play successfully in WinAmp, though, so there is still a bug or two there.

Posted by: Pixy Misa at 01:15 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 118 words, total size 1 kb.