Sunday, January 22

Geek

Mongo Revisited

When I first met MongoDB, I was unimpressed, because in my testing it very quickly died and lost all my data.  The proximate cause for this was that I was running it under OpenVZ, which will switfly kill any process that runs out of memory (which Mongo did).  The reason MongoDB ran out of memory is that... 

Well, it didn't, not really; MongoDB works by memory-mapping the entire database and treating it as a persistent data structure, relying on the operating system to provide the persistence layer.  One problem with that is that an inopportune event at an inopportune moment can leave you with a pile of unreadable crap where your database used to be.  And another problem is that OpenVZ treated it as having run out of memory and killed it...  Which meant that on my test bench server - which runs OpenVZ precisely so that I can test things - MongoDB could be consistently made to crash and corrupt itself on small but realistic workloads.

/images/MongoDB.jpg
MongoDB standalone.  What's the worst that could happen?

Contrast that with CouchDB, the Redis AOF, or Riak's Bitcask, which are all append-only and pretty much bullet-proof: If the entire server crashes before it can write a record, well, you lost that record.  But short of going in manually and deleting files, that's the worst that can happen.

So the problem I had was that while MongoDB had the closest semantic fit of all databases I'd seen to what I was trying to achieve with Minx, handing your data to it was like handing your collection of Wedgwood china to an inebriated juggling troupe - it's only a question of when.  You could replicate, but then you'd have to make damned sure that the same problem didn't happen to both copies.  And running it on OpenVZ was like handing your collection of Wedgwood china to an inebriated juggling troupe - and then setting your house on fire.

/images/MongoDBReplication.jpg
MongoDB replicated.  All your eggs in two (four?) identical baskets?

MongoDB now has (has for two releases, actually) a journalling facility that will replay lost writes after a power/hardware/software failure. On the whole  I'd rather have a persistence mechanism that was  inherently safe than an unsafe mechanism with a bungee cord attached for when it inevitably runs into trouble.  But while diving off a perfectly serviceable bridge never struck me as a particularly bright idea in the first place, diving off a bridge while firmly attached to it with a bungee cord is rather less likely to end in tears and crocodiles.

Still, if you wanted to run MongoDB, you needed either full-stack virtualisation, a ton of memory, or a dedicated server.  For me, the obvious solution is to add a ton of memory and leave the rest of the architecture intact.  The problem with being hosted at SoftLayer is that while they offer great support and a great network, their pricing is not so great, and their memory pricing is abominable; a ton of memory costs about three tons of money.*

Apparently that's also now been fixed from the other end: The latest versions of OpenVZ (based on the Red Hat Enterprise Linux 6 branch) bring with them a feature labeled VSwap (virtual swap), which both simplifies and flexiblises memory management and keeps MongoDB under control without mandating either a fiscal or architectural arrow to the knee.

/images/MongoDBOnOpenVZ.jpg
Hurrah!  MongoDB and OpenVZ!

But our current server is running on RHEL 5 - actually, I think it's CentOS 5, but basically the same thing - so that requires a reinstall.  And if we're going to do that, we'd want to build a new server, test it, and swap over once it's all working.  And if we're going to do that, we'd be best advised to wait for a hardware refresh, which didn't happen at all for mid-range Intel servers in 2011, and still won't happen for another couple of months.

Which means...  I'll play around with MongoDB a bit more, I think.

* I'm looking into alternatives if they don't smarten up their game, and quickly.

Posted by: Pixy Misa at 05:32 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 680 words, total size 5 kb.

Comments are disabled. Post is locked.
49kb generated in CPU 0.0155, elapsed 0.1531 seconds.
54 queries taking 0.1434 seconds, 324 records returned.
Powered by Minx 1.1.6c-pink.