Saturday, July 09



A while back, in between houses falling on me, I was working in a database written in Python, which I called Pita.  I actually got it working, enough to start doing some performance tests...

At which point I shelved the project, because (a) I was absurdly busy what with the houses and all and (b) even though it had pluggable low-level storage engines, the overhead of the Python layer made it significantly slower than just using MySQL.

What Pita could do, which was nice, was (a) offer a choice of in-memory or on-disk tables using identical syntax and selectable semantics and (b) provide a log-structured database that did sequential writes for random updates.  Cassandra also has this trick.  The advantage here is that it (a) can cope with a huge volume of incoming data, and (b) doesn't fry consumer-grade SSDs the way MySQL would.

Unfortunately, Cassandra is a bit of a cow.  Undeniably useful, but indubitably bovine.

Redis with AOF can offer similar performance, but only so long as your data fits in memory, because it's simply snapshot+log persistence (like Pita) and single threaded (unlike Pita) so it can't cope with I/O delays.  This makes Redis and its support for data structures beyond simple records (hashes, lists, sets, sorted sets) great for your hot data but no use for your long tail - if, say, you've been running a blogging service for 8 years.

What you could do in that situation is use Redis for your hot data (great performance, easy backups, easy replication) and stick your cold data in a key-value store.

Like Keyspace, except that's dead.
Or Cassandra, except that's a cow.
Or MySQL, except that defeats the purpose.
Or MongoDB, except that you'd like to keep your data.

Or Kyoto Tycoon, which has pluggable APIs (don't like REST - use RPC or memcached protocol)  and pluggable storage engines...  Like Google's LevelDB.  Kyoto Tycoon running Kyoto Cabinet uses snapshot+log for backups, but the database itself is a conventional B+ tree, so it needs to do random writes.  LevelDB, on the other hand, uses log-structured merge trees - sequential writes, even for the indexes.

So Redis and Kyoto Tycoon with LevelDB both provide:
  • Key-value store
  • Range lookups
  • Sequential writes (SSD friendly)
  • Snapshot+log backups (bulletproof)
  • Instant replication (just turn it on, unlike MySQL replication, which is a pain)
  • Lua scripting (not yet in mainstream Redis, but coming)
  • Key expiry (for caching)
Redis also provides:
  • Data structures
    • Hashes
    • Lists (which can be used to provide stacks, queues, and deques)
    • Sets
    • Sorted sets
    • Bitfields
    • Bytestrings (update-in-place binary data)
  • Pub/Sub messaging
And Kyoto Tycoon provides:
  • Support for databases larger than memory
  • Very fast data loads
Together they make a very powerful team.

Posted by: Pixy Misa at 03:11 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 450 words, total size 3 kb.

Comments are disabled. Post is locked.
46kb generated in CPU 0.0147, elapsed 0.1901 seconds.
54 queries taking 0.1805 seconds, 332 records returned.
Powered by Minx 1.1.6c-pink.