Pitafied

A while back, in between houses falling on me, I was working in a database written in Python, which I called Pita.  I actually got it working, enough to start doing some performance tests...

At which point I shelved the project, because (a) I was absurdly busy what with the houses and all and (b) even though it had pluggable low-level storage engines, the overhead of the Python layer made it significantly slower than just using MySQL.

What Pita could do, which was nice, was (a) offer a choice of in-memory or on-disk tables using identical syntax and selectable semantics and (b) provide a log-structured database that did sequential writes for random updates.  Cassandra also has this trick.  The advantage here is that it (a) can cope with a huge volume of incoming data, and (b) doesn't fry consumer-grade SSDs the way MySQL would.

Unfortunately, Cassandra is a bit of a cow.  Undeniably useful, but indubitably bovine.

Redis with AOF can offer similar performance, but only so long as your data fits in memory, because it's simply snapshot+log persistence (like Pita) and single threaded (unlike Pita) so it can't cope with I/O delays.  This makes Redis and its support for data structures beyond simple records (hashes, lists, sets, sorted sets) great for your hot data but no use for your long tail - if, say, you've been running a blogging service for 8 years.

What you could do in that situation is use Redis for your hot data (great performance, easy backups, easy replication) and stick your cold data in a key-value store.

Like Keyspace, except that's dead.
Or Cassandra, except that's a cow.
Or MySQL, except that defeats the purpose.
Or MongoDB, except that you'd like to keep your data.

Or Kyoto Tycoon, which has pluggable APIs (don't like REST - use RPC or memcached protocol)  and pluggable storage engines...  Like Google's LevelDB.  Kyoto Tycoon running Kyoto Cabinet uses snapshot+log for backups, but the database itself is a conventional B+ tree, so it needs to do random writes.  LevelDB, on the other hand, uses log-structured merge trees - sequential writes, even for the indexes.

So Redis and Kyoto Tycoon with LevelDB both provide:

• Key-value store
• Range lookups
  
• Sequential writes (SSD friendly)
  
• Snapshot+log backups (bulletproof)
• Instant replication (just turn it on, unlike MySQL replication, which is a pain)
• Lua scripting (not yet in mainstream Redis, but coming)
• Key expiry (for caching)

Redis also provides:

• Data structures
• Hashes
• Lists (which can be used to provide stacks, queues, and deques)
  
• Sets
• Sorted sets
• Bitfields
• Bytestrings (update-in-place binary data)
  
• Pub/Sub messaging

And Kyoto Tycoon provides:

• Support for databases larger than memory
• Very fast data loads
  

Together they make a very powerful team.

Posted by: Pixy Misa at 03:11 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 450 words, total size 3 kb.