Saturday, July 09

Pitafied
A while back, in between houses falling on me, I was working in a database written in Python, which I called Pita. I actually got it working, enough to start doing some performance tests...
At which point I shelved the project, because (a) I was absurdly busy what with the houses and all and (b) even though it had pluggable low-level storage engines, the overhead of the Python layer made it significantly slower than just using MySQL.
What Pita could do, which was nice, was (a) offer a choice of in-memory or on-disk tables using identical syntax and selectable semantics and (b) provide a log-structured database that did sequential writes for random updates. Cassandra also has this trick. The advantage here is that it (a) can cope with a huge volume of incoming data, and (b) doesn't fry consumer-grade SSDs the way MySQL would.
Unfortunately, Cassandra is a bit of a cow. Undeniably useful, but indubitably bovine.
Redis with AOF can offer similar performance, but only so long as your data fits in memory, because it's simply snapshot+log persistence (like Pita) and single threaded (unlike Pita) so it can't cope with I/O delays. This makes Redis and its support for data structures beyond simple records (hashes, lists, sets, sorted sets) great for your hot data but no use for your long tail - if, say, you've been running a blogging service for 8 years.
What you could do in that situation is use Redis for your hot data (great performance, easy backups, easy replication) and stick your cold data in a key-value store.
Like Keyspace, except that's dead.
Or Cassandra, except that's a cow.
Or MySQL, except that defeats the purpose.
Or MongoDB, except that you'd like to keep your data.
Or Kyoto Tycoon, which has pluggable APIs (don't like REST - use RPC or memcached protocol) and pluggable storage engines... Like Google's LevelDB. Kyoto Tycoon running Kyoto Cabinet uses snapshot+log for backups, but the database itself is a conventional B+ tree, so it needs to do random writes. LevelDB, on the other hand, uses log-structured merge trees - sequential writes, even for the indexes.
So Redis and Kyoto Tycoon with LevelDB both provide:
Comments are disabled.
Post is locked.
A while back, in between houses falling on me, I was working in a database written in Python, which I called Pita. I actually got it working, enough to start doing some performance tests...
At which point I shelved the project, because (a) I was absurdly busy what with the houses and all and (b) even though it had pluggable low-level storage engines, the overhead of the Python layer made it significantly slower than just using MySQL.
What Pita could do, which was nice, was (a) offer a choice of in-memory or on-disk tables using identical syntax and selectable semantics and (b) provide a log-structured database that did sequential writes for random updates. Cassandra also has this trick. The advantage here is that it (a) can cope with a huge volume of incoming data, and (b) doesn't fry consumer-grade SSDs the way MySQL would.
Unfortunately, Cassandra is a bit of a cow. Undeniably useful, but indubitably bovine.
Redis with AOF can offer similar performance, but only so long as your data fits in memory, because it's simply snapshot+log persistence (like Pita) and single threaded (unlike Pita) so it can't cope with I/O delays. This makes Redis and its support for data structures beyond simple records (hashes, lists, sets, sorted sets) great for your hot data but no use for your long tail - if, say, you've been running a blogging service for 8 years.
What you could do in that situation is use Redis for your hot data (great performance, easy backups, easy replication) and stick your cold data in a key-value store.
Like Keyspace, except that's dead.
Or Cassandra, except that's a cow.
Or MySQL, except that defeats the purpose.
Or MongoDB, except that you'd like to keep your data.
Or Kyoto Tycoon, which has pluggable APIs (don't like REST - use RPC or memcached protocol) and pluggable storage engines... Like Google's LevelDB. Kyoto Tycoon running Kyoto Cabinet uses snapshot+log for backups, but the database itself is a conventional B+ tree, so it needs to do random writes. LevelDB, on the other hand, uses log-structured merge trees - sequential writes, even for the indexes.
So Redis and Kyoto Tycoon with LevelDB both provide:
- Key-value store
- Range lookups
- Sequential writes (SSD friendly)
- Snapshot+log backups (bulletproof)
- Instant replication (just turn it on, unlike MySQL replication, which is a pain)
- Lua scripting (not yet in mainstream Redis, but coming)
- Key expiry (for caching)
- Data structures
- Hashes
- Lists (which can be used to provide stacks, queues, and deques)
- Sets
- Sorted sets
- Bitfields
- Bytestrings (update-in-place binary data)
- Pub/Sub messaging
- Support for databases larger than memory
- Very fast data loads
Posted by: Pixy Misa at
03:11 PM
| No Comments
| Add Comment
| Trackbacks (Suck)
Post contains 450 words, total size 3 kb.
46kb generated in CPU 0.0147, elapsed 0.1901 seconds.
54 queries taking 0.1805 seconds, 332 records returned.
Powered by Minx 1.1.6c-pink.
54 queries taking 0.1805 seconds, 332 records returned.
Powered by Minx 1.1.6c-pink.