The Cabinet Of Doctor Kyoto

Kyoto Cabinet is the successor (more or less) to Tokyo Cabinet, and like it, is a fast, modern, embedded database, not unlike Berkeley DB.

How fast can we stash movies in that, I wonder?

Method	Elapsed	User	System
Kyoto Cabinet/JSON	1.23	1.18	0.05
Kyoto Cabinet/MsgPack	1.65	1.19	0.45
SQL Individual Insert	10.4	4.1	1.8
SQL Bulk Insert	2.1	1.6	0.04
SQL Alchemy	33.2	26.4	2.1
SQL Alchemy Bulk	33.4	26.8	1.9
Elixir	35.8	28.7	2.3

Pretty darn fast, is the answer. That's a little unfair, since Kyoto Cabinet is an embedded database, not a database server.

(Interestingly, SimpleJSON outran MessagePack in this test, and for a very odd reason - system time went up ninefold.)

Oh, and one other thing - the random movie generator itself takes about 1s of that 1.23s.  So it's not 30x faster than Elixir here, it's more like 100x faster.

Kyoto Tycoon is a server layered atop Kyoto Cabinet, so I'm off to tinker with that next.

Update: 0.43 seconds to create a list of 100,000 random movies; 0.52 seconds to pack them all as JSON; 0.11 seconds to write them to a BTree-indexed Kyoto Cabinet database.  I've seen benchmarks that put Kyoto Cabinet at over a million records per second, and here it is running in a virtual machine under Python, single-threaded, at ~900,000.  So, yeah, it's quick.

Update:  Teehee!

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
11509 pixy      17   0  512m  82m 3932 S 237.4  0.2   0:39.00 python
Why yes, that Python program is using 2.37 CPUs.

It looks like two (busy) threads are the limit for Kyoto Cabinet, though; at four threads throughput actually went down by about 20%.

Posted by: Pixy Misa at 08:49 AM | Comments (1) | Add Comment | Trackbacks (Suck)
Post contains 289 words, total size 4 kb.