Sunday, January 16

Geek

The Cabinet Of Doctor Kyoto

Kyoto Cabinet is the successor (more or less) to Tokyo Cabinet, and like it, is a fast, modern, embedded database, not unlike Berkeley DB.

How fast can we stash movies in that, I wonder?

Method
Elapsed
User
System
Kyoto Cabinet/JSON
1.23
1.18
0.05
Kyoto Cabinet/MsgPack
1.65
1.19
0.45
SQL Individual Insert
10.4
4.1
1.8
SQL Bulk Insert
2.1
1.6
0.04
SQL Alchemy
33.2
26.4
2.1
SQL Alchemy Bulk
33.4
26.8
1.9
Elixir
35.8
28.7
2.3

Pretty darn fast, is the answer. That's a little unfair, since Kyoto Cabinet is an embedded database, not a database server.

(Interestingly, SimpleJSON outran MessagePack in this test, and for a very odd reason - system time went up ninefold.)

Oh, and one other thing - the random movie generator itself takes about 1s of that 1.23s.  So it's not 30x faster than Elixir here, it's more like 100x faster.

Kyoto Tycoon is a server layered atop Kyoto Cabinet, so I'm off to tinker with that next.

Update: 0.43 seconds to create a list of 100,000 random movies; 0.52 seconds to pack them all as JSON; 0.11 seconds to write them to a BTree-indexed Kyoto Cabinet database.  I've seen benchmarks that put Kyoto Cabinet at over a million records per second, and here it is running in a virtual machine under Python, single-threaded, at ~900,000.  So, yeah, it's quick. smile

Update:  Teehee!

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
11509 pixy      17   0  512m  82m 3932 S 237.4  0.2   0:39.00 python

Why yes, that Python program is using 2.37 CPUs. grin

It looks like two (busy) threads are the limit for Kyoto Cabinet, though; at four threads throughput actually went down by about 20%.

Posted by: Pixy Misa at 08:49 AM | Comments (1) | Add Comment | Trackbacks (Suck)
Post contains 289 words, total size 4 kb.

Comments are disabled. Post is locked.
43kb generated in CPU 0.06, elapsed 0.2738 seconds.
51 queries taking 0.2471 seconds, 275 records returned.
Powered by Minx 1.1.6c-pink.