Couch Tomato

So I did a bunch of reading on CouchDB, and it seems pretty interesting, and while flawed, it's not flawed in the direction of oops I just lost all your data (I'm looking at you, MongoDB) and it's not like I've found a database without flaws, so it's worth a shot.

Only none of the available Python libraries supports all the functionality of CouchDB, and one of the main ones is just abysmally slow. But on the other hand, one of the main weaknesses of CouchDB - it only has a REST API - means that it's easy to write a library - after all, it only has a REST API.

So I'm doing exactly that. The goal is to have a simple but complete library with the best performance I can manage. It will cover all the REST API calls but not try to do anything beyond that. I've isolated out the actual call mechanism so that it can be trivially swapped out if someone comes up with something more efficient. (Or if Couchbase creates a binary API.)

Soon as I have it mostly working I'll open source it and put it up on Bitbucket or something.

As it stands, you can connect to CouchDB, authenticate, create and delete databases, and read and write records, as well as various housekeeping thingies that were easy to implement. It supports convenient dict-style access db[key] as well as db.get() and db.set().

It's called Settee, unless I think of something better. It's a sort of spartan couch.

And in a simple test of 1000 separate record inserts:

Settee: 3.668s
CouchDB-Python: 41.53s

Either one would be much faster using the batch update functions, but there's no reason to grind to a complete halt when you need to do many smal queries.

(Scrabbling sounds ensue for a couple more hours.)

Okay.

CouchDB-Python is slow specifically for record creates and updates; it seems to be fine otherwise. There's a fork called CouchDB-Python-Curl that is much faster on writes - slightly faster than my code too - and uses significantly less CPU time apparently because it uses libcurl, which is written in C. Only problem there is that libcurl is horrible.

I've tweaked my code now so that it supports both the stylish and elegant Requests module and the simple and functional Httplib2, both of which are miles ahead of Python's core library in functionality and PyCurl in friendliness. PyCurl is still the fastest, though:

	Write/s	Read/s	User	Sys	Real
CouchDB-Python	24	603	1.248	1.108	85.7
CouchDB-Psyco	24	654	1.112	0.848	85.3
CouchdDB-Curl	422	665	0.536	0.368	7.93
CouchdDB-Curl-Psyco	421	711	0.576	0.452	7.93
Settee-Requests	272	418	2.704	1.564	12.4
Settee-Requests-Psyco	305	453	1.96	1.47	11.2
Settee-Httplib2	376	636	0.98	0.784	8.48
Settee-H2-Psyco	392	687	0.6	0.77	8.48

CouchDB-Curl runs only slightly faster than Settee-H2, but uses about half the CPU time, which is a nice advantage. Running with the Psyco JIT compiler makes up much of the difference - speeding up Settee while increasing the system time on CouchDB-Curl. Can't do anything for poor CouchDB-Python, though. I think it's been Nagled, though I don't know why it only affects that part of the test.

Update: Bulk insert of records, 100 at a time, single-threaded:

Settee-Rq: ~7000
Settee-H2: ~7700

That's healthy enough.  Still trying to get the bulk read working the way I want it, though.

Update: Okay, so there's a fair bit of overhead in the API - write queries seem to take a minimum of 2.5ms and read queries at least 1.5ms.  (MySQL queries get down to about 0.5ms, and Redis is even faster.) 

But the underlying data engine suffers from no such limitations: I'm firing 20 million small (~250 byte) JSON objects at it, single-threaded, batches of 100, and it's happily loading them at about 6,000 records per second, 5 million records in.  That's with CouchDB at about 60% of a CPU, Python at about 6%, and disk (6-disk software RAID-5) at about 10% busy.

That's a pretty good showing, though I'm not stressing it at all yet - the data is being loaded in key order.  We'll have to see how it copes with multiple compound indexes next.

Posted by: Pixy Misa at 05:52 PM | Comments (3) | Add Comment | Trackbacks (Suck)
Post contains 698 words, total size 7 kb.