Saturday, February 13

Geek

Rhymes With Camel

I was looking at YAML as a serialisation option for Pita (it's already supported for exporting data in the development version of Minx).

So I ran some benchmarks.

It's sloooooooooooooooooooow.

Here we are, encoding and decoding a 2k record:

[andrew@eineus ~]$ python jsonbench.py
10000 iterations on 1973 bytes
Python
json: 10000 encodes+decodes in 4.9 seconds, 2055.8 per second
simplejson: 10000 encodes+decodes in 0.5 seconds, 20686.2 per second
pickle: 10000 encodes+decodes in 4.3 seconds, 2338.0 per second
cPickle: 10000 encodes+decodes in 0.6 seconds, 17297.6 per second
yaml: 10000 encodes+decodes in 213.4 seconds, 46.9 per second

json is Python's built-in JSON library, which is written in Python and thus somewhat sluggish.  simplejson is the same JSON library with an optional C implementation.  Essentially the same applies for pickle vs. cPickle.

yaml is PyYAML, which includes a C implemenation if you have LibYAML installed.  Which I do, but I can't seem to get the C implemenation to run...  Unless that's it, which would be pretty sad.

On the one hand, simplejson is the fastest of these options, which is good because it's also the most widely supported format and the easiest to parse.

One the other hand, 20,000 records per second is not all that much.

Posted by: Pixy Misa at 04:37 PM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 205 words, total size 1 kb.

1 > One the other hand, 20,000 records per second is not all that much.

have a look at yajl: http://lloyd.github.com/yajl/
it is one of the fastest json parsers / encoders out there. not sure if there are any python bindings, though ...

Posted by: tim at Monday, February 15 2010 07:41 AM (aIgXN)

2 Thanks for the tip!  And yes, there's a Python binding. smile

Posted by: Pixy Misa at Monday, February 15 2010 08:18 AM (PiXy!)

Hide Comments | Add Comment

Comments are disabled. Post is locked.
46kb generated in CPU 0.0133, elapsed 0.1093 seconds.
56 queries taking 0.1014 seconds, 327 records returned.
Powered by Minx 1.1.6c-pink.