CAN I BE OF ASSISTANCE?

Take That You Talking Tuna

Posted by: Pixy Misa at 03:46 AM | Comments (7) | Add Comment | Trackbacks (Suck)
Post contains 5 words, total size 1 kb.

He Was A Visionary, You Know

Those who would give up Essential Liberty to purchase a little Internet Bandwidth, deserve a Kick in the Crotch.

Benjamin Franklin

Posted by: Pixy Misa at 12:36 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 27 words, total size 1 kb.

Language

German is without question the best language for describing technical faults:

Das Archiv dieser Gruppe ist momentan nicht verfügbar

Wir entschuldigen uns für eventuell entstandene Unannehmlichkeiten. Bitte versuchen Sie es in Kürze noch einmal

Knowing that the Archiv is momentan nicht verfügbar makes it seem better, somehow.

Posted by: Pixy Misa at 05:07 PM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 46 words, total size 1 kb.

Desperately Seeking The Last Six Months

About six months ago, I was looking for a database that didn't suck.

At the time, there weren't any, though some showed promise.  Redis, for example, had a great feature set but was limited to in-memory databases.

This has now been fixed.

Fortunately (or... otherwise) my day job ate all my time for six months, and now that I'm finally returning to this project, Redis is ready for me.

While it's not the solution for every problem (the main data engine is single-threaded and blocking, so your overall throughput is limited by CPU speed) it now supports virtual memory and multi-threaded I/O, so it can scale to significantly larger datasets.

The big advantage of Redis is that it provides lightweight implementations of a variety of different data storage methods - not just key/value stores, but lists, hashes, sets, ordered sets, and message queues.  You can also set expiry times on entries to use it as a cache.  So with a bit of lateral thinking one server can replace a regular database like MySQL, a messaging system like ActiveMQ, a caching system like Memcached, and a document store like CouchDB.

It doesn't support strict transactions, but it has transactional features that are good enough for most tasks.  It also has a nice simple replication model.

You're not going to build the next Facebook on Redis - it simply won't scale - but if you're working on something a bit smaller (like a multi-user blogging app) it might be just what you need.

Not that I'm thinking of rewriting Minx - but the new module I'm developing is going to use Redis for most of its data, and if that experiment works well, I'll naturally look at extending it further.

Update: Huh.

So, as is my wont (and as is the wont of any programmer who wishes to retain what's left of his or her sanity) I ran a benchmark on Redis before going any further.  My own benchmark.  Redis comes with a benchmark that cheerfully reports ~100,000 reads or writes per second.  I do not trust it.

So:

[andrew@synclavier ~]$ python redistest.py
String write * 10000: 0.58s 17097.4/s
String read * 10000: 0.54s 18436.4/s
JSON write * 10000: 1.13s 8869.5/s
JSON read * 10000: 0.72s 13799.1/s
Hash write * 10000: 2.24s 4455.1/s
Hash read * 10000: 5.42s 1845.1/s

For the string benchmark, I am creating records each containing ~1kb of static text.  For the JSON benchmark, a Python dict is converted to ~1kb of JSON text.  For the hash benchmark, the same dict is stored as a Redis hash - basically, an extensible record structure.

String performance is not to bad for a single-threaded client/server code with no optimisations - approaching 20k ops/s for both reads and writes with a meaningful payload.  Good.

The JSON benchmark is reading and writing exactly the same amount of data, showing the overhead of the JSON (actually, SimpleJSON) library.  We still get over on the order of 10k ops/s.  With MySQL on the production server, that's what I see on cached queries.  So far, so good.

The hash benchmark is not so hot, particularly the reads.  An order of magnitude slow that strings; most of an order of magnitude slower than JSON for the same data.  Won't be using hashes for the high-volume stuff.

Then, curious, I tried one more thing:

Object write * 10000: 0.84s 11846.0/s
Object read * 10000: 0.54s 18473.1/s

I took that same Python dict and wrote it to Redis.  Without converting it to JSON.  Without mapping it to a hash.  Without doing anything.  And it worked.

That's kind of scary.  I'm not sure how it's doing that.

If you pass the Python Redis library something that's not a string, it turns it into a string.  No magic.

Update: Tweaked and re-run, with some interesting results:

Running native
String write * 10000: 0.66s 15075.6/s
String read * 10000: 0.52s 19051.1/s
JSON write * 10000: 1.07s 9359.8/s
JSON read * 10000: 0.72s 13905.8/s
JSON multi-read * 10000: 0.50s 19878.3/s
Hash write * 10000: 2.64s 3788.8/s
Hash read * 10000: 6.11s 1636.9/s
Pickle write * 10000: 0.88s 11335.1/s
Pickle read * 10000: 0.68s 14810.8/s

Running with Psyco
String write * 10000: 0.68s 14742.0/s
String read * 10000: 0.41s 24369.1/s
JSON write * 10000: 0.88s 11398.6/s
JSON read * 10000: 0.69s 14493.1/s
JSON multi-read * 10000: 0.39s 25882.6/s
Hash write * 10000: 2.06s 4845.8/s
Hash read * 10000: 2.66s 3752.5/s
Pickle write * 10000: 0.89s 11212.0/s
Pickle read * 10000: 0.63s 15953.6/s

With Psyco, hash write performance is up 20% and hash read performance is up over 100%.  That certainly suggests the bottleneck is not Redis itself.

Posted by: Pixy Misa at 06:00 PM | Comments (1) | Add Comment | Trackbacks (Suck)
Post contains 778 words, total size 6 kb.

Blurgleblub

Been one of those weeks.

Anyway, on the previous subject, this puppy would likely be a little more budgetarily feasible:

Supermicro SC111LT-330 1U Chassis $215
Supermicro H8SCM-F Motherboard $235
AMD Opteron 4180 CPU $206
4GB DDR3-1333 ECC Registered RAM x2 $276
Seagate Momentus 7200.4 500GB x 3 $261
Intel 80GB X25-M SSD $249
Supermicro Slim DVD-ROM $55

Total: $1497

That's for a 6-core 2.6GHz processor, 8GB RAM, 1TB available disk in RAID-5, and a nice 80GB SSD.  Dual GbE and IPMI for remote management.

Alternately, this:

Supermicro SC111LT-330 1U Chassis $215
Supermicro X8SIE-LN4F Motherboard $225
Intel Xeon X3430 CPU $219
4GB DDR3-1333 ECC Registered RAM x2 $276
Seagate Momentus 7200.4 500GB x 3 $261
Intel 80GB X25-M SSD $249
Supermicro Slim DVD-ROM $55

Total: $1500

Very similar overall; this CPU is a quad-core 2.4GHz, but with a somewhat better architecture and hyper-threading, so performance would be close to the AMD chip.  This motherboard comes with quad GbE, so I can have lots of fun inventing weird network topologies.

Either one would run Minx with no problems at all; they wouldn't run the CPanel accounts, though; Apache and PHP chew up memory and CPU like nobody's business.  There's not much expansion space, though, except for RAM - up to 32GB of affordable modules, and on the AMD board, up to 64GB of really expensive ones.

Posted by: Pixy Misa at 10:09 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 225 words, total size 2 kb.

The Possibilities...

Supermicro SC217 Chassis $919
Supermicro H8DGT Motherboard x4 $1700
AMD Opteron 6128 CPU x8 $2288
4GB DDR3-1333 ECC Registered RAM x32 $4256
Seagate Momentus XT 500GB x24 $3240

Total: $12,403

Okay, that's a lot of money, yes.

But what you get for that is a 64-processor server with 128GB of RAM and 12TB of disk with a 96GB non-volatile cache.

Which fits in 2U of rack space.

It actually has room for 48 CPUs and 512GB of RAM, but that would blow the price out to $33,555, which is a wee bit expensive.

There's also a version of the motherboard with built-in QDR Infiniband, but it costs twice as much.

Or this, for storage:

Supermicro 36-bay chassis with DP motherboard and RAID controller $2569
AMD Opteron 6128 CPU x2 $572
4GB DDR3-1333 ECC Registered RAM x8 $1104
Seagate Barracuda LP 2TB x36 $4140

Total: $8385

Or finally:

Supermicro 6-bay chassis with QP motherboard: $1993
AMD Opteron 6128 CPU x4 $1144
4GB DDR3-1333 ECC Registered RAM x32 $4256
Seagate Barracuda ES.2 1TB x6 $954

Total: $8347

I don't need 72TB of disk space, but I know someone who does.

Posted by: Pixy Misa at 01:53 PM | Comments (10) | Add Comment | Trackbacks (Suck)
Post contains 190 words, total size 1 kb.

More Moyashimon

Live action this time.

Not that there's anything wrong with that.

Posted by: Pixy Misa at 02:09 PM | Comments (1) | Add Comment | Trackbacks (Suck)
Post contains 13 words, total size 1 kb.

Go Team Budgie!

Oz politics update:

Asked on Triple J’s Hack program this afternoon if the Coalition would vote for the [internet filter] policy if Labor won the election, Hockey’s response was short and to the point.

“No,” he said.

Finally.  Someone I can vote for.

Screw you, Team Ranga.  Shoulda dumped Conroy when you had the chance.  And Team Watermelon - green on the outside, red on the inside - while consistently against the Great Firewall, are equally consistently the party of economic suicide.

Posted by: Pixy Misa at 12:18 PM | Comments (1) | Add Comment | Trackbacks (Suck)
Post contains 81 words, total size 1 kb.

No, You Can't Have A Pony

We seek the following from the BDFL:

• Approval for the overall concept of adding a just-in-time compiler to CPython, following the design laid out below.
• Permission to continue working on the just-in-time compiler in the CPython source tree.
• Permission to eventually merge the just-in-time compiler into the py3k branch once all blocking issues have been addressed.
• A pony.

Posted by: Pixy Misa at 01:27 AM | Comments (3) | Add Comment | Trackbacks (Suck)
Post contains 61 words, total size 1 kb.

Hitchens On Chavez

In Slate:

Chávez, in other words, is very close to the climactic moment when he will announce that he is a poached egg and that he requires a very large piece of buttered toast so that he can lie down and take a soothing nap.

Posted by: Pixy Misa at 05:35 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 47 words, total size 1 kb.