This accidentally fell out of her pocket when I bumped into her. Took me four goes.

Wrong Button!

Oh, yeah.

The reason the performance issue came up was that I was testing one of the three** methods for sharing posts between blogs.  The three being cross-posting (post one entry on two blogs), folder peering (set an entire folder - a category, for example - to appear on two blogs), and direct template hackery.

The last is what I was testing.  It works, but with a catch.  The old template relied on standard site and folder data to construct the URLs for the permalink and comments pages (which are the same by default, but need not be).  That's okay for cross-posting and  peering, because the post really is on your site, either at the thread or folder level. 

When you engage in template hackery, though, you are calling up posts entirely ad-hoc from another blog (assuming said blog has set access rights to allow you to do so).  If we rely on your site data to generate links for posts that only exist on the other site, then those links don't work.

The solution to that is simple - join the folder and site tables in the query and add their public fields to the dataset for the post tag, and modify the templates to use those new fields to construct the links.*  But that means a slightly slower query and a little more processing for the tag engine for every page, because while I actually could make the template auto-sense cross-site ad-hockery, it's too much trouble and would scramble the brains of any poor user who just wants to tweak their layout a little.

And so, millisecond by millisecond, performance decreases, and CPUs aren't getting faster at the same rate anymore.  The road to Hell is paved not with good intentions, but with "minor" code changes.

* This now works, by the way, though you do need to use the (absolute) post.url tag instead of the (relative) post.path in your templates.  Um, once 1.2 is released, that is.

** Four.  You can also view friends' and friends-of-friends' posts using template hackery.

*** Five!  I just realised that it's dead easy to allow smart folders to work across sites.  (Smart folders are much like smart playlists in iTunes - they automatically collect posts from other folders based on whatever search criteria you specify.)  The only real work left is setting up the access control.

Posted by: Pixy Misa at 01:40 AM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 397 words, total size 2 kb.

Push The Button, Max!

I'm preparing for the rollout of Minx 1.2, which has a whole bunch of improvements over 1.1...  Speed, on the whole, not being one of them.*  I never add anything without optimising it to within an inch of its life, but constantly adding features doesn't come without some cost.

Fortunately Minx is plenty fast for most cases.  It can even produce a full page of Ace of Spades with many hundreds of inline comments at a reasonable speed.  But I don't like to see those milliseconds slipping away from me.

I've implemented a microcaching system****, and tried it out on the comment sanitiser, which is probably the slowest deterministic function in the system.  The results were disappointing, providing only a 10-15% improvement overall.  I had thought that the comment sanitiser would have dominated the page generation time, but it turns out not to be the case: If I turn off santisation completely, it's not measurably faster than the cached version.

I need to do another round of profiling, I guess.

In the old days, this could have been resolved simply by swapping the servers for newer and faster ones.  Now, though, the fastest servers offered by Softlayer - in terms of single-threaded performance - are only 25% better than what we already have.  (And between 50% and 150% more expensive, which is why we have what we have.)  I can get 16 cores no problem, but I can't get 4GHz at any price.*****

Intel is coming out with a new CPU architecture this year, codenamed Nehalem.  While plenty is known about Nehalem's interconnect architecture (Intel is finally catching up to AMD), not much has been said about the CPU core itself, and even less about its performance.

A presentation recently leaked by Sun (oops) showed Nehalem servers delivering 2.8x the SPECint_rate2006 performance of a 3.0GHz Core2 server.  But SPECint_rate2006 is a multi-threaded benchmark, and the details showed that this compares an 8-core dual-threaded system against a 4-core single threaded one, so the single-threaded performance gains could be as little as, well, zero.  Rumours since then have put it at between 10% and 25% clock-for-clock.  Unless Nehalem also brings a significant clock boost, that's not a lot.

The throughput situation is a lot brighter, but throughput for medium-scale web apps is a solved problem anyway.  Multiple cores, multiple CPUs, multiple servers and lots and lots of RAM.

Curse you, laws of physics.

* Except for a few instances like recent comment listings, which were a hack in earlier versions and have been rewritten to use a new folder->post mapping table which provides O(n) performance instead of the previous O(n² log(n)).**

** Or... Not.  Okay, now it takes 2.5 minutes.  Back to the query analyser for you, buster!***

*** Ah, there we go.  70 milliseconds worst case.  Much better.  The whole point of the mapping table was to eliminate the need to sort large sets, so sorting on a field dependent on a join didn't really help.

****  Python provides a remarkably simple and fairly elegant way to do this, using "decorators".  Simply by adding the line @qikcache(10000) before a function definition (where qikcache is a caching function I've defined), I change the function to perform a lookup in an LRU cache before carrying out its normal calculations - and to automatically calculate and write the result to the cache if it's not there already.

***** Except from IBM, and the Power6 is a speed-demon design that doesn't deliver signficantly better performance on integer loads anyway.  In our case it would be worse, because the Psyco JIT compiler Minx uses only works on x86.

Posted by: Pixy Misa at 01:25 AM | Comments (6) | Add Comment | Trackbacks (Suck)
Post contains 606 words, total size 4 kb.

Strange Worries

I was fretting just now while in the shower because Midori wasn't exceeding its bandwidth limits this month.  Normally I get a daily email after a server reaches 85% of its quota, and this month, nothing.

The problem with that is that if bandwidth is down, page views are down.

But not to worry, Midori is set to hit 2.3TB despite the short month.  I just missed the email or something.

Which means $30 in excess usage fees for me.  Once that hits $50, it becomes cheaper for me to pay for a virtual private rack and share bandwdith across the servers.  (Softlayer's virtual private racks are $25 per server.)  Unfortunately I have to commit to that a month in advance, so if we suddenly hit 2.7TB I'll be out twenty bucks.  Guess I can deal with that...

Posted by: Pixy Misa at 02:15 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 140 words, total size 1 kb.

Naming Names

It seems that my new server naming scheme of Japanes-girls-names-used-in-anime-that-are-also-colours is feasible after all.

A few months ago we consolidated down from four small servers to two larger ones, Midori and Aoi.  But what about the future?

Well, the future is safe, judging by this handy site and this wiki page.  Often, the form given is something-iro, meaning something-coloured, but I can live with that.

So I can use Sakura (pale pink), Mikan (a shade of orange), Akari (red), and Momoko (peach).  That should keep us going for a while.

Posted by: Pixy Misa at 01:15 PM | Comments (9) | Add Comment | Trackbacks (Suck)
Post contains 92 words, total size 1 kb.

Still 100% Content-Free!

Boogaloo Xenu, at your service:

Posted by: Pixy Misa at 10:15 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 8 words, total size 1 kb.

The Best Tribute

To a man who changed the world.

Posted by: Pixy Misa at 07:21 PM | Comments (1) | Add Comment | Trackbacks (Suck)
Post contains 10 words, total size 1 kb.

In Lieu Of Content

This!

Posted by: Pixy Misa at 09:01 PM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 5 words, total size 1 kb.

Ooh, Shiny!

Hey, physics/chemistry/biology puzzle/simulator fans!  While you're waiting for Spore and Little Big Planet, here's another game to wait for:

World of Goo!

Posted by: Pixy Misa at 10:47 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 24 words, total size 1 kb.

And We Like It That Way

The Finke River, starting roughly west of Alice Springs is thought to be the oldest riverbed in the world and although it flows for only a few days a year (in many years it does not flow at all) is home to 7 species of fish, 2 of which are found nowhere else. The waters of the Finke disappear into the sands of the Simpson Desert and are not definitely known to ever make it as far south as Lake Eyre, although the story is told that this happened once early in the 20th century.

Posted by: Pixy Misa at 11:49 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 101 words, total size 1 kb.

Not Just For Pretty Pictures Anymore

Fractal Wrongness:

The state of being wrong at every conceivable scale of resolution. That is, from a distance, a fractally wrong person's worldview is incorrect; and furthermore, if you zoom in on any small part of that person's worldview, that part is just as wrong as the whole worldview.

Debating with a person who is fractally wrong leads to infinite regress, as every refutation you make of that person's opinions will lead to a rejoinder, full of half-truths, leaps of logic, and outright lies, that requires just as much refutation to debunk as the first one. It is as impossible to convince a fractally wrong person of anything as it is to walk around the edge of the Mandelbrot set in finite time.

If you ever get embroiled in a discussion with a fractally wrong person on the Internet--in mailing lists, newsgroups, or website forums--your best bet is to say your piece once and ignore any replies, thus saving yourself time.

Posted by: Pixy Misa at 10:28 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 166 words, total size 1 kb.