Wednesday, November 20

Geek

Mission Accomplished-ish

One of my goals for the next version of Minx was to reduce build times by a factor of ten, from around 100ms to 10ms for simple pages, and for larger pages from 1s to 100ms.

Here's a post with 1400+ comments. Before (worst case, with neither the element cache or the MySQL query cache in effect):
573kb generated in CPU 4.09, elapsed 3.662 seconds.
66 queries taking 1.0599 seconds, 1591 records returned.
After:
573kb generated in CPU 0.02, elapsed 0.0534 seconds.
41 queries taking 0.0396 seconds, 55 records returned.
The new element cache eliminates the slow database queries, comment text filtering, and much of the template processing, and the performance improvement is dramatic.

Here's my main page, before:
97kb generated in CPU 0.88, elapsed 0.5679 seconds.
31 queries taking 0.2722 seconds, 121 records returned.
And after:
97kb generated in CPU 0.01, elapsed 0.0064 seconds.
14 queries taking 0.0029 seconds, 25 records returned.
What I need to do now is make the cache smarter, so that I can deliver more from the cache rather than rebuilding it. The cached performance is fantastic, but we're only hitting the cache about 40% of the time. I want to get that up above 90%.

Posted by: Pixy Misa at 09:00 PM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 195 words, total size 1 kb.

1 Good job!

Posted by: Steven Den Beste at Thursday, November 21 2013 01:27 AM (+rSRq)

2 One of the bonuses of this work is that I can use simple and robust database queries rather than worrying about micro-optimisation.  (Proper indexing is still critical, of course; nothing will fix an unindexed query.)


I'm planning to switch from MySQL to MongoDB in the following version.  I want to use MongoEngine, a very nice Python library that lets you treat MongoDB pretty much as native Python data - and automatic data validation to precisely the degree you need it.  The only problem is that it imposes a pretty significant processing overhead.

The element cache will work to eliminate that overhead just as effectively as it does with MySQL and the comment formatter, so I'm free to go ahead and use MongoEngine.

Posted by: Pixy Misa at Thursday, November 21 2013 06:15 PM (PiXy!)

Hide Comments | Add Comment

Comments are disabled. Post is locked.
46kb generated in CPU 0.2306, elapsed 22.7464 seconds.
56 queries taking 22.7303 seconds, 340 records returned.
Powered by Minx 1.1.6c-pink.