(Currently going for a doc crawl on the theory that it can't be that broken.)

Huh.

Okay, the problem I'm having involves modules, threads, and thread-specific global data. I haven't solved the problem with modules yet, but it turns out there was some magic added to Python 2.4 for thread-specific globals (threading.local). That's a comfort, because I knew that worked, but I couldn't figure out how. CherryPy, the web framework I'm using, supports this under 2.3, but it turns out that it's a hack and it's very slow. So I'm not going mad. Or at least, no more than usual.

I'm moving Minx from the test design, where it is a single CGI program (and so each request is perfectly isolated and I can slap the code together any which way) to production, as a multi-threaded persistent server. Which is much more fiddly in terms of structuring the code and variables, but is twenty to thirty times faster.

Up to 95% of the time taken by the CGI version is overhead: starting a shell, then starting a Python interpreter, loading the twenty or so libraries used, opening a connection to MySQL, and so on. The multi-threaded version does all of that once. (Or at worst, once per thread, for a persistent thread pool.) It also uses Psyco, the Python compiler, which adds a 30% to 50% speed boost for this sort of app. For the CGI version, Psyco takes long enough to do the compile that the overall performance is worse in most cases...

Only because the threads weren't actually isolated from one another, it didn't work at all. I could either add an extra parameter to all the roughly 100 functions I've written so far, or I could work out how to do thread-specific globals.

Update: Okay, all is forgiven. The threading.local trick works flawlessly, even with modules. Threading-local global data for one module is not visible in another, but even if that complicates things for me, that's right. They're modules, not include files. So I have thread-local module-local global variables... Yay!

Update: And it works perfectly with CherryPy. I expected that, because it only makes sense that CherryPy would be using the standard threading module, but there's a difference between being the only sensible way to do something and actually testing it.

Posted by: Pixy Misa at 08:34 AM | Comments (5) | Add Comment | Trackbacks (Suck)
Post contains 399 words, total size 2 kb.