...

They're 2GHz Athlon 64 X2s. The results are Exactly. The. Bloody. Same.

I guess that 2GHz Athlons are the baseline for decent performance, for either desktops or servers. But I got my first one in 2003! Okay, so the new ones are 64-bit and dual core (and about 40% cheaper, from memory), but still...

Posted by: Pixy Misa at 12:02 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 90 words, total size 1 kb.

Doh!

Yeah. That one might work.

Update: The art of helpful commenting:

# code duplication because ow.

Update: Yuggech.

Okay, so it's been a while since I installed the standard BitTorrent package on Linux.

The last time I did so was 4.0.something. The current release is 5.0.3.

4.0.x required Python 2.2 or later.

As for 5.0.3:

* Install Python, version 2.3 or later. (2.4 recommended) -- Fine. Got that.
* Install wxWidgets, version 2.6 or later, in the Unicode flavor. -- I'm not installing wxWidgets on a server just to make BitTorrent happy! I hope that this is only actually required for the GUI apps.
* Install wxPython, version 2.6 or later, in the Unicode flavor. -- Sod off.
* Install GTK, version 2.6 or later. -- Bugger that.
* Install Twisted, version 2 or later. -- No.
* Install PyCrypto, version 2 or later. -- Now that is actually a darn good idea.
* Install Psyco. -- Psyco is good. Requiring Psyco is very very bad.
* Install Zope.Interface. -- Sod off!

So 4.4.0 it is. 4.4 seems to be the end of a stable release, followed by a completely new release with these features:

* completely new UI -- Don't need it or want it.
* smart download/queuing behavior -- Don't need it.
* smart seeding behavior -- Don't need it.
* torrent priority system -- Don't need it.
* detailed progress bar and "piece bar" progress bar -- Don't need it.
* better save location management ("incomplete" and "finished" locations) -- Completely irrelevant.
* automatic bandwidth management - Hmm.
* download rate control -- Don't need it.
* transfer rate graphs -- Don't need it.
* choose which files to download first from a torrent -- Don't need it.
* better error handling and reporting -- Hmm.
* fast extensions (see bittorrent.org) -- Useful, yes, but not here.
* torrent "title" support (see bittorrent.org) -- There's nothing on bittorrent.org!
* multiple tracker support -- Actually, that might be something I could use.
* encryption support -- Don't need it.
* Zeroconf ("Rendezvous") local discovery -- Don't need it.
* sparse files -- Don't need it.
* threaded Disk IO -- Don't need it.
* translation update system -- Don't need it.
* now using wxWidgets instead of GTK on Windows, GUI is now native and should be a lot more stable -- Don't need it, don't want it, just make it go away!
* removed support for Python 2.2 -- Whatev.

Thanks, BitTorrent peeps, for at least keeping 4.4 around.

Update: libtorrent won't compile. Somewhere, deep in the jungle, a library is missing a .h file. Or something. Ick.

Update: The other thing I want to note, is that for something so popular, BitTorrent is incredibly underdocumented. The only place that the options to the tracker module are documented are in the program, and the only way to find them is by running the program, because they're all in translation files, and if you don't have all the crazy dependencies installed you can't run the program.

Blah.

Here's to statically-linked binaries. Yeah, I know they suck too. But they suck differently.

Posted by: Pixy Misa at 11:40 PM | Comments (5) | Add Comment | Trackbacks (Suck)
Post contains 605 words, total size 4 kb.

Just the thing for timing a long-running benchmark.

Sigh.

Posted by: Pixy Misa at 06:51 PM | Comments (7) | Add Comment | Trackbacks (Suck)
Post contains 21 words, total size 1 kb.

First I suspected that it might be the thread-local data, which can persist between requests. But a bit of debugging showed that this data wasn't growing.

And a bit more debugging showed that the leak still occurred when I disconnected the entire Minx module from the dispatch code and just returned a static string.

And a bit more debugging showed that the leak even occurred with the CherryPy "Hello, World!" example.

If the tools.sessions option was set in the config file.

So then - a bug in the CherryPy session tracking code!

...

Not quite. Well, possibly a small one, but I'm not certain of that.

By default, CherryPy sessions last for 60 minutes, and (also by default) it can be another 5 minutes before the background cleanup task removes them.

Meanwhile, I was hitting CherryPy with anything up to 150 requests per second. And since I was using wget for the testing, rather than a browser, I was throwing the session cookies away.

So I had something like 500,000 sessions active at any time. And CherryPy appears to use about 300 bytes per session.

I didn't notice this with my torture test because that was generating less than 1000 requests an hour. 300kB of overhead is hard to spot on a 70MB process (10 threads, 5MB pages). 150MB of overhead on a 25MB process (100 threads, 100kB pages) is rather more noticeable.

There's still a problem with Minx slowing down over time. The page generation time in my current test has slowed by 75μs over the last 50,000 pages. I had chalked this trend up to the memory leak, but there may be another problem I need to address.

Well, at least I've made progress. I haven't fixed anything - except my test bed - but I have ascertained that the bug doesn't exist.

And I also know that CherryPy can cope with 500,000 simultaneous sessions...

Posted by: Pixy Misa at 02:35 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 378 words, total size 2 kb.

Currently I'm seeing a time per page of 7.0 ms and 87 pages per second. The system load is about 60% Python, 2.3% memcached, 1.1% MySQL, and 33% idle. I can't seem to push it much higher than that with my current test framework.

Which tells me that:

1. It's plenty fast.
2. memcached uses no CPU, in the broad scheme of things.

The live deployment will have multiple instances of Minx per server, and load-balance across them as well as across the servers themselves. And the servers will be about 3.5 times as fast (comparing the Xeon 3060 to the Athlon XP 2800+).

Multiply it out and I'll get, hmm, about 465 pages per second.

As for the 7ms of overhead, well, even cached pages get passed through the template engine (because there are template tags that take effect after the page is generated) and it still has to check your login against the database (to see if you have any preference settings that would require a custom page to be presented) and so on. I'll try to keep the overhead down, of course, but that's not too terrible, considering.

Now I'll leave it to soak. Python is using 85MB at the moment (with 50 threads), but seems to be creeping up a bit. One odd thing with CherryPy is that the virtual space is much larger - currently 650MB. Which limits me to about 300 threads total per instance.

The other thing I noticed is that memcached was developed by the people at LiveJournal. LiveJournal has since been acquired by SixApart. SixApart are the Movable Type people. I originally started writing Minx just so I could get off Movable Type.

Well, I was amused.

Update: Hmm. 96MB now. Admittedly, it reached 70MB with 10 threads after a long soak, and I was expecting it to be on the order of 350MB this time.

As long as it stops growing eventually.

Update: 121MB and 8.2ms per page at the 130,000 page mark. That smells like a leak, but a pretty small one.

Posted by: Pixy Misa at 05:42 AM | Comments (1) | Add Comment | Trackbacks (Suck)
Post contains 388 words, total size 2 kb.

I was going to run some little benchmarks to compare the speed of my web servers (Akane, Nabiki and Ranma) and my development servers at home (Naga and Martina).

But the results weren't very interesting, because they're all the same.

Akane and Nabiki are Opteron 170s: dual-core 64-bit 2.0GHz.
Ranma is an Athlon XP 3000+: single-core 32-bit 2.16GHz.
Martina is an Athlon XP 2800+: single-core 32-bit 2.08GHz.
Naga is an Athlon 64 3200+: single-core 64-bit 2.0GHz.

Naga is the only one running a 64-bit kernel, and hence a 64-bit Python.

I also checked up on Namo, the little Celeron box I got to run The Jawa Report when we were getting DDoSed. Kei and Yuri seem to be down right now... Uh, which is bad. They should be alive until the end of the month.

After making my benchmark self-timing, so that I can run it on Windows, I can add:

Lina: Pentium 4 2.6GHz
Amelia: Core Duo 1.66GHz
Haruhi: Core 2 Duo 2.4GHz

I'm not going to bother running tests on Sylphiel and Kyon, my Linux virtual machines, because the times vary by +/- 20% due to the clock problems I mentioned earlier.

You can see why all the servers for the New Site are going to be Core 2 Duos. I'll probably migrate mu.nu to Core 2 Duo servers some time in '07 as well - the monthly charge is the same. I just have to wait until SoftLayer are offering their double-memory deal again.

And here's the reason why I'm going to stick with the 32-bit kernel for the application servers:

Psyco is a JIT compiler for Python. While it doesn't always improve performace, when it does, the advantage can be huge, and you can tell it to compile only specific functions if you need to.

Psyco produces 32-bit code. It doesn't work at all on 64-bit Python, and if you run a 32-bit Python on a 64-bit kernel, the overhead of the compatibility layer makes Psyco slower than the standard interpreter.

more...

Posted by: Pixy Misa at 09:54 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 438 words, total size 4 kb.

Plesk seems to be pretty good - though admittedly I don't have 250 users on it yet. It's certainly prettier and better organised. CPanel with the Xcontroller theme isn't bad at all, but that does nothing for the administration system that lives behind CPanel - WHM - which is what I get to use. "Disorganised" is too kind a word for WHM, unless you were to prefix it with "hopelessly".

I bought a license for Plesk to try it out; I probably didn't need to do that, since as it happens you can download it for free as long as you only want one domain and one email address. I have it installed on Martina and Sylphiel that way; my license key languishes unused. And I got it installed on Ranma as well.

Both my bought license and Ranma's monthly license are for 30 domains. One thing about CPanel - it costs a bundle, but there are no limits to the number of users you can load onto it. Plesk comes in 30, 100 and 300 domain licenses as well as the even more expensive unlimited version.

And, as I looked at putting my site on Ranma, I counted the domains. I set up ranma.mu.nu as a valid web site, so that Host-Tracker has something to track. I have seven pixymisa domains (com, net, org, nu, biz, info, and name*), four domains pointing to this blog (ambientirony.com/net/mu.nu and ai.mu.nu), then there's animeclips.mu.nu, and whatever my short-lived anime downloads blog was.** So that's 17 out of 30 gone right away.

Well, as it turns out, it's a bit better than that. It's not 30 domains, exactly. Subdomains don't count. Neither do add-on domains - domains that point to the same site as an existing one. And since subdomains seem to be able to have their own FTP and web user logins, and mu.nu is of course just one domain, and most of the domains here that aren't mu.nu are actually add-ons to mu.nu subdomains, I could probably host the entire thing sprawling mass of Munuvia on a single 30 domain Plesk license.

Or maybe not, since I can't work out how to point an add-on domain to a subdomain.

But it certainly is pretty.

And while CPanel has not, after three years, managed to support Apache 2.x***, Plesk in the same amount of time finally managed to incorporate Awstats.

On the yet-another hand, their SpamAssassin module is an extra $50.

Meh.

* Uh... they were cheap.

** I'd revive it, except that it was largely a front-end to my Broadcast Machine installation, and I ran into some nasty bugs in Broadcast Machine about the same time the lead developer called it quits. (I think that's what happened. There hasn't been even a patch for it issued in six months.) I'll probably revive it anyway, now that I have another 1.5 terabytes a month to throw around and don't need BitTorrent so much. Minx has some features for doing Broadcast Machine style sites (if anyone remembers what it looked like), but doesn't do torrents, and won't for some time.

*** Okay, it appears that they recently added Apache 2.0 support to the "EDGE" release, essentially their public beta. Of course, Apache is now on 2.2.

Posted by: Pixy Misa at 11:10 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 590 words, total size 3 kb.

Since I reinstalled my PC with Windows MCE, I've had time troubles. Windows shows the correct time. Most applications show the correct time. But some - I'd noticed this particularly with iTunes and Thunderbird - are three hours ahead. All my email has the wrong time on it, and iTunes keeps resetting my iPod to the wrong time.

I was just doing a bit of digging, and found that Cygwin and Firefox are also screwed up.

And also, it's not that it's 3 hours ahead - it's 21 hours behind.

My timezone is +1000, and currently in daylight saving time, so it's +1100.

And all these unixy-type apps think it's -1000.

Why? Well, I think it has to do with the TZ system variable being set; they're looking at that instead of the proper Windows information. But TZ just says +1000, so why would they do... what they're doing? And it's consistent across all those applications. And how do I unset TZ anyway?

Posted by: Pixy Misa at 04:00 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 184 words, total size 1 kb.

Yeah, budget server with a 500GB drive. Yay!

Now no-one go blowing up Texas, because that's where all the munu servers is at.

Kasumi and Ukyo will be joining the team in January or early February, and I'm looking at another budget server to be called Ryoga.*

I think I'm going to have to branch out to Maison Ikkoku and Urusei Yatsura before the end of '07.

* While all the other servers are in Texas, either Dallas or Fort Worth, Ryoga would be in Chicago.

Posted by: Pixy Misa at 11:08 PM | Comments (3) | Add Comment | Trackbacks (Suck)
Post contains 138 words, total size 1 kb.

The server itself didn't go down, so what most likely happened is someone did a DOS-lite and tied up all 200 Apache threads. Usually it's spammers, but sometimes it's a bad web spider.

I saw one the other day that was apparently fetching every page on one of the munu blogs in parallel. This is, of course, dead simple for the developer and speeds the indexing process up enormously; it will also take anyone stuck on Apache 1.x* effectively off-line.

Meanwhile, my soak test has delivered 1.5 million pages in 24 (CPU) hours. No signs of problems of any sort; I turned off caching to make it do more work, so it hasn't tripped over that bug.

Update: Ooh, another race condition in the SQL caching code that memcached will eliminate.

* Thank you CPanel.

Posted by: Pixy Misa at 07:44 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 145 words, total size 1 kb.