No boom today. Boom tomorrow. There's always a boom tomorrow.

Busy Week Ahead

The plan is to set up 14 15 19 more servers this week.

Nabiki, Kasumi, and Ukyo will handle mu.nu (non-Minx) and the other hosted sites.  They'll be equipped with CPanel.

Shampoo will be the Minx database server.  Kodachi, Azusa, and Hinako will be the three Minx application servers.  Lin-Lin and Ran-Ran will be the mee.nu and mu.nu Minx load-balancers respectively.

Mariko will handle database replication.  Nodoka, monitoring via Cacti and Nagios.  Akari will be the stats engine, running Urchin 6.  And Natsume will run the new search engine, based on Xapian.

Then there's the two new messaging servers, the file server, and the cache server.  Need to find some names for those.*

And finally, Honoka and Sumire will be the test environments, running Minx 1.2 beta and 1.3 dev respectively.

All up, that works out to $56 a month - exactly less than $4 $3 per server.

* And when I run out of Ranma characters, I'll just move on to Urusei Yatsura and Maison Ikkoku.

Posted by: Pixy Misa at 05:01 AM | Comments (9) | Add Comment | Trackbacks (Suck)
Post contains 170 words, total size 1 kb.

Very Random Thoughts

'Twas gloaming and the Assyrian
Did downcome like a wolfolding.
All purplegold his cohorts and
Like seastars their spearsheen.

"Like forest leaves when Summer's green
That bannerhost at sunset's seen.
Like forest leaves when Autumn's blown
That morrowhost lay witherstrown."

Posted by: Pixy Misa at 02:28 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 43 words, total size 1 kb.

Pootsync

rysnc 2.8 blows up with an out of memory error when asked to transfer 6.7 million files.

Well, crap.

The good thing about running something like RedHat Enterprise Linux (we actually run CentOS, which is a free distribution recompiled from the RHEL sources) is that it's stable.

Really stable.

Nothing ever gets frigging updated.  Bugs get fixed, particularly security and stability bugs, but it's still on Python 2.4.3, which came out about the same time as Julius Caesar's memoirs.

I do have OpenVZ installed.  So I can set up, say, a Fedora 8 VE, update rsync there, and use that just for rsyncing.

Not at 2AM though.

Update: Whoa.  Just hit the Minx restart/cleanup timer while posting this - right at 2AM - and got the "Service is not available" message.  I knew that in theory it could do that, but I'd never seen it before.  There you go.

One second later and everything is fine again.

Posted by: Pixy Misa at 02:02 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 158 words, total size 1 kb.

It Is Done

Okay, it took a couple of weeks (because I was busy doing other stuff) but OpenVZ is now installed and working on Akane.  That means I can now start firing up virtual environments and moving things across.  And shutting down servers so that I don't get charged (unmentionable) dollars next month.

Might just install it on wossname as well.  Momoko?  Kurumi?  Mikan, that's the one.

Update: Just reinstalling the OS now to do a clean run of installing OpenVZ and the Intel drivers.  I have a /27 block of VLAN-routable public and private IP addresses (i.e. 29 usable addresses on each network) so I can create up to 29 OpenVZ VE's.  That's a bit of a squeeze with 24GB of RAM, but the server can go to 72GB, so I thought I'd grab enough to last us a while.  And a Minx VE can probably run just fine in 256MB, so I might be setting up a few of those.

So, mee.nu and other Minx users - migration is tentatively scheduled for Saturday May 3rd.  I'll keep you posted.

Posted by: Pixy Misa at 04:03 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 183 words, total size 1 kb.

Oops

After carefully conserving bandwidth for two weeks, I just accidentally downloaded about 700 podcasts.

Posted by: Pixy Misa at 11:44 PM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 15 words, total size 1 kb.

Pystones 2

We've just installed a new server at my day job with dual Xeon 5570's (2.93GHz Nehalems), effectively Akane's big sister.  (It also has 72GB of RAM and 6 Intel SSDs.)

Naturally, I ran Pystone on it.

Db3: 111,482

And for reference:

Akane: 82,169
Yurie+Psyco: 346,020

Posted by: Pixy Misa at 09:12 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 47 words, total size 1 kb.

KyoAni Strikes Silver

After one awesome blockbuster hit, KyoAni pissed it all away with series after adequate-to-downright-awful series.  There may be a new Haruhi series airing later this year, but after all Kadokowa's stupid tricks I won't believe it until I've actually seen the entire thing.

But the current season's K-On!, while not being (or trying to be) anything amazing, is nonetheless genuinely likeable and fun.  And the opening and closing songs grow on you...  Like some sort of fungus.  The OP is currently stuck in my head and will not go away.

Recommended, with EWW (earworm warning).

Posted by: Pixy Misa at 08:53 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 98 words, total size 1 kb.

Next Stop, Akane Town!

While I was distracted by the fires and explosions at my day job, someone else got Virtuozzo working on a SoftLayer Nehalem server and made the necessary network driver available as an RPM.

Very cool.  I should be able to get Akane up and running tomorrow.

A little late, since I've already been hit with an enormous bill because we're running four servers at SoftLayer right now instead of just two, but I should easily be able to get us moved across over the next couple of weeks.

Posted by: Pixy Misa at 06:54 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 92 words, total size 1 kb.

What I Want To See

Is a moeblob show about music, or mahjong, or sad girls in snow, only by the end of the first episode they blow up the moon and just keep escalating from there.

Posted by: Pixy Misa at 02:49 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 37 words, total size 1 kb.

What The World Needs Now

Apart from a live-action Kim Possible movie - is an Erfworld anime.

Posted by: Pixy Misa at 04:06 AM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 17 words, total size 1 kb.

Dreams

So humans and aliens finally make contact - in a horrific starship collision, where the warp fields of the two ships interact in such a way as to draw them inexorably together into a gravitational singularity.  As time dilation effects spread out differentially from the engine cores, giving everyone more than enough time to realise their fates, the aliens resign themselves to the inevitable, while the human captain (yours truly) figures out a way to save his ship and crew, while dooming himself and the alien craft - and takes it.

The aliens respond, punishing the captain by trapping him in a time loop, where every morning he awakens to find an unfamiliar pistol on his dressing table, and slowly comes to the realisation that he is trapped in a time loop on a strange planet, with no way out, no way to ever see his family again, no escape except to end it all with a bullet to the head, whereupon he awakens to find an unfamilar pistol...

And manipulates the idea of quantum immortality to survive his own death, escape the time loop and the growing gravitational singularity both, and return to the real world, where he finds that he has been reading a little-known story by Fritz Leiber (or George R. A. Murray, whoever he is) in a small-press limited-print collection at Galaxy Bookshop, the old one on Bathurst St, where his 19-year-old self is trying to update his store discount card, only they won't accept his identification, and he has to get his mother to sign for him.

I don't think my mother ever set foot in Galaxy, though I do remember her buying Dragon magazine for me, back in the day.

Posted by: Pixy Misa at 02:49 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 287 words, total size 2 kb.

Ars Munuvica

Just browsing Ars Technica, as is my wont of a Saturday evening, when I find....  This:

One such Taliban propaganda site billed itself as the official voice of the "Islamic Emirate of Afghanistan," and regularly provided enthusiastic (if highly dubious) reports of successful attacks on coalition troops in the region. It was pulled offline last week after conservative blog The Jawa Report urged readers to complain to companies "unknowingly and unwittingly giving the Taliban some of the necessary tools they need to prolong the war." (The blogger also provided Taliban officials' contact emails, and suggested signing them up for sheep porn.)

Posted by: Pixy Misa at 07:24 PM | Comments (8) | Add Comment | Trackbacks (Suck)
Post contains 102 words, total size 1 kb.

Yeah Baby

Mythbusters is back for a new season.

Three words: Radio-controlled bus.

You know you're on something big when your small scale involves dropping cars from a crane.

Posted by: Pixy Misa at 09:48 PM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 29 words, total size 1 kb.

Pystones

Akane: 82,169 (Xeon 5520 (Nehalem) 2.26Ghz, 64-bit)
Mikan: 84,033 (Xeon 3230 2.66GHz, 64-bit)
Yurie: 55,463 (Athlon 64 X2 5200+ 2.6GHz, 32-bit)
Haruhi: 66,533 (Athlon 64 X2 5200+ 2.6GHz, 64-bit)

Posted by: Pixy Misa at 03:50 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 216 words, total size 2 kb.

Ooookay, That Will Probably Suffice

SoftLayer now support IPv6, and you can get a free block of IPv6 addresses with new servers.

You get one master IPv4 address, four free secondary IPv4 addresses, and 64 free IPv6 addresses.

At least, that's how I read it.  I was wondering why they were being so stingy with IPv6; they're not exactly in short supply.

Well.

Actually.

It's not 64 IPv6 addresses, it's a /64 block.  In other words, 18,446,744,073,709,551,616 addresses.  Over 18 quintillion.

They also offer additional IPv6 blocks at extra cost.  I don't think I'll need that service in a hurry.

Posted by: Pixy Misa at 02:56 AM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 100 words, total size 1 kb.

And Mikan Makes Two

The second server for the new hypercluster is being provisioned right now.  Mikan* is just a baby compared to Akane - a quad-core Xeon 3230, but with just 2GB of memory and one 250GB drive.

Fortunately, Mikan's main job is to sit there and look pretty...  And to contribute 4TB of bandwidth to our monthly quota.  I used a different promo code this time - double bandwidth instead of double memory and disk - because Akane has all the memory and disk we need but only comes with 2TB of bandwidth against the 3TB+ we are using.

Thanks to the special offer, Mikan gives us a spare server and 4TB of bandwidth for the price of 2TB of excess bandwidth charges.  That means that right now it's effectively a very cheap spare server, and going forward it will actually be saving me up to $200 a month.

I can also run a instance of Minx on there to use the spare CPU power, which was why I went for the Xeon 3230.  The cheapest option was a Xeon 3050, a dual-core 2.13GHz CPU.  The 3230 cost 25% more, but it's a quad-core 2.66GHz chip, so it's 2.5x as fast.  Which seems like a good deal to me.

This will be really nice once Akane is configured (trying another OpenVZ kernel now).  We won't need to move to a new server for a good long time: We have 4TB of RAID-5 disk and an SSD for the databases, and room for another 5-disk RAID array and another SSD.  And we can upgrade from 24GB of memory to 72GB now, and to 144GB once 8GB modules come down in price.  And we can upgrade our 2.26GHz CPUs to quad-core 2.93GHz CPUs now, and to six-core CPUs in the future.

I can just add little FEP (front-end processor) / bandwidth pool machines like Mikan as we continue to grow; the next one is destined to be named Kurumi.**

Update: Splatooie!  Oh well.  I have an Intel kernel module driver thingy here to try next.

* A mandarin, also a shade of orange.
** Walnut, both the nut itself and the colour.

Posted by: Pixy Misa at 11:32 PM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 362 words, total size 2 kb.

It's Going Too Fast! I Want To Get Off!

Vroom!  5 x 1TB disks in RAID-5:

[root@akane ~]# hdparm -t /dev/sdb

/dev/sdb:
 Timing buffered disk reads:  1230 MB in  3.00 seconds = 409.96 MB/sec

Posted by: Pixy Misa at 05:23 AM | Comments (4) | Add Comment | Trackbacks (Suck)
Post contains 33 words, total size 1 kb.

I Is Goings To Beds

Cause I tireds.

Nights.

P.S. Watched the first ep of K-On!  Not spectacular, but an agreeable little bit of fluff in a way that the previous five thirty-seven four outings from Kyo-ani haven't been.  Will continue to watch.

P.P.S. Upgraded the RAID array in Akane to 4TB (5x1TB drives, RAID-5).  Glad I did this before installing a few hundred blogs on there, as it required a controller swap, a full OS reinstall, and an estimated twelve-hour array rebuild (still going!)

On the plus side, I now know that even during a rebuild I can sustain 100MB/sec off the aray.  I'm hoping to see 300MB/sec once the rebuild is complete.

P.P.S. Seems that the automatic installer at SoftLayer uses old-style partitions, and so has a 2TB limit.  So I now have two RAID volumes, a 100GB system volume and a 3.9TB everything else volume.  Which actually suits me perfectly - the base system goes on the 100GB volume, and the 3.9TB goes under LVM for the VPSes.

P.P.P.S. Oh yeah, beds.

Posted by: Pixy Misa at 11:18 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 175 words, total size 1 kb.

I Think You Just Authored A Web Browser

Wrote a search engine over the weekend (when I wasn't building new super servers that don't work).

It pulls data out of a SQL database and creates full-text indexes for it, then provides a web server (no Apache needed!) and query engine with output as template-driven HTML or JSON.

4K of code.

CherryPy and Xapian, an awesome combination.

Posted by: Pixy Misa at 12:35 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 66 words, total size 1 kb.

The Internet Has Everything

Even this.

(via)

Posted by: Pixy Misa at 06:21 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 7 words, total size 1 kb.

Support Above And Beyond

I mentioned on SoftLayer's forums that OpenVZ wasn't working with the new Nehalem server.  SoftLayer don't support OpenVZ; it's a custom kernel that I downloaded and installed myself.  If you need virtualisation and support, they offer Citrix XenServer, Virtuozzo, and Microsoft's Hyper-V.

Which is fine.  I  was hoping to do it myself with OpenVZ.  I couldn't get it to work because OpenVZ doesn't support the new network controllers and needed support.  SoftLayer don't support OpenVZ, so I asked them to sell me Virtuozzo.

What they did was deploy a test server with the same configuration as mine and assign an engineer to getting OpenVZ running.  For free.  That's awesome.

Posted by: Pixy Misa at 05:02 AM | Comments (5) | Add Comment | Trackbacks (Suck)
Post contains 113 words, total size 1 kb.

While We're Waiting

[root@akane ~]# hdparm -t /dev/sda /dev/sdb

/dev/sda:
 Timing buffered disk reads:  628 MB in  3.00 seconds = 209.27 MB/sec

/dev/sdb:
 Timing buffered disk reads:  786 MB in  3.00 seconds = 261.77 MB/sec

sda is a 3-disk RAID-5 volume.  sdb is the Intel X25-E SSD.

Vroom!

Posted by: Pixy Misa at 01:04 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 47 words, total size 1 kb.

I Have Bad News, And Expensive News

Installed OpenVZ on Akane (installing OpenVZ is a piece of cake). 

Rebooted.

No server.

This is not good.

It could be a whole lot worse, since there's nothing on the server, no files, no users.  Still, I can't reach it at all.

One of the things I love about SoftLayer is that IPMI - remote monitoring and KVM - is standard on all servers.  Many other providers are heading that way now, but SoftLayer was one of the first.

So I log in via IPMI - which is good in itself, because it means the server is up.  And I poke around a bit, and find that I have no network cards.  Funny, they were around here a minute ago.

So I restart the networking subsystem and am informed:

Bringing up interface eth0: igb device eth0 does not seem to be present, delaying initialization   [FAILED]

That's not good.  The fact that this message does not appear anywhere on the internet is extra not good with cream.

Some more digging revealed the fact that the network chips on Akane's motherboard - the Intel 82576 - was first supported in the 2.6.25 kernel.  OpenVZ runs 2.6.18.  So do CentOS 5.3 and RedHat Enterprise 5, but someone there must have backported the driver.

I downloaded the OpenVZ 2.6.26 kernel - a development version, not recommended for production - and that was, if anything, worse.  Definitely no network, anyway.

So it looks like we're getting Virtuozzo after all.  Which will cost me extra, but...  Well, mostly it will just cost me extra.  It has some nice features, like a user control panel so you can look after your own virtual machine.  I don't have any users who actually need that...  Well, I'm sure I could find some, but it's not what I'm here for.

Anyway, I've sent an email off to sales to get the move to Virtuozzo-land under way - which will involve wiping Akane clean and reinstalling her again, probably at least twice - and then I can get things under way once more.

Posted by: Pixy Misa at 11:46 PM | Comments (4) | Add Comment | Trackbacks (Suck)
Post contains 347 words, total size 2 kb.

Reboot, Reinstall... CentOS 5.3

Just reinstalling the operating system on Akane from scratch.  I always do this at SoftLayer because (a) it's easy, (b) it's a good test; if I can't do a clean reinstall, there's something wrong, and (c) I can make sure that it's installed the way I want.

Then on with the OpenVZ.

Then I order a bunch of VLAN-routable IPs.

Then I start creating VEs, and installing CPanel, and so on.

Then I test-copy the whole of Aoi / mee.nu across.  Official migration probably over Easter, but I'll do a quick test first.

Then migrate people off Sakura.

Then off Midori.

Then back to migrating people off Movable Type and WordPress to Minx.

Then upgrade to Minx 1.2.

Then Pixy goes to happy land.

Update: Well, that didn't work.

Posted by: Pixy Misa at 08:35 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 134 words, total size 1 kb.

New Toys Indeed!

Look what I've got!

top - 22:12:36 up 14 min,  1 user,  load average: 0.01, 0.02, 0.01
Tasks: 280 total,   1 running, 279 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu8  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu9  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu10 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu11 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu12 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu13 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu14 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu15 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  24677364k total,   374464k used, 24302900k free,    19348k buffers
Swap:  1052248k total,        0k used,  1052248k free,   187340k cached

Posted by: Pixy Misa at 01:13 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 193 words, total size 2 kb.