No boom today. Boom tomorrow. There's always a boom tomorrow.

Essence Of Cool

The closing credits of Hanamaru Kindergarten might win the nod for the best anime of the season.

Posted by: Pixy Misa at 11:53 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 20 words, total size 1 kb.

I Hate Sauerkraut!

Also crazed starving weasels.

Posted by: Pixy Misa at 04:24 PM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 7 words, total size 1 kb.

Move Over Mendeleev

Now with added mouseoverness.

Posted by: Pixy Misa at 02:00 AM | Comments (5) | Add Comment | Trackbacks (Suck)
Post contains 7 words, total size 2 kb.

Periodicity



The Grand Unified Minx Theory


The mee.nu User Domains


The Minx Components

Posted by: Pixy Misa at 05:40 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 13 words, total size 1 kb.

Gamma Lama

Naturally I had to try this...

100%


50%


Oops!

Now, that's a deliberately constructed corner case, but there is a problem there.

Posted by: Pixy Misa at 12:40 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 24 words, total size 1 kb.

Bitzenpieces

Posted by: Pixy Misa at 05:44 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 2 words, total size 1 kb.

Where's Pixy?

I'm in here:


(Click for full screenshot.  Thanks go to Steam and GOG's insane holiday sales.)

Actually, I'm not; I'm doing work for my day job, making some progress with Pita, reorganising Meta, and have finally come to a design decision on Miko (all parts of the Minx project for those who haven't been paying attention), redoing the documentation in Sphinx - which will itself be supported in an upcoming version of Meta - and planning for this year's server upgrade.*  I did play a bit of Dragon Age over the holidays, but games are taking a back seat for a while.**  Despite the fact that I have 224 of them currently installed.

* If things go right we'll be moving from a lowly 8-processor (16-thread) 2.26GHz server with 24GB of RAM to a spiffy new 12-processor (24-thread) 2.66GHz server with 48GB of RAM.  That's at least partly to prepare for the move to Pita, which loves to store stuff in memory.  Because I can just copy the OpenVZ virtual machines across, the move should be quick and painless.

** Apart from Billy vs. SNAKEMAN!

Posted by: Pixy Misa at 02:16 PM | Comments (6) | Add Comment | Trackbacks (Suck)
Post contains 188 words, total size 1 kb.

Hoshimittsu

Hidamari Sketch continues to exist in its own little universe, and all is right with the world.

Posted by: Pixy Misa at 04:12 AM | Comments (3) | Add Comment | Trackbacks (Suck)
Post contains 18 words, total size 1 kb.

If I Don't Lesnerize, The Feegs Can't Get Me

In SQL* you say select sum(sales) from accounts where state="NY".  In Pita, the way to do this is:

results = accounts.aggregate(state='NY')**

which will calculate for you the count, length, sum, minimum and maximum, as appropriate, for all the fields in the table at once, so the value you need is results.sales.sum.  Since the table scan is typically slower than any calculations you're likely to be doing, this seems a reasonable approach.

In addition, I've added a

results = accounts.stats()

which provides all those, plus mean,*** median, mode, standard deviation, and geometric and harmonic means.   Aaaaand standard error, coefficient of variation, sample and population variance, skewness and kurtosis.  I even sort of know what kurtosis is.

I'm working on two more functions now, group and break, though I may need to come up with another name for the latter because break is a Python keyword.  This:

for result in accounts.group('state', country='US'): ...

would give you the aggregate sales figures for each state in the US, sensibly enough.  And this:

for result in accounts.break('state', country='US'): ...

would give you the individual sales figures, and then automatically provide totals after the last sales record for each state.

As long as I don't come down with kurtosis...

Update: Kang and jag.  Or rather, agg and tab.  For aggregate and tabulate. 

for line in accounts.aggregate('state', country='US'): ...

will give you one summary line for each state, where

for line in accounts.tabulate('state', country='US'): ...

would give you both detail and summary lines.  I need to put subtotal and total flags on the records for tabulate.  Have to watch the keywords, there.  And keep my closet doors closed.

* Boo, hiss!

** Or indeed

results = accounts(state='NY').aggregate()

Either way should perform the same and produce the same results.  I think...

*** Which should come out the same as the average; just one I'm calculating myself and the other I'm pulling out of a stats module.

Posted by: Pixy Misa at 03:14 AM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 316 words, total size 3 kb.

Just Trying Something...



Okay, yeah, they needed that sharpening filter.  That's Minx's built-in upscaling.  Quality is not so hot, as it turns out.  I'll check on what filter it's using; normally it's only used for downscaling, which works great:

Posted by: Pixy Misa at 11:35 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 39 words, total size 1 kb.

I Want One!

TI Blaze.  Screw the iPad.  This is the future.

On the one hand, it's a huge clunky thing.



On the other hand, it's a development platform, not a consumer device; it has two 800x480 touchscreens, HDMI out, and a built-in DLP projector; it has two five-megapixel cameras at front and a twelve-megapixel camera at rear; a dual-core 1.2GHz Arm Cortex A9 (superscalar out-of-order SMP); accelerometer, compass, ambient light, proximity, barometric and temperature sensors; Wifi, Bluetooth, and GPS; and easy and open access to all the electronics, networking, and software.

Tech's slate (from the Accountancy story further down) isn't that much more advanced than this beastie.

Posted by: Pixy Misa at 04:04 AM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 108 words, total size 1 kb.

Free At Last!

EA have just earned themselves brownie points with millions of gamers by re-releasing three of the older Command and Conquer games - Red Alert, Tiberian Dawn, and Tiberian Sun - free.

Get downloading!

Posted by: Pixy Misa at 12:26 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 36 words, total size 1 kb.

Oodles Of Noodles

I have a working base storage class for Pita.  Unfortunately, most of my weekend was eaten up by my day job and other miscellanea, but it does work.

I'll post the full code later in the week once I have a derived class or two that does something more useful, in the meantime, here's the test code to give you an example of how it's used:

def oodle_test():
  # Create a base view
  pets = Oodle()
 
  # Create some pets
  log('Creating pets')
 
  # Create a dog, and save it
  pet = pets.new()
  pet.animal = 'dog'
  pet.sound = 'woof'
  pet.save()
  log('Dog saved, %s pets' % pets.count(),1)
 
  # Create a cat from a dict, and save it
  pet = pets.new({'animal': 'cat', 'sound': 'meow'})
  pet.save()
  log('Cat saved, %s pets' % pets.count(),1)
 
  # Append an aardvark
  pet = pets.append({'animal': 'aardvark', 'sound': 'snorf'})
  log('Aardvark appended, %s pets' % pets.count(),1)
 
  # Append a hippopotamus too
  pet = pets.append(animal = 'hippopotamus', sound = 'hrooonk')
  log('Hippopotamus appended, %s pets' % pets.count(),1)
 
  # What pets do I have?
  log('Selecting all pets')
  for pet in pets.select():
    log('My %s says %s' % (pet.animal, pet.sound),1)
   
  # Select and find on fields
  log('Selecting specific pets')
  # What does my dog say?
  for pet in pets.select(animal = 'dog'):
    log('Selected my %s; it says %s' % (pet.animal, pet.sound),1)
   
  # Can I find my cat?
  pet = pets.find(animal = 'cat')
  log('Found my %s; it says %s' % (pet.animal, pet.sound),1)
   
  return pets.count() == 4

The base view class, which has no indexes, no persistence, and no support for sorting, is called an Oodle.

The results of the test?

Creating pets
  Dog saved, 1 pets
  Cat saved, 2 pets
  Aardvark appended, 3 pets
  Hippopotamus appended, 4 pets
Selecting all pets
  My dog says woof
  My cat says meow
  My aardvark says snorf
  My hippopotamus says hrooonk
Selecting specific pets
  Selected my dog; it says woof
  Found my cat; it says meow
Oodle OK

Update: We've hit version 0.02 wih a successful hash-table implementation.  Next up is persistence...  And deletes.

Update: 0.03!  I deleted my pet hippopotamus!

Update: 0.04!  The idiom for pet in pets now works.  You can't slice it or select within it yet.

Posted by: Pixy Misa at 11:34 PM | Comments (8) | Add Comment | Trackbacks (Suck)
Post contains 356 words, total size 3 kb.

Stuff







Come to think of it, I need to redo them to insert Jy and Pi.  So there may be another colour shift coming.

Posted by: Pixy Misa at 08:49 PM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 24 words, total size 1 kb.

Can't Sleep, Brain Busy

Idea popped into my head for a story set in the Mina Smith universe.  Mina's a customs agent, but this time our protagonist is an accountant.  As much an accountant as Mina is a customs agent, anyway.

Just a snippet that I'll likely never finish, but anyway...
more...

Posted by: Pixy Misa at 06:38 AM | Comments (4) | Add Comment | Trackbacks (Suck)
Post contains 1937 words, total size 12 kb.

Dear Dell Australia

I can't configure a server on your site with more than 4GB of RAM or SATA drives bigger than 250GB.  Last time I checked it was not 2002, so could you please FIX IT?

Posted by: Pixy Misa at 02:15 AM | Comments (1) | Add Comment | Trackbacks (Suck)
Post contains 37 words, total size 1 kb.

Pixy, the Language

Well, sort of.  Kind of sort of.*

Okay, quick, tell me what language this is (without Googling the source code):

print "Eratosthenes' Sieve, in some funny language"

function print_sieve (limit):
  local sieve, j = { }, 2
  while j<limit:
    while sieve[j]:
      j=j+1
    print(j)
    for k = j*j, limit, j:
      sieve[k] = true
    j=j+1

print_sieve(100)

Hint: That's not it.  And I don't understand the first line of print_sieve at all.  Oh, right.  Logically, (sieve, j) = ({}, 2), so the local variables sieve and j are initialised as an empty dict and 2, respectively.

Hint the second: It's the same language as this (believe it or not):

map    = |f,x| x ? %{ hd=f(x.hd), tl=map(f,x.tl) }
filter = |p,x| x ? p(x.hd) ? %{ hd=x.hd, tl=filter(p, x.tl) }, filter(p, x.tl)
take   = |n,s| n<=0 ? { }, { s.hd, unpack(take(n-1, s.tl)) }
ints   = %{ hd=1; tl=map (|x| x+1, ints) }

f = |seq| %{ hd=seq.hd; tl=f(filter (|x| x%seq.hd~=0, seq.tl)) }
primes = f (ints.tl)

table.print(take (100, primes))

Which implements the exact same function.

Hint the third: You'll be able to script your mee.nu blog like this soon.

* Good language designers borrow.  Great language designers swipe someone else's metaprogramming project.

Posted by: Pixy Misa at 01:48 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 199 words, total size 2 kb.

Rhymes With Camel

I was looking at YAML as a serialisation option for Pita (it's already supported for exporting data in the development version of Minx).

So I ran some benchmarks.

It's sloooooooooooooooooooow.

Here we are, encoding and decoding a 2k record:

[andrew@eineus ~]$ python jsonbench.py
10000 iterations on 1973 bytes
Python
json: 10000 encodes+decodes in 4.9 seconds, 2055.8 per second
simplejson: 10000 encodes+decodes in 0.5 seconds, 20686.2 per second
pickle: 10000 encodes+decodes in 4.3 seconds, 2338.0 per second
cPickle: 10000 encodes+decodes in 0.6 seconds, 17297.6 per second
yaml: 10000 encodes+decodes in 213.4 seconds, 46.9 per second

json is Python's built-in JSON library, which is written in Python and thus somewhat sluggish.  simplejson is the same JSON library with an optional C implementation.  Essentially the same applies for pickle vs. cPickle.

yaml is PyYAML, which includes a C implemenation if you have LibYAML installed.  Which I do, but I can't seem to get the C implemenation to run...  Unless that's it, which would be pretty sad.

On the one hand, simplejson is the fastest of these options, which is good because it's also the most widely supported format and the easiest to parse.

One the other hand, 20,000 records per second is not all that much.

Posted by: Pixy Misa at 04:37 PM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 205 words, total size 1 kb.

Jsyn

Announcing Jsyn, the JSON syndication format for blogs and everything else.

Specification for version sqrt(-1):

1. Put a bunch of stuff in a data structure.
2. JSON-encode it.
3. Make it available somehow.

...

What, you need more of a spec?  But I've registered a domain and everything!

Seriously, freedom from so-called "simple" syndication, coming soon!  Part of the Pita* project.

Update: Spec version -1:

A Jsyn feed object will have precisely three first level sub-elements:

1. A feed element, an object containing the feed properties (required).
2. A schema element, an object containing advisory schema information (optional).
3. A items element, an array of objects representing the data items (required, but may be empty).

Example:

{"feed": {"source": "http://ai.mee.nu/feed.jsyn"},
 "schema": {"source": "http://jsyn.net/schemas/blog.jsyn", "version": 1.0},
 "items": []}

A client may use a local copy of the schema so long as the version matches that specified in the schema object.  The server must increment the version when updating the schema.  The server may revert to an older schema with a lower version number; the client must not continue to use the local copy of the schema in this case.

* Which is part of the Minx project**, which is part of the make-Pixy-rich-or-drive-him-insane-either-is-fine project.

** I've subdivided Minx into three parts, like Gaul, only with less garlic: Minx, the bliki; Meta, the template, formatting, and scripting engine; and Pita, the database engine/abstraction layer.  In addition, there's Miko, the planned desktop client.

Posted by: Pixy Misa at 01:17 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 240 words, total size 2 kb.

Syntactistaticality

Now where the heck was I, before being buried under an avalanche of poorly-considered Atom feeds and Chinese replica watch spam?

Ah, right.

We can't do a full Progress-style where clause in Python, unfortunately.  Or not without more trickery than I intend to apply; someone did make a working goto - never mind that, a working comefrom - but I'm not inclined to go to that sort of length.

So.  I want the first 20 posts in a given folder of a given blog, sorted by date order (descending, of course).  Pythonically.  No SQL.  Let's see:

db = Pita.Connect(host, user, pass, database)
posts = db.views.posts

I've connected to the Pita server and have a view open.  Now:

for post in posts(folder=f,order='date-',limit=20):
    ...

That's not bad.  With Python's named parameters, you can use any field in a flat record structure, so:

authors = db.views.authors
a = authors.find(name='Pixy Misa')
for post in posts(author=a.id, tag='databases',order='score-',limit=20):
    ...

If we have a nested structure, though, it doesn't work.  Python doesn't let you say:

for post in posts(author.name = 'Pixy Misa')

even if we have the code to automatically resolve the relation.  It's not valid syntax.  So that's one place where it breaks down.  Another place is ranges; we can say

for author in authors(country = 'Australia')

to get a list of authors who live in Australia, but we can't say

for author in authors('Andorra' < country < 'Azerbaijan')

even though that is a valid Python expression.  It will get evaluated, and we'll just pass either True or False to authors() (or throw an exception), and it just won't work.

Now, the design of Pita is that it's primarly a document database with advisory schemas.  It's not schemaless like many or the key-value stores, and it's not fixed-schema like most traditional relational databases.  Each view has a schema, which specifies what fields should be there, and if they are, what type they should be.  Fields can be missing, in which case the schema may specify a default value.  And you can stick in whatever additional data you want, so long as the schema doesn't specifically conflict with that.

What this means is that we can know that country is a string, and if we do an equality comparison between a string and a list, we mean that we want to know if the string is in the list.  So we can also do this:

for author in authors(country = ['Australia', 'New Zealand', 'Canada'])

to get authors from any of those countries.

By returning a generator or iterator, we can efficiently replace this:

for post in posts(blog=b,tag='databases',order='score-',limit=20)

with the more Pythonic

for post in posts(blog=b,tag='databases',order='score-')[:20]

Slicing (as it is called) is very general in Python and very useful, so adapting it to database selects will come naturally.

But what about range searches?  There's no obvious Pythonic syntax for this, at least, not one that works.  Here are a few possiblities:

for house in houses(price = '<100000'):
    ...
for house in houses(price = ['>50000','<100000']):
    ...


We know price is of type money, so we look at that string, and the leading < means it's a range match.  Goody!  Doesn't work so well - or at all, for that matter - for strings, because we could be perfectly well looking for those exact strings. 

We could have an explicit range function:

for house in houses.price.range(50000,100000):
    ...

That's not too bad either; it's pretty clear syntactically and semantically, and it requires no parsing.  Doesn't let you differentiate between > and >= though - and you can't do a range match on more than one field.  (You can't effectively use a binary tree for such a search anyway.)  We can still slide in our other parameters like so:

for house in houses.price.range(50000,100000,suburb='Wondabyne'):
    ...

But (again due to the strictures of Python), they must come last.

Since we're building a generator, it should be possible to do this LINQish trick:

for house in houses(suburb='Wondabyne').price.range(50000,100000):
    ...

The first operation produces a view that knows to search on the suburb field for Wondabyne.  This derived view has the exact same attributes of the original view, and price is one of those attributes, and we can use the range selector on price just like before.

We should be able to keep doing that sort of thing, until we get something like:

for house in houses.suburb('Wondabyne').bedrooms.range(3,5).bathrooms.min(2).price.max(150000).order('price+'):
...

But it's not terribly dynamic.  So, next stop, dynamism.

Posted by: Pixy Misa at 03:13 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 701 words, total size 5 kb.

3 Across

5 letters, priority queue.

Posted by: Pixy Misa at 10:48 PM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 6 words, total size 1 kb.

Die, Atom, Die!

Atom feeds are evil and must be killed.

Posted by: Pixy Misa at 09:14 PM | Comments (3) | Add Comment | Trackbacks (Suck)
Post contains 11 words, total size 1 kb.

That Is Not The Way To Do It!

I've mentioned LINQ, Microsoft's attempt to make accessing datasources more idiomatic.  And I've mentioned IronPython, Python running on Microsoft's .NET platform.  Since LINQ is part of the .NET library, IronPython can use LINQ.  So I went looking for examples, and found this:

songs = ThenBy(OrderByDesc(  
          Select(content.Elements(xhtml.ns +'tr'), ScrapeSong),   
          lambda s: s.added), lambda s: s.artist)

That is not only not the right way to do it, it actually leaves me wondering if it is possible to construct a more wrong way to do it, short of INTERCAL.

Posted by: Pixy Misa at 05:27 AM | Comments (2) | Add Comment | Trackbacks (Suck)
Post contains 92 words, total size 1 kb.

All My Servers Are No Longer The Same Speed

I was planning to spend the weekend working with MongoDB, but those plans evaporated when it crashed and destroyed my test database. So instead I dug out my toy Python benchmark and ran it on Eineus. And just for fun, did the same in Psyco and Cython and Jython. Results are... Mixed. Yeah, that's a good word, particularly since the IronPython benchmark is still running.

System	CPU	Clock	Python	Loop	String	Scan	Total
Eineus	Phenom II 945	3.0GHz	2.6.4/32	0.950	1.483	0.437	2.870
Eineus	Phenom II 945	3.0GHz	2.6.4/Pysco	0.013	0.180	0.477	0.670
Eineus	Phenom II 945	3.0GHz	2.6.4/Cython	0.000	84.750	0.490	85.240
Eineus	Phenom II 945	3.0GHz	2.5.1/Jython	0.682	499.936	0.758	501.376
Nagi	Phenom 9750	2.4GHz*	2.6/IronPython/32	0.544	3502.652	1.541	3504.739
Nagi	Phenom 9750	2.4GHz*	2.6/IronPython/64	0.916	5399.020	1.264	5401.202
Potemayo	Core 2 Duo	1.8GHz	2.6/IronPython/32	0.559	1.632	1.567	3.759
Miyabi	Phenom II 945	3.0GHz	2.6.4/64	0.637	1.003	0.530	2.170
Akane	Opteron	2.0GHz	2.5	1.887	2.733	0.880	5.500

* Normalised to 3.0GHz for ease of comparison.

I'll paste in the IronPython results if it ever finishes. (Update: Done now.)

Some notes:

64-bit Python is now a good bit faster than 32-bit for many cases. It's actually a bit slower in string scanning; I don't know why.

A 3GHz Phenom II running Python 2.6 is 2x faster than a 2GHz Opteron running Python 2.5 from 3 years ago. Someone's been doing some good work, either the Python people or AMD or the Gnu compiler team.

CPython (the standard Python) has some really neat string optimisations that I depend on in Minx. These flow through nicely to Psyco, but are conspicuously absent from Cython, Jython, and IronPython, which are 60, 400, and a couple of thousand times slower for heavy string concatenation (as I said, that benchmark hasn't finished yet...) It's certainly possible to avoid that idiom, and instead, for example, create a list of substrings and then join them all in one operation.

Apart from that, Jython seems to perform fairly well; certainly, if you need to run heavily multi-threaded Python code and can avoid doing millions of concatenations of large strings, Jython could be a winner. The Python interpreter can only run one thread at a time, though other threads can be handling I/O or library functions. The Jython runtime is fully multithreaded, so if you have a multi-threaded application and more than two CPUs - which I do - then Jython can provide an overall performance boost even if the single thread performance declines somewhat. (And it's actually faster on one of the tests, so depending on your code you might win both ways.)

As for IronPython, well, the string concatenation results are just terrible. Looping is comparable with Python or Jython (but far behind Psyco or Cython), and string scanning is the slowest of the lot, though only by a factor of 2, not 2000. It should be fast enough for most tasks as long as you really avoid concatenating large strings. I wonder what list performance is like - I'll have to add a test for that.

Bumped and updated: Tested this just now on Potemayo with a StringBuilder implementation for the strcat part of the benchmark, which delivered about 2000x faster performance (!) on that part of the benchmark and 1000x on average, which would make a .Net deployment of Minx pretty feasible.  It would be nice if all these people deploying immutable string libraries could also deploy the Python concatentation performance trick, because StringBuilder is not a general-purpose string library, just a faster way to concatenate and modify a stringlike object.

Posted by: Pixy Misa at 05:09 AM | Comments (1) | Add Comment | Trackbacks (Suck)
Post contains 527 words, total size 5 kb.

Obectification

Just pootling about benchmarking stuffs again, which is what I do when I'm too lazy to actually code something but want to appear productive.  This time I'm seeing how many little objects Python can create per second.  Here's the code:

import time

class iwb(int):
    def __setattr__(self, k, v):
        pass
    def __getattr__(self, k, default=None):
        return None

def perf(count):
    t0 = time.time()
    for i in range(count):
        j = iwb(i)
    print '%s per second' % (count / (time.time() - t0))

perf(10000000)

And here are the numbers:

Python 2.6.4/32: 3044540.34221 per second
Psyco 2.6.4/32: 5378755.4303 per second
Jython 2.5.1: 795102.161706 per second  (significantly slower, but better than I'd expected)
Cython 2.6.4/32: 3830629.40698 per second (faster than Python, slower than Psyco)
Python 2.6.4/64: 3617701.37013 per second (again nearly 20% faster than the 32-bit version)
IronPython 2.6: 1931247.06236 per second (On my laptop, a 1.8GHz Core 2 Duo, compared to the 3GHz Phenom II for the preceding results, so not too bad.  I need to see if it works under Mono.)

The reason I'm fiddling with this is that I'm looking at an optimisation/simplification for Minx, but it would involve creating about 10,000 objects to load the current main page of my blog.  It's part of some syntactical trickery I'm trying to work out that creates virtual methods on objects in trees....  Without the objects being visibly objects, which involves some hackery when you put them in the tree in the first place.

Posted by: Pixy Misa at 03:16 AM | No Comments | Add Comment | Trackbacks (Suck)
Post contains 236 words, total size 2 kb.