Without sharding, their performance would have degraded gradually as the working set exceeded available RAM; if you can avoid full table scans, 2x RAM is still fast, and 3x is pretty good (modulo virtual-server bugs). With sharding in its current state, they either needed to start out with twice as many shards, or put SSDs into them to reduce the cost of paging (not really an option in Amazon's cloud).

I've spent a lot of time talking with the guys at 10gen, and they're smart and capable, but I make sure I never forget that Eliot's dev laptop has a fast Core i7, 8GB of RAM, and a 512GB SSD. :-)

(...and my MongoDB server would dance with joy if it had 66GB of RAM or an SSD...)

-j

Significantly, when I had Eliot and Roger on site a while back, one of my questions was about dealing with fragmentation. At that time, it had only just become possible to query the server and find out how badly fragmented you were, and the now-urgent online defragmentation (as opposed to tedious copy/promote or offline repair) was still in the "yeah, we can do that at some point" stage.

After a few months of deliberate testing, my collated daily archive collection is now heavily fragmented, to the point that even though it has over 400GB of free space, it still occasionally allocates a new 2GB extent for indexing. Someday I'll need to free up a terabyte on another server, copy the collection to it, then make it the new master. Before I fill the disk with half-empty datafiles.

-j

They get huge buzz because it's easy to throw something together in a real hurry and pitch the demo to an investor, usually in combination with Ruby On Rails or another rapid-prototyping framework. And, just like Rails, they tend to fail in interesting ways when you take your demo and put it into production. The MongoDB rant Pixy linked to covers it nicely, and the followup on Ruby nails it to the wall (while sadly overstating the virtues of PHP, but that's a side issue...).

It's easy to start throwing data into MongoDB and slapping a web interface on it with Rails. It's also often a substitute for defining the actual problem and designing the right solution. Pixy's blogging on the subject is more effort than most people put into their VC pitch, much less their code. :-)

-j