Monday, October 18
Anyways, there's this site called Foursquare, and apparently it fall down go boom just recently. Possibly more than once.
They're running MongoDB, a database I tested and rejected earlier this year when looking for more elegant substitutes for MySQL. I don't like SQL - never have - but it works, and MongoDB... Doesn't.
Foursquare had two database servers with their database split ("sharded") across them. The servers are Amazon EC2 instances, with 66GB of RAM each, fairly large by most people's standards. The problem arose when one of the database shards got bigger than the available RAM.
...
If you had any experience with databases before, say, 2007, you'll be shaking your head in disbelief at this point. If you had any experience before, say, 1997, you'll be dumbfounded. Yes, the site fell over because the database didn't fit in RAM.
The total database size at the time of the outage was around 120GB. At my day job - and though Foursquare is a small startup, my day job is at an even smaller startup - we add 400GB of data to our databases every single day. Mind you, we do things to MySQL that would make the average DBA give up and go into volcanology as a safer bet, but it works.
Now, when I was considering MongoDB, the first thing I did was test its performance. The second thing I did was test how it behaved in out-of-memory conditions. (It crashed.) Seems that Foursquare forgot to ask the second question.
Posted by: Pixy Misa at
11:54 PM
| Comments (8)
| Add Comment
| Trackbacks (Suck)
Post contains 254 words, total size 2 kb.
Posted by: Pete Zaitcev at Tuesday, October 19 2010 12:56 AM (9KseV)
Without sharding, their performance would have degraded gradually as the working set exceeded available RAM; if you can avoid full table scans, 2x RAM is still fast, and 3x is pretty good (modulo virtual-server bugs). With sharding in its current state, they either needed to start out with twice as many shards, or put SSDs into them to reduce the cost of paging (not really an option in Amazon's cloud).
I've spent a lot of time talking with the guys at 10gen, and they're smart and capable, but I make sure I never forget that Eliot's dev laptop has a fast Core i7, 8GB of RAM, and a 512GB SSD. :-)
(...and my MongoDB server would dance with joy if it had 66GB of RAM or an SSD...)
-j
Posted by: J Greely at Tuesday, October 19 2010 01:14 AM (2XtN5)
Posted by: Pixy Misa at Tuesday, October 19 2010 12:33 PM (PiXy!)
Significantly, when I had Eliot and Roger on site a while back, one of my questions was about dealing with fragmentation. At that time, it had only just become possible to query the server and find out how badly fragmented you were, and the now-urgent online defragmentation (as opposed to tedious copy/promote or offline repair) was still in the "yeah, we can do that at some point" stage.
After a few months of deliberate testing, my collated daily archive collection is now heavily fragmented, to the point that even though it has over 400GB of free space, it still occasionally allocates a new 2GB extent for indexing. Someday I'll need to free up a terabyte on another server, copy the collection to it, then make it the new master. Before I fill the disk with half-empty datafiles.
-j
Posted by: J Greely at Tuesday, October 19 2010 02:27 PM (2XtN5)
Posted by: Steven Den Beste at Tuesday, October 19 2010 02:40 PM (+rSRq)
Posted by: Pete Zaitcev at Wednesday, October 20 2010 01:04 AM (9KseV)
Posted by: Steven Den Beste at Wednesday, October 20 2010 02:09 PM (+rSRq)
They get huge buzz because it's easy to throw something together in a real hurry and pitch the demo to an investor, usually in combination with Ruby On Rails or another rapid-prototyping framework. And, just like Rails, they tend to fail in interesting ways when you take your demo and put it into production. The MongoDB rant Pixy linked to covers it nicely, and the followup on Ruby nails it to the wall (while sadly overstating the virtues of PHP, but that's a side issue...).
It's easy to start throwing data into MongoDB and slapping a web interface on it with Rails. It's also often a substitute for defining the actual problem and designing the right solution. Pixy's blogging on the subject is more effort than most people put into their VC pitch, much less their code. :-)
-j
Posted by: J Greely at Wednesday, October 20 2010 04:45 PM (2XtN5)
56 queries taking 0.2817 seconds, 334 records returned.
Powered by Minx 1.1.6c-pink.
