- Recent
- Popular
- Tags (0)
- Subscribers (1)
- Scaling memcached at FacebookDecember 12 2008
- If you've read anything about scaling large websites, you've probably heard about memcached. memcached is a high-performance, distributed memory object caching system. Here at Facebook, we're likely the world's largest user of memcached. We use memcached to alleviate database load. memcached is already fast, but we need it to be faster and more efficient than most installations. We use more than 800 servers supplying over 28 terabytes of memory to our users. Over the past year as Facebook's popularity has skyrocketed, we've run into a number of scaling issues. This ever increasing demand has required us to make modifications to both our operating system and memcached to achieve the performance that provides the best possible experience for our users. Because we have thousands and thousands of computers, each running a hundred or more Apache processes, we end up with hundreds of thousands of TCP connections open to our memcached processes. The connections themselves are not a big problem, but the way memcached allocates memory for each TCP connection is. memcached uses a per-connection buffer to read and write data out over the network. When you get into hundreds of thousands of connections, this adds up to gigabytes of memory-- memory that could be better used to store user data. To reclaim this memory for user data, we implemented a per-thread shared connection buffer pool for TCP and UDP sockets. This change enabled us to reclaim
- How to Hack Out an Election CounterNovember 19 2008
-
As part of a team flushing out Facebook’s 2008 Election campaign designed to get more people to register to vote and then actually vote, I was responsible for the message on top of the News Feed. We had our design and messaging ready to go as the weekend before election day was approaching. The message was up and ready to be turned on. During the week, someone had suggested that we create an “I voted!” button that a user could click and tell his or her friends about the patriotic action of the day. I was excited; clearly this would encourage more people to go vote so they could tell their friends. So I buckled down, flushed out a News Feed and Minifeed story, and checked it in with plenty of time before Tuesday.
On Sunday someone threw out an idea to have a counter that people could watch and see that their vote was indeed being counted on Facebook as well. The election team decided this would be awesome functionality during a quick chat on Monday morning. I had about 12 hours to design a counter that could handle millions of clicks and live update via ajax regularly enough to be exciting on millions of browsers at the same time.
Here’s the last-second solution I came up with.
- Every time a user clicked “I voted!”, I logged a message to our internal open-source logging system Scribe that included time, user id, and which U.S. State the user is in.
- I wrote a quick aggregation script that
- Facebook's Scribe technology now open sourceOctober 24 2008
- Here at Facebook, we're constantly facing scaling challanges because of our enormous growth. One particular problem we encountered a couple of years ago was collection of data from our servers. We were collecting a few billion messages a day (which seemed like a lot at the time) for everything from access logs to performance statistics to actions that went to News Feed. We used a variety of different technologies for the different use cases, and all of them were bursting at the seams. We decided to build a unified system (called Scribe) to handle all of these cases, and do it in a way that would scale with Facebook's growth. The system we built turned out to be enormously useful, handling over 100 use cases and tens of billions of messages a day. It has also been battle tested by just about anything that can go wrong, so I encourage you to take a look at the newly opened Scribe source and see if it might be useful for you. To give the code some context, I'm going to go through the major design decisions we made to allow the system to scale. The first decision we made was to not lock ourselves into a particular network topology. The Scribe servers are arranged in a directed graph, but each server only knows about the next server in the graph. This flexible topology allows for things like adding an extra layer of fan-in if the system grows too large, and batching messages before sending them b
- The All-Night Hackathon Is Back!October 24 2008
- Every few months, our engineers unleash their talents in one epic, all-night coding session. These are the Facebook Hackathons. They start with takeout Chinese food around 8 p.m. and end with a dawn breakfast at any pancake house or donut shop that will have us. In between, dozens of Facebook engineers create working prototypes of projects that they always wanted to build but couldn’t ever pursue during their regular hours. Hackathon XI – also known as The Presidential Hackathon – will take place the evening of Wednesday, Nov. 5. In past years, Facebook Hackathons have been the starting point for all sorts of new features that rapidly became mainstays of the site, such as Facebook Chat, internationalization, the type-ahead feature in search, the friend suggester, etc. etc. Not every experiment pays off right away, but it’s a low-risk, high-reward setting that encourages engineers to bring to life some of the great ideas that are floating around here. We have plenty of in-house ideas already in the mix for Hackathon XI, but there’s no reason we can’t welcome outside input, too. If Facebook users have ideas about new features or site improvements that could be tackled in our one-night time frame, please accept our invitation to submit those concepts to the address hackathon-ideas@facebook.com. The deadline is noon Pacific time, Monday, Nov. 3. In doing so, you agree to our standard
- 10 billion photosOctober 14 2008
-
We recently hit a really cool milestone, our users have now uploaded over 10 billion photos to the site. Now, that’s a big number, but we actually store four image sizes for each uploaded photo, so that’s over 40 billion files.
To celebrate, we got a bunch of cupcakes and handed them out to our engineering and operations groups. One of our engineers calculated that if we had gotten one cupcake for each of our photos, and lined them up side by side, the line could reach halfway to the moon.
Here’s some other interesting recent stats on photos:
- 2-3 Terabytes of photos are being uploaded to the site every day
- We have just over one petabyte of photo storage
- We serve over 15 billion photo images per day
- Photo traffic now peaks at over 300,000 images served per second