Toluu

Twitter Technology Blog

On SpecializationSeptember 26

It's been a while since we last shared what's going on with our engineering efforts here at Twitter.  In the past few months since our acquisition of Summize, we've grown large enough to be able to specialize.  More than just an engineering team and an operations team, the technical part of Twitter now has teams for search, user experience, back-end services, and so forth.  Your humble author is leading up the API team.

This specialization is letting us do great things.  The UX team has launched a well-received redesign and has a number of performance and usability improvements in store for the site.  The search team has been fighting spam and working with operations to meet the growing demands on search.twitter.com.  Engineers working on back-end services are getting ready to test a major overhaul of how we store and deliver timelines of tweets.  On the API side, we've just completed a strict, robust test suite for the API that will allow us to make major changes and deliver our next big release with confidence.

When I started at Twitter in early 2007, everyone in the company had their hands in everything, as is typical of early-stage startu

Status of the World - More than just human updatesAugust 13

First, let me introduce myself (Abdur Chowdhury) I am one of the founders of Summize and now Chief Scientist at Twitter. In my quest to understand the Twitter data repository I have had some interesting revelations and will post them here from time to time. My first revelation is it’s not just for humans anymore. Twitter's original question - "What are you doing?" now seems to be catching on with some machines. This is an important addition to the communication channel as it allows people to not only know what their friends are doing but also get the status of inanimate things from the world around us.

In this post, I’ll share some of my favorite examples of this trend like the Chandra X-Ray observatory that posts its location as it circles the globe every 20 minutes or so.


While knowing where a satellite is interesting, some examples that might be more helpful are Red Jets automatic posting of the arrival and departure of their ferries. Sydney Australia posts the traffic issues for the enti

Summize and The Future of the Twitter APIJuly 15

Earlier today we announced that Twitter has acquired Summize, the leading Twitter search service. I'd like to explain how this change will impact the Twitter developer community.

Summize — now search.twitter.com — has their own API that a number of applications are already using. Over at the Twitter Development Talk group I've been encouraging developers with search and data-mining needs to investigate the Summize API. Developers can now trust that Summize has the entire corpus of public Twitter updates at their disposal for search going forward.

For the time being, the Summize API will live at search.twitter.com as a separate entity from the main Twitter API. We'll try to get the two APIs as coordinated as possible in the short term, but merging Summize's architecture with ours will be a gradual process. That said, we have a clear opportunity for cleanly merging the two APIs.

The Twitter API has much room for improvement. As it's grown over the last year and a half, our API has become increasingly less RESTful. Resource names need to be syn

You've Got Q's, We've Got A'sMay 29

We had a lot of feedback on our architecture update. In the spirit of continued openness and transparency, I'd like to address a number of the questions that came up in the comments on that post.

Donnie asks if we're making a slow exodus from Ruby, and Scabr asks if our key problems were Ruby problems.  We've got a ton of code in Ruby, and we'll continue to develop in Ruby with Rails for our front-end work for some time.  There's plenty to do in our system that Ruby is a great fit for, and other places where different languages and technologies are a better fit.  Our key problems have been primarily architectural and growing our infrastructure to keep up with our growth.  Working in Ruby has been, in our experience, a trade-off between developer speed/productivity and VM speed/instrumentation/visibility.

RBL asks how we're instrumenting our application. We've used DTrace on a couple of occasions, but for the most part we do a lot of print-style logging to a combined syslog. We also recently ad

Twittering About ArchitectureMay 22

Here at Twitter HQ, we're not blind to the flurry of discussion over the past weeks about our architecture. For many of our technically-minded users, Twitter downtime is an opportunity to muse about what the source of our problems might be, and to propose creative solutions. I sympathize, as I clearly find our problems interesting enough to work on them every day.

Part of the impetus for this public discussion extends from the sense that Twitter isn't addressing our architectural flaws. When users see downtime, slowness, and instability of the sort that we've exhibited this week, they assume that our engineering progress must be stagnant. With the Twitter team working on these issues on and off for over a year, surely downtime should be a thing of the past by now, right? Shouldn't we be able to just "throw more machines at it"?

To both rhetorical questions, the answer is "not quite yet". We've made progress, and we're more scalable than we were a year ago, but we're not yet reliably horizontally scalable. Why? Because there are significant portions of our system that need to be rewritten to meet that goal.

Twitter is, fundamentally, a messaging system. Twitter was not architected as a messaging system, however. For expediency's sake, Twitter was built with technologies and practices that are more appropriate to a content management system. Over the last year and a half we've tried to make our system behave like a messaging system as