Pablo Castro's blog

ADO.NET Data Services in Windows Azure: pushing scalability to the next levelNovember 1

The announcement of Windows Azure is a big milestone for us in the Astoria team. We got a chance to add our little contribution to the platform by providing data service interfaces for a couple of the Azure services.

Currently there are two services that use the ADO.NET Data Services runtime: the Windows Azure Tables Service, which was announced this week as part of the whole Windows Azure story, and SQL Data Services, which has been around for a while but got a new experimental Data Services interface this week to coincide with the PDC.

These services -and others that will come in the future also based on Data Services- share a common aspect: they have extreme scalability requirements.

In order to enable them to use our Data Services server runtime we had to extend the data service framework to make it scale in various new dimensions. In the rest of this post I'll summarize some of the walls we hit and the changes we made to the system to handle these scenarios.

Things that already scaled

The Data Services runtime already incorporates many design principles that help with scalability.

For example, the system does not keep any required state between requests (we do cache stuff, but we can throw it away at any time), so scale out of the front-end servers

Now you know...it's Windows AzureOctober 28

Since we shipped ADO.NET Data Services v1 in .NET 3.5 SP1 (and actually before that as well) I've been working on a few things that I could share (such as offline/sync support for data services) and some that I couldn't discuss publicly until all the big plans where announced.

This week at PDC Microsoft announced Windows Azure. A lot has been written about it, so I won't go into the details.

On our side, in the data services team, we made our small contribution to the big picture.

The Windows Azure table service is a structured storage facility that's part of the core part of Azure. Access to the table service is done through a data-services compatible RESTful interface that uses the Astoria conventions over an HTTP binding. That means that you use either any client with an HTTP stack to talk to it, or you can use the ADO.NET Data Services client, which does a nice job exposing data as .NET objects, letting you write simple queries using LINQ instead of URLs, etc.

Another cool thing about the table service (and the blobs and queuing service for that matter) is that they are accessible both from the virtual compute environment and from anywhere in the Internet. In both cases, if you're using .NET, you can use the data services client to interact with it. In the case of code running in the Windows Azure hosting environment, the client is already present (the environment includes .NET 3.5 SP1) so you can use

Trying something: short videos in our design notesSeptember 28

During the design of Data Services (Astoria) v1 we did the transparent design thing. We're quite happy with the result, we got a lot of feedback and were able to adjust many aspects of the project based on that.

Now that we're in full swing with v2 design work, we're going to be posting regularly again (hopefully :).

Andy got a new tiny camera and he had to use it for something...so he suggested that we do short clips that may help explain design points. Sometimes a short explanation is just much better than a bunch of writing.

The first post with the write-up + short video format discusses "Friendly Feeds" and is here. If you get a change to take a look, let us know what you think about the new style (and friendly feeds)!

-pablo

Timeline of Project AstoriaAugust 20

It's amazing how much information is there in our email archives. Now that we've shipped the thing, I thought I would share my summarized (still long), partial view of how the ADO.NET Data Services Framework ("Project Astoria") came to be. I left out a ton of partners, important events and features that came and went for the sake of brevity, but most key points in time are there.

Idea
Hack
Pitch
Prototype
Review
Announcement
Team
Design
Release
Future

Idea

September July 2006. We were having lunch at the building 35 cafeteria with Alex Barnett (back then our Community Program Manager) and he brought up that a bunch of people out there were doing

Coming from under the rocks just to celebrate for a bitAugust 11

I've been sort of under a rock for a while, but I thought I'd come out for a minute to celebrate. Today we made available .NET 3.5 SP1 and Visual Studio 2008 SP1. There are two components in the release I spent a bunch of time on, which interestingly enough have very different origins and get to RTM through very different processes. One is the ADO.NET Entity Framework, which has been cooking for several years and survived controversies, comparisons with non-shipped previous attempts and other natural disasters; the other one is the ADO.NET Data Services Framework or Project Astoria, which was built, well...fast.

I won't go into details of the release, folks have discussed the Astoria, Entity Framework and general data-related features in the release already.

Why have I been under a rock? In the last few months I've been spending time working on various things related to Astoria, online services and data interfaces. Some I can discuss, some will need to wait a bit until the stakeholders are comfortable to talk about it publicly.

Moving Astoria as a framework forward: we were ready (modulo bug fixing and last minute tweaks) some time ago, and we've been thinking abou