In Search of a Highly Available Persistence Solution

Our cloud-based SaaS offerings are hosted in Azure. Like many applications, some of our apps began life in the era before cloud computing became mainstream. As a result many design decisions had to be rethought. Some involved some rather extreme makeovers just to be able to run there – like removal of CLR stored procedures (gah!) since SQL Azure didn’t support it. Others were more fundamental to multi-tenant apps, but required change nonetheless.

Along the way, we have made numerous changes that brought higher performance, stability, scalability and reliability to the products. Frankly, the Azure compute infrastructure for PaaS is excellent. Combined with Windows Azure Traffic Manager our compute has performed admirably. Scaling is a snap. Deployment is beyond easy. Then we hit the snag. Persistence. State sucks. Compute is easy because it can be treated statelessly; but the persistence layer is another story. If your database goes belly up, you’re dead. Having an offline copy of your database could help, right? But then you think, “How recently did you make that copy?” Or, “What about data loss between backups? My user really wants to see that last transaction!” And once your original database is back online, you ask yourself, “How do you sync up the changes?” Wouldn’t it be great if that copy was a transactionally-consistent copy?

SQL Azure touts high availability in the datacenter through its replicas. Every database is actually implemented as a master and two replicas. Their stated goal is 99.9% availability. On paper, you might think that having two replicas of your data would be sufficient to give you high availability. Our experience is that it is not. They do not offer a cross-datacenter high availability option. In recent conversations with Microsoft, surprisingly they are pointing people towards SQL Server in IaaS for high availability. Their Tutorial: Database Mirroring for High Availability in Azure further supports this view. C’mon guys – you can do better than this! This is a cop out. I want a scalable, highly available, cloud-ready repository.

Edit: The new database SKUs for Azure SQL Database (basic, standard, and premium) maintain “at least three replicas”. Please refer to this overview of the new SKUs for more information. I want to point out that our current product deployments are using the GA SKUs (Web and Business) and not the new SKUs outlined in this overview.

Time for another design check. In reality, what your application must do to get closer to the magical 5 9’s of availability is to move replication logic into the application. If you’re contemplating more than one form of persistence (we are) and if you want high availability (we do), then be prepared to roll your own. A number of architectural design sessions and a review of the options available to the Azure ecosystem makes it clear that they don’t have a complete answer here. It’s not entirely surprising. They give you the primitive infrastructure and framework bits and you do the rest. In this case the platform affords you nothing leaving you to do the rest a.k.a. everything else…

Moreover, if you intend to be a persistence polyglot, your options are complicated. How many of you put everything in the database? Does that image really belong in there? How about those PDFs? How about your application configuration and set up information? Couldn’t blob storage do a much better job of saving that binary data? Couldn’t you use a document database to store that configuration data? Modern enterprise applications have a very diverse set of data and using a relational database for all of it means you’re fitting square pegs in round holes.

A number of design patterns can help you to abstract your application business logic from how you persist your data. You *are* using design patterns, aren’t you? A simple pattern to use is the Repository pattern. Done properly, you can completely hide the complexities of how to persist your data completely from the rest of your application leaving you the option of doing synchronous replication to one or more replica copies of your data, which can help to ensure you have a transactionally-consistent copy of your data. Or you may choose to use asynchronous replication for higher performance at the risk of some small amount of data loss. Both approaches require your application to detect failures and to switch from the master to the replica copy.

Other more complex design patterns can help you with this problem at the expense of complexity.Simplicity is king in software design. Embarking on an implementation of CQRS may leave you wondering why you started in a career of software development. But it can also leave you with a superior solution to a complex problem. Moreover, by implementing this in your application, you can decouple yourself from a platform and its requirements for high availability. Like everything in software development – it’s a balance of choices. Flexibility, simplicity, maintainability, speed to market, performance, etc.

Like some on my team are fond of saying, “If it were easy, everyone would play the game.”

Creating a Dependency Injected WCF Data Service

In this post, I’ll explain how to decouple your WCF data service from a specific data context. This is useful in many ways including changing out your context implementation at runtime or mocking out your context for testing purposes. The example code will use MEF (Managed Extensibility Framework), but any dependency injection framework, service locator implementation, or factory pattern could be used.

Getting Started

The first thing you will need to get started is an implementation of DbContext. I’ve chosen to go with a code first approach in this example because it gives me explicit control over the code in the context. Entity Framework 4.1, which is available as a Nuget package, provides a simple, purpose-driven API, which allows me to create a new DbContext and specify the entity types that it is responsible for in just a few lines of code. The base class and EF4.1 plumbing does all the hard work.

public class PersonContext : DbContext
{
    public IDbSet<Person> People { get; set; }
    public IDbSet<Address> Addresses { get; set; }
}

Figure 1: Basic PersonContext implementation

This implementation satisfies the most basic requirements of EF4.1 for a DbContext, but as you’ll see as we go along, we’ll want to flesh it out a bit more in order to support MEF’s requirements.

Give Your DbContext an Interface

Uncle Bob Martin’s SOLID object oriented principles is a must-read for all programmers. The ‘D’ in SOLID is for Dependency Inversion. In a nutshell, dependencies between objects should be based on abstractions not concretions.  Our next step is to create an interface that will represent our DbContext.

 

public interface IDbContext : IDisposable
{
    IDbSet<TEntity> Set<TEntity>() where TEntity : class;
    int SaveChanges();
}

Figure 2: IDbContext interface

 

Initializing Your Context

The EF4.1 DbContext class can take a connection string in as a constructor parameter. For simple cases, this might be enough for you. However, if your context requires additional information to operate properly, you may want create a configuration class that you can inject into your context through its constructor. I’ve take this approach because it allows for greater flexibility in initialization of the context.

[Export]
public class DbContextConfiguration
{
    public string Name { get; set; }
    public string ConnectionString { get; set; }
}

Figure 3: DbContextConfiguration needed to initialize your DbContext

 

[ImportingConstructor]
public PersonContext([Import("PersonContextConfiguration", typeof(DbContextConfiguration))] DbContextConfiguration configuration)
    : base(configuration.ConnectionString)
{
    _configuration = configuration;
}

Figure 4: PersonContext constructor

 

The first thing you’ll notice in the code above are the [Export], [ImportingConstructor], and [Import] attributes that are applied to the DbContextConfiguration class and PersonContext constructor. These are MEF attributes, which are used to support the dependency injection pattern. MEF will handle auto-magically wiring up the dependencies through a call to ComposeParts(). If you are new to MEF or unfamiliar with how it works, here are a couple of useful links to get you started.

 

Managed Extensibility Framework Overview

Building Composable Apps in .NET 4 with the Managed Extensibility Framework

 

Creating Your WCF Data Service

Writing a WCF data service couldn’t be easier. In a matter of a few mouse clicks, you can be serving up REST-based data. Microsoft has dramatically reduced the effort required to implement a service by providing a rich API that provides a lot of functionality under the covers.

Figure 5: Add New Item

Adding a new WCF Data Service through the Add New Item dialog results in an entry point to that API via the DataService<T> class.

   1: public class PersonDataService : DataService< /* TODO: put your data source class name here */ >

   2:     {

   3:         // This method is called only once to initialize service-wide policies.

   4:         public static void InitializeService(DataServiceConfiguration config)

   5:         {

   6:             // TODO: set rules to indicate which entity sets and service operations are visible, updatable, etc.

   7:             // Examples:

   8:             // config.SetEntitySetAccessRule("MyEntityset", EntitySetRights.AllRead);

   9:             // config.SetServiceOperationAccessRule("MyServiceOperation", ServiceOperationRights.All);

  10:             config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V3;

  11:         }

  12:     }

Figure 6: Boilerplate data service code

Simply replace the boilerplate TODO between the generic template brackets with a class that extends DbContext and you’re practically good to go. They even provide code snippets via comments that serve as placeholders for configuration changes that you can use to set access rules for the entity sets exposed by your DbContext.

*Note: in my example above, the DataServiceProtocolVersion.V3 is implemented in the October 2011 CTP release for data services. I began using it in order to integrate with the Entity Framework 4.1 DbContext API.

In our case, we will declare the data service this way:

public class PersonDataService : DataService<IDbContext> {…}

Override CreateDataSource

Next is the final piece of the puzzle. The DataService class provides a convenient way to control the instantiation of your service’s data source dependency. Namely - we will override the CreateDataSource method with an implementation such as this:

protected override IDbContext CreateDataSource()
{
    context = compositionContainer.GetExportedValue<IDbContext>();
    return context;
}

Figure 7: CreateDataSource implementation

 

Your implementation may differ, but essentially you want to use your MEF composition container, service locator, or other factory implementation in order to get an instance of your context based on the interface type. MEF has numerous ways for you to resolve this dependency.

 

A critical concern for your code in this area is to handle the multiplicity of dependencies you may find in your container. In a future post, I’ll demonstrate how to specify metadata to filter the results of your composition request.

 

Conclusion

Creating a loosely coupled design can sometimes be a challenge. In the case of WCF Data Services, Microsoft had the foresight to make this chore easier. Through abstraction of your data context, use of an extensibility framework like MEF, service locator or factory implementation, and a little glue code, you can decouple the service and context and reap the rewards of runtime composition and increased testability.

 

Using Architecture to Improve Team Velocity

I’ve been following a number of discussions lately on more than one LinkedIn group that I’m a member of that have been focused on an important concern – namely the role of architecture and design in an Agile project. Many people that I’ve spoken with over the last few years are unclear how much or how little time should be spent on this activity, its relative importance to a team, and ultimately the benefit that it brings to a project. The participants in these recent discussions are asking the same questions. In  this post, I will share my perspective on architecture’s role in an Agile team and how it has benefitted us on our current project. First, I’ll focus on three important things that can impact the answer.

Team Composition Impact on Architecture

A recurring theme amongst many of the discussion participants was team composition. More than once it was mentioned that a team of seasoned software developers could forego up front architectural work and “just start coding”. The architecture would then evolve sprint by sprint. Interestingly, no one discussed the scenario of a team composed of a significant percentage of more junior members.

Project Requirements’ Impact on Architecture

What I found most interesting in these discussions was the absence of emphasis on project requirements and the bearing that they may have on the need for architecture. Perhaps it was an oversight or perhaps it’s because there is little or no difference in need based on project requirements. I’m surprised because in my opinion the more complex a system is, the higher the need for a guiding architecture.

Delivery Timeline’s Impact on Architecture

Without question, nearly every team feels an intense level of pressure to deliver more in a shorter period of time with fewer bugs. It’s the universal facet of what we do – bringing business value to a company in the shortest time and in the most cost effective way. A number of participants in these discussions felt that the demand to drive out a product as quickly as possible made the concept of architecture impossible to consider. This feeling was often supported by arguments that the time spent on architecture would negatively impact the ability of the team to focus on the creation of functioning production code.

What’s the Right Answer?

One thing that I’ve found to be universally true is that no two teams and no two projects are exactly alike. And probably the greatest truth I’ve found in software development is that there is no silver bullet that will solve all of your challenges. What works extremely well on one team or for one project may be completely useless in another setting.

With that in mind, however, in my experience there are some recurring themes and enough similarities that I can say with some confidence that architectural activities in a project have proven to be more beneficial than harmful. Perhaps not to the same degree, but certainly the exercise is not a fruitless endeavor. So what about from the perspectives of the three concerns above?

While team composition can certainly affect how a team perceives the degree of need for architecture, my experience has been that even the most seasoned team can benefit from defining architectural guidance for their project. I like to think of it in terms of an orchestra. The most talented musicians need the notes of the musical score to play to. It’s simply a way for them to all stay in tune with one another. While the action of defining architecture is done by the role of “architect”, the person fulfilling that role may alternate within a team of pros. The net result is a team that operates at a higher level of coordination yielding a higher overall velocity.

The need for architecture on a team with junior members is even more pronounced. Without a clear picture of what these team members need to do, the project can and will degrade into a mire of spaghetti and unmaintainable code. Architectural guidance gives these junior team members development goals and a clearer picture of what they are trying to achieve. It makes clearer the constraints that they need to operate within. This has been invaluable as a communication tool for our team, which has a number of junior team members. It helps to frame conversations and eliminates ambiguity. Rework is down substantially since going with a process that involves an architect role.

The solutions that my company develops are not your average run of the mill line of business applications. They are sophisticated custom solutions that bring predictive analytical models to different vertical markets. They challenge us from many angles including data acquisition, cleansing, modeling, and delivery. Scalability, security, and flexibility all contribute to additional complexity. As a result, delivery of these kinds of products requires management of the technical details and the best way to do that is through architecture and design. However, don’t be fooled into thinking that only sophisticated projects require this level of attention. Even basic LOB applications would benefit from architecture and design in order to reduce duplication, improve reuse, and generally improve maintainability.

Our company is no stranger to tight delivery timelines. Our clients need results quickly to improve their bottom line and our solutions are intended to help with that. In the end, this becomes a value proposition where architecture and design is pitted against time to market. A product that doesn’t meet its objectives because it can’t perform well, can’t interoperate well, or is overly expensive to maintain isn’t a solution – it’s an impediment. Many projects forego the effort of defining architecture and design and instead jump right to the implementation phase. While it can get you over the hump in a hurry to deliver a product, the long-term costs of this can be quite extreme. How many projects have had to be rewritten simply because of decisions like this?

Wrapping Up

This has been a long post and if you made it this far, thanks! In conclusion, I argue in favor of applying sound application architecture and design to the projects I work on because I see their benefit. That benefit comes to teams of all types, projects of varying degrees of complexity, and timelines that are either reasonable or tightly compressed. My advice to you? Take the time to define the architecture of your solution, apply sound design principles to the product, and reap the rewards. Those rewards will be overall improved team velocity and a lower defect and rework rate. These will also have the effect of improved morale on the team because they’ll come away with an improved sense of accomplishment and pride in their workmanship.