We got a lot of great entries for the AppHarbor API Contest and we've been busy testing and trying out the submissions. We were excited to see all the stuff you guys built on top of the API and we can't wait to see more. All contestants who submitted an entry get an AppHarbor T-shirt and we'll be mailing those out shortly.

The winners of the main prices are:

1. AppHarbify – instantly deploy open source software to AppHarbor, built by Chris Sainty
2. Lookout – an AppHarbor build watcher by Matt Warren
3. MobileAppHarbor – a mobile optimized HTML client by Wyatt Preul

AppHarbify has been around for a while but we still think it's the best use of the AppHarbor API. AppHarbify uses the API to elegantly provision and deploy open source .NET code to AppHarbor including provisioning add-ons and adding configuration variables required for the project to run. Even better, AppHarbify is template-based and adding support for additional .NET open source projects is simple. The templates are JSON-based and adding support for your favorite project is as easy as forking, adding the project, testing that everything works and issuing a pull request.

Lookout is a desktop build notifier that sits in your system tray and monitors your AppHarbor application builds. We love Lookout because it profiles the cloud continuous integration aspects of AppHarbor and we think that deserves more attention. Remember, AppHarbor builds your code and runs any unit tests and reports both compilation and test errors.

MobileAppHarbor is an impressive mobile-optimized HTML client that uses almost all aspects of the API. We're keen on making appharbor.com work on more devices and MobileAppHarbor does a great job of showing what's possible. We're also happy to see that the API is complete enough to duplicate most aspects of appharbor.com in a client that uses the API.

As mentioned in the contest announcement the winners get to pick between a new iPad, a 27" Apple display and a Xbox in the order they placed in the contest. We're contacting the winners to figure out who gets what.

Please don't stop building stuff on the AppHarbor API, we're excited about the possibilities and we'd love to profile more great API-using apps on the blog. For inspiration, check out some of the great open source code already out there: AppHarbify, AppHarbor.NET and the AppHarbor command line interface.

Friday evening on July 2nd 2012 Amazon Web Services in the US East Region suffered a service disruption that negatively affected applications running on the AppHarbor platform. This post details how AppHarbor was affected and what steps we're taking to mitigate the effects of similar disruptions in the future.

What happened

Each AWS region is composed of multiple availability zones which are separate physical datacenters with independent cooling, networking and power systems. The availability zones are engineered to isolate failures from one another. On Friday, one AWS availability zone suffered a utility power failure followed by multiple emergency generator failures. This caused the availability zone to go offline. Other availability zones in the US East region were not affected. AWS has posted a full rundown of the events of that evening.

The affected availability zone housed most of the infrastructure critical to running the AppHarbor platform and the outage caused AppHarbor and applications on the platform to become unresponsive. AppHarbor and our users were primarily affected for two reasons:

1. EC2 instances and EBS volumes were unavailable and some EBS volumes became corrupted
2. One multi-availability-zone MySQL instance became unavailable for an extended period

EC2 and EBS unavailability

As EC2 instances became unavailable AppHarbor routing infrastructure, application run-time infrastructure and database servers were all affected. AppHarbor staff was alerted immediately and started monitoring the situation. As soon as individual EC2 instances began to come back online we restored the AppHarbor components associated with those instances. Unfortunately, many instances were restored without associated EBS volumes required for correct operation. When volumes did become available they would often take a long time to properly attach to EC2 instances or refuse to attach altogether. Other EBS volumes became available in a corrupted state and had to be checked for errors before they could be used.

Most instances critical to running AppHarbor were fully restored 4-5 hours after they initially became unavailable and appharbor.com and other platform applications began serving requests around this time. A small subset of SQL Server databases were in an inconsistent state when brought online and required manual intervention to become available. Also, some applications had problems accessing their databases because legacy hardcoded connectionstrings were used. We strongly recommend AppHarbor users take advantage of our connectionstring replacement feature to avoid similar problems in the future.

Shared MySQL database unavailability

The AppHarbor shared MySQL add-on runs on top of the AWS RDS service. All MySQL databases run on multi-availability-zone RDS instances which are supposed to be resilient to single-availability-zone outages like the one on June 29th because they can automatically fail over to a standby instance in a different zone in case the primary instance becomes unavailable. As detailed in the AWS report a software bug prevented this fail-over from happening for a small subset of multi-az RDS instances. Some AppHarbor MySQL databases were located on an RDS instance affected by this problem.

Since we assumed the multi-az instances to be OK it took us some time to become aware of the problem. Once the problem came to our attention we tried unsuccessfully to remedy the situation by rebooting the affected instance. When these efforts proved fruitless we finally contacted Amazon support who were able to manually fail over to the instance running in an unaffected availability zone. Amazon is in the process of fixing the bug in their RDS fail-over code.

Action items for AppHarbor

We've identified several actions that we're going to take improve our response to availability-zone failures like the one on June 29th:

1. Run routing and application run-time infrastructure concurrently in multiple availability zones
2. Improve SQL Server recovery processes when encountering corrupted storage volumes
3. Add a status-page to be used for communicating updates in case of platform outages

The best way for us to ensure that applications on AppHarbor are not affected by individual availability-zone outages is to offer our users the option of running applications concurrently in multiple zones. When scaling an application to multiple web workers, we already guarantee that those workers will run on separate EC2 instances. This insulates the app from individual instance failures. We plan to extend this guarantee so that apps with multiple web workers are always running in multiple availability zones. To make this work, we'll also need to make changes to our routing infrastructure to make that resilient or at least permit fast hot-spare fail-overs in case of availability zone failure.

We realize that an application is no good without its database. For that reason, we recommend users opt for multi-az MySQL databases either from the AppHarbor MySQL add-ons or directly from AWS RDS or that applications use other types of replicated data stores from our add-on providers. We're also looking at ways to offer replicated multi-az Microsoft SQL Server databases to go with multi-az applications as outlined above.

Offering hosting of the same application in multiple regions is another interesting prospect we're exploring. We already took some steps towards this when we launched AppHarbor support for EU applications. We're working with the AppHarbor add-on providers to provide even more sophisticated and resilient application hosting scenarios.

The June 29th outage also demonstrated that we need to put in place better procedures for recovering SQL Server databases with corrupted storage volumes. We learnt a lot during the outage and are working on our response procedures.

Lastly, the outage made it clear that we need better ways to communicate AppHarbor service status during major outages. Our support site (provided by Tender) was also affected by the AWS outage and this frustrated our efforts to address individual support requests during the outage. We did publish intermittent updates from the AppHarbor Twitter account but we recognize that a dedicated service status dashboard is required.

If you have additional questions about our response to the outage then feel free to post them in the comments or through the regular support channels.

Today we're talking to Sam and Peter of Appfail about their experiences building their exception logging service and deploying it on AppHarbor.

Why did you decide to start AppFail?

We are professional .NET developers and we have always used tools like ELMAH and Log4Net to find and track problems in our applications. Over time, however, we realised that our approach was ineffective in some areas and really deficient in others. We spent too much time searching and sorting exception emails and exception logs to find and diagnose problems. It was hard to pick patterns – such as seeing that issues occurred on certain days or in certain conditions – and to keep a full historical perspective of our failures and their fixes. We realised that we needed better tools and a better approach. We decided that our solution must help us to:

• expose all of our application’s failures
• find the failures that matter most – those that have the biggest impact;
• track failure progress over time as we deploy fixes;
• show developers and administrators failure information about each page as they browse the live site;
• provide the ability to review and replay failure conditions directly in the page;
• have flexible notifications like ELMAH but with a tighter mobile integration;
• get notified only about important failures as soon as they happen;
• get notified of failures via mobile messaging;
• allow multiple collaborators to work together;
• become a cloud SaaS offering – easily scalable and accessible by anyone with an internet connection;

We also decided to put a lot of care and effort into our UI/UX. We focused on creating a straightforward and unobtrusive user interface with a lot of visual cues, balance and white space. Appfail needed to be a very easy to configure and a cinch to use. It needed to be useful if the developer just wanted a snapshot of failure information and yet provide enough depth to be useful in solving complicated problems. There are other tools on the market which aim to do similar things to Appfail. We looked at them but we weren’t too happy with what we saw. We felt that we could build a more useful product with a better user experience and better functionality for us and for the others out there. And so we made Appfail.

What technologies are you using?

Appfail is built on .NET 4.0 using C#. The main portal – appfail.net – is built on with ASP.NET MVC3. It runs on top of AppHarbor and uses MongoDB (via MongoHQ) for its database. We use Memcacher for sharing state between instances and we use New Relic for diagnosing performance issues. We leverage Amazon Web Services for our queue (SQS) which stores exceptions before they are processed by Appfail and email (SES) for notifications. Furthermore, we employ services such as ViaNett for TXT/SMS notifications. Our code is stored on GitHub and we use Tender to maintain a collection of our KB articles and engage with our users. We use Trello to plan future work, Google Apps, Blogger and Google Analytics.

What advice would you give to other developers running their apps on AppHarbor?

Using AppHarbor enabled us to rapidly develop, test and deliver Appfail. AppHarbor’s integration with GitHub allowed us to push code and immediately play with the changes online. AppHarbor’s add-ons helped us to quickly leverage other tried & tested technologies such as MongoDB. A big draw toward Appharbor was the low cost during development. With Appharbor we had no costs while we prototyped Appfail -- $0 for single instances of our web portal and API endpoint and $0 for a sandboxed MongoDB database. Just a couple of days before going live with beta, we ramped up our instance count, upgraded our MongoHQ plan and added Memcacher for a shared cache.

We’d encourage all passionate .NET developers to try out AppHarbor, play with add-ons and see how it fits them. It has certainly helped to lower the barrier to entry and eased a lot of our pain points. The staff at AppHarbor are impressively responsive (generally responding within a few hours) and have even helped with issues in our own code and suggested fixes.

What's in store for the future?

We have just launched our public beta of Appfail and we have a lot planned. One of our first priorities is to start offering support for other languages and frameworks. We plan to target ruby, python, php and others. We also plan to release a raft of new features, which we have found particularly useful, such as the ability to label, group and classify failures and we are expecting to make a few cool announcements in the mobile space as well. Our other major focus is to foster an Appfail community that help to solve each other’s problems. Wouldn’t it be great if Appfail could tell you about others who have encountered a particular failure, and show you how they solved it? We will extend collaboration options to allow this, and we’ll also link with other sites such as StackOverflow. Our aim is to harness collective knowledge and experience to make our lives as developers easier. This is what Appfail is all about.

Last but not least we are planning on introducing an AppHarbor add-on for our service. Look out for it in the very near future!

Any final thoughts?

We are excited to launch the public beta of Appfail. We have had a great response and a lot of interest so far. The only way we can make it better, however, is for more people to try it (the beta is absolutely free) and to give us their feedback. Please try it and if you have any questions or suggestions please drop us a line at support.appfail.net or [email protected]

As we grow we will continue to leverage AppHarbor as our platform because of the excellent experience we have had. We hope that other developers will give AppHarbor a try and we are thankful for the support and encouragement we have received.

We often help users who want to deploy .NET solutions with multiple output web projects. The conventions for handling this are already covered in the solution file convention documentation, but this blog post will expand on the details. We'll cover how to work with web projects to be run by web workers but the approach outlined is analogous for background worker projects.

How build and deploy works

When AppHarbor builds a solution file, the output of any web projects are placed by MSBuild in a folder in the build output called _PublishedWebsites. Each web project is placed in its own directory.

When the time comes to deploy your application, AppHarbor looks at the contents of _PublishedWebsites. If there's just one folder, we'll deploy that. If there are multiple folders (because your solution had multiple web projects) we will deploy the first one by alphabetical order. Note that the "Set as StartUp project" setting has no effect on the MSBuild process – it's a user-setting stored in the .suo user setting file which should not be committed to source control.

This convention sometimes causes confusion because people have been hacking on a web project in their solution, pushed to AppHarbor and verified that the code was deployed. Visiting the site results in an expected error, however. Often, this turns out to be because AppHarbor has deployed an API or sample web project, not the main site.

Deploying the right site

Imagine you have a solution layout like the one above (Core contains functionality shared between API and Website) and you want to deploy the Website project instead of API, which is what AppHarbor happens to deploy per default. To do this, we'll take advantage of the solution file convention and include an AppHarbor.sln file that references just the Core and Web projects:

1. Create a copy of the solution file
2. Rename the copy to AppHarbor.sln
3. Open AppHarbor.sln in Visual Studio
4. Remove the API project

That's it! Add the new solution file to your repository and push to AppHarbor. When deciding what solution file to build, we'll choose AppHarbor.sln over any other candidates. This approach generally low-maintenance, and you'll only have to worry about the AppHarbor.sln solution if you rename or restructure the projects in your main solution.

Deploying multiple applications

What to do if you want to deploy two applications from the same solution? On AppHarbor, this is accomplished by creating two applications and pushing the same repository to both. Note that if you're using our GitHub integration you can add multiple application slugs to the service hook and if you're using Bitbucket you can add multiple instances of the AppHarbor service, each pointing at different applications.

Take the Web and Api solution from above and say we've created two AppHarbor applications that ended up getting slugs sampleweb and sampleapi (the parts of the urls that identify your apps). To get the Web project deployed to the sampleweb application, simply rename the AppHarbor.sln solution file (the one that referenced the Web and Core projects) to sampleweb.sln. When building code pushed to the sampleweb application, AppHarbor will notice that there are multiple solution files – none of them named AppHarbor.sln – and pick sampleweb.sln to build. To deploy the Api project to the sampleapi application, create a sampleapi.sln solution file referencing the API and Core projects and add that to your repository. This will be the solution file used when building code for sampleapi.

There you have it! Hopefully this post has given you a good idea of how to use AppHarbor's convention-based build process to successfully deploy solutions with multiple web projects.

This post describes how to do OAuth authentication for non-web (eg. desktop) .NET applications. It uses the AppHarbor API as an example, but the approach should be applicable to other OAuth authenticated resources. The approach is inspired by the "Using OAuth 2.0 for Installed Applications" Google documentation.

We only realized the need for a non-web OAuth flow after announcing the AppHarbor API contest where we gamely suggested you guys build various desktop apps using the API, without providing a useful way to authenticate these kinds of apps.

If you're working on your AppHarbor API contest entry and not interested in all the details, just grab the AppHarbor SDK from GitHub and have a look at the console application sample. We've folded the flow into the SDK to make authorizing desktop apps as simple as possible.

The crux of the approach outlined below is that the user never has to enter his username and password into the 3rd party client app. Username/password authorization only happens on AppHarbor, and the client app then authorizes future API calls with the token obtained using OAuth. The token can be reset by the user later on if he wants to revoke access for the application. The fact that the user never types his username and password into the application prevents skullduggery of the sort where the client app stores the users credentials forcing the user to change his password to revoke access.

Overview

1. Application decides it needs to authenticate user
2. Application starts a web server listening on a known localhost url using System.Net.HttpListener
3. App fires up user's favourite browser and opens the AppHarbor OAuth authorization page with a localhost redirect url
4. User signs in to AppHarbor (if not already logged in) in the browser window
5. User authorizes the application on AppHarbor
6. User is redirected to the localhost url
7. The localhost web server gets the request and the application sniffs the OAuth authorization code from the request url
8. The application can now trade the authorization code for a token and start issuing API requests

Check out the AppHarborClient.Auth.cs for all the gory details of how this works.

Since introducing the GitHub integration thousands of AppHarbor users have configured their applications to automatically deploy code from GitHub repositories. Setting up the service hook to get AppHarbor to deploy from GitHub was never very intuitive though: It required navigating to the service hook section on GitHub and copy-pasting values retrieved from the AppHarbor application build-url.

Improvements

We've now greatly improved on that workflow by leveraging the AppHarbor and GitHub API's to automate the process. Now, when you create a new application and click the "Configure GitHub to deploy to AppHarbor" link, we'll redirect to a utility-app that will log you in to GitHub and let you select which of your GitHub repositories to add the AppHarbor service hook to. We'll also add the apphb user as a collaborator on GitHub to let us pull the code if it's private.

The utility-bridge app accomplishes it's magic using NGitHub and the AppHarbor SDK. Once we've cleaned up the code, we expect to open source it for use as sample code to illustrate how to use these API's.

Remember that you can add multiple application slugs to the GitHub service (the token is user specific and shared between applications) . The automatic setup doesn't currently support multiple slugs, it'll wipe existing slugs if the service is already configured for a GitHub repository, but we're working on improvements.

Flow

The new flow is illustrated below:

1. Create a new AppHarbor application
2. Click the "Configure GitHub to deploy to AppHarbor" link
3. You'll be redirected to the utility app and required to log onto GitHub if you're not already signed in
4. Once redirected back to the utility app, select the repository you want to deploy from
5. You'll be redirected back to AppHarbor and should see the application being built

We've just added SearchBox to the AppHarbor add-on program. SearchBox is a fully hosted and managed ElasticSearch full-text search service. ElasticSearch is incredibly powerful and fast full-text search software built on Apache Lucene.

SearchBox is still in beta, so only a free plan is currently available. The free offering is fully functional though, and you can experiment with all ElasticSearch features.

Using SearchBox in your AppHarbor-hosted application is incredibly simple, just add the add-on and follow the instructions for .NET apps in the SearchBox docs. The preferred .NET client for ElasticSearch is called NEST (available on NuGet) and the NEST site has additional documentation. The SearchBox guys have also built a .NET sample app that stores documents in SQL Server and uses SearchBox for full-text search. To experiment with the app, just grab it from GitHub and push it to an AppHarbor application with the SQL Server and SearchBox add-ons provisioned, and you're ready to go.

Don't miss out on full-text search, faceting, highlighting and other advanced ElasticSearch features, provision SearchBox right now.

A few months ago we released the AppHarbor API, giving you control over your account via a simple HTTP and JSON based interface. The API allows you to create and edit applications, manage configuration variables and hostnames for those applications and retrieve your build and test status along with application error logs. Learn more about the available resources in the API Documentation.

We’re excited by the possibilities offered by the API and today we're launching our first API contest to tap into the collective creativity of our users and reward you for the awesome things you build. Read on below for details on prices and ideas.

The Prizes

AppHarbor staff will judge the entries and pick a first, second and third prize based on the combination of usefulness, originality, completeness and overall awesomeness of the entry.

In addition to fame and community love won by successful entries, we have a great set of prizes lined up:

• Apple 27" Thunderbolt Cinema Display
• The new iPad, 32GB, WiFi
• Xbox 250GB with Kinect and 12-month Xbox Live Membership

First prize gets first pick from the prize pool, second prize gets next pick and third prize gets the remaining item. Everyone who enters will receive an AppHarbor t-shirt.

Some Ideas to Get You Started

Entries for the contest can use any part of the HTTP API, service hooks or build URLs. That gives you a lot of areas to explore, but here are some ideas we drummed up for things we'd love to see you build:

• Desktop notifications for builds, errors, etc.
• Email/SMS/push notifications
• Browser extensions
• Acceptance testing and auto-rollback
• Visual Studio add-ins or integration
• Mobile account control
• Application health dashboard or analytics
• WebDeploy bridge using a build URL

You could also let yourself be inspired by AppHarbify which we think is a great use the AppHarbor platform and the API. More about AppHarbify here.

You don't have to build something on this list, surprise us with your creativity!

Save Time with AppHarbor.NET

Nikolas Tziolis has created AppHarbor.NET, a wrapper library for the AppHarbor API. We’ve since adopted the project and are keeping it up to date. You can use AppHarbor.NET to quickly get started on your entry. To install it into your project, add the AppHarbor.NET NuGet package which will pull in the required dependencies as well.

In addition to the REST API wrapper, Nikolas has put together a sample ASP.NET MVC application demonstrating how to obtain user authorization with OAuth.

The source code for AppHarbor.NET is available on GitHub and we accept contributions and feedback.

If you want to see more sample code that uses AppHarbor.NET and the API, then check out the AppHarbor command line interface source which is open source and on GitHub

The Fine Print

This contest is open to anyone and everyone, except those living in places where contests like this are illegal. If you live somewhere where it is infeasible to send you a prize, we'll work something out with you.

Entries are due Tuesday July 3rd at Noon Pacific Time (in two weeks). Submit your entry here.

Image credits: andrew-gw, Evan Amos and Zach Vega.

Today, we're excited to welcome MemCachier into the add-on catalog. MemCachier is a hosted Memcached service.

You should add MemCachier to your AppHarbor-hosted app to cache frequently used data and speed up your application. We've written a short guide on how to get started with Memcachier. We're working with the MemCachier guys to support easy configuration of their add-on for more advanced use cases such as using Memcached as a session state backing store.

A few weeks ago, we published a blog post on how to configure automatic migrations with Entity Framework. Here's the opening quote from that article on why you should care about automatic migrations (read the rest of the article here):

Having your application automatically update its database schema greatly eases development and makes frequent deployments to platforms like AppHarbor a cinch.

The main AppHarbor codebase actually uses NHibernate and Fluent NHibernate augmented with EventStore CQRS. Our NHibernate setup is configured to automatically update the database schema as we push updates to our model classes. To do this, we use a little-published feature of NHibernate called SchemaUpdate. In conjunction with Fluent NHibernate, this can be used to create SessionFactories in the following way (this example is for PostgreSQL):

protected ISessionFactory CreateSessionFactory()
{
    var connectionString = "connectionString";

    var autoMap = AutoMap.AssemblyOf<Entity>()
        .Where(t => typeof(Entity).IsAssignableFrom(t));

    return Fluently.Configure()
        .Database(
            PostgreSQLConfiguration.Standard.ConnectionString(connectionString))
        .Mappings(m => m.AutoMappings.Add(autoMap))
        .ExposeConfiguration(TreatConfiguration)
        .BuildSessionFactory();
}

protected virtual void TreatConfiguration(NHConfig.Configuration configuration)
{
    var update = new SchemaUpdate(configuration);
    update.Execute(false, true);
}

Also note that we us AutoMap to create schema only for all classes that inherit from Entity. We can use the code above in conjunction with an IoC container like StructureMap and ASP.NET like this:

ObjectFactory.Initialize(x =>
{
    x.For<ISessionFactory>()
        .Singleton()
        .Use(CreateSessionFactory());

    x.For<ISession>()
        .HttpContextScoped()
        .Use(context => context.GetInstance<ISessionFactory>().OpenSession());
});

With this configuration, whenever a new ISessionFactory is created, the running schema is compared with schema dictated by the model classes (here all classes inherited from Entity) and the running schema is updated as needed. Our SessionFactory is a singleton, so this process will only happen when the application is restarted.

We have been running with a setup like this on AppHarbor since the very first model class was created. Once in a while we have to plan manual schema- or data-migrations, but for for the most part this simple setup quietly syncs our database with our model every time new we push new versions of AppHarbor.

If you want to see how this works in the context of an ASP.NET MVC application then check out this sample we've published for using some of the PostgreSQL-compatible add-ons we offer. It uses the above setup in a simple ASP.NET MVC 3 application that will automatically configure its database the first time it's deployed and keep it updated as the application model is expanded.