ASP.NET Forms Authentication Considered Broken

UPDATE: We have published a replacement for ASP.NET forms authentication.

Forms Authentication is the default way for ASP.NET applications (both MVC and Web Forms) to keep track of logged in users. Unfortunately forms auth is subtly broken in a way that can cause problems for applications running on multiple machines in load-balanced setups and web farms such as the one found on AppHarbor's cloud platform.

Forms authentication in an ASP.NET application works by setting encrypted cookies in users browsers on log in (forms authentication also has a uri-mode for clients that don't support cookies). When a user returns to the site, ASP.NET will identify the user by decrypting the cookie passed by the users browser.

How the cookie is encrypted and made tamper-proof is configured in the machineKey element in the application web.config. The standard ASP.NET behaviour is to AutoGenerate the keys. This setting causes new keys to be generated every time the application pool is started. If a user shows up on the site with a cookie created with old keys, the cookie is not recognized and the user will have to log in again. This is not desirable, especially since IIS may decide to stop an inactive application to conserve resources and then restart it when a visitor shows up.

Auto-generating machine keys also won't work if an application is running on multiple servers. This is because cookies issued by one server, using the keys that server has auto-generated for the application, will not be recognized by another server that has generated a different set of keys.

AppHarbor overcomes these limitations in the following way: When the AppHarbor platform deploys an application, the web.config machineKey element is examined. If the machineKeys are set to auto-generate, that setting is overriden and machineKeys unique to that application are specified. If the machineKeys are already set to a fixed value, they are left alone. This means that sessions in applications hosted on AppHarbor survive application pool restarts. Also, because all web workers of an application that is scaled to multiple workers use the same machineKeys, cookies issued by one worker are accepted by all other workers running that application.

Here's an example machineKey element inserted by AppHarbor.

<machineKey validation="SHA1"
    decryptionKey="someDecryptionKey"
    validationKey="someValidationKey" />

So where's the breakage? As long as all application workers are using the same set of machineKeys, everything should be peachy, right? Well, not quite. It turns out that Microsoft has a tendency to change the way forms authentication cookies are generated as patches to ASP.NET are issued. Changing the way cookies are encrypted might make sense for major version updates or if security problems are found, but changes seem to happen more often than that. It's especially strange since the encryption algorithm to be used (SHA1, AES, etc.) is specified in the machineKey element, suggesting that something else is changing with the patches (i.e. a salt).

While we can't always nail down what patches cause cookie-incompatibilities it definitely happened (and was announced) for the late December 2011 security update. We have also detected problems both before and after that event, however.

Semi-frequent updates of the cookie-algorithm makes sysadmins lives (including ours) more difficult. Instead of doing rolling patch updates across platform servers — i.e. move all application worker off a server, patch and reboot it, move application workers back — we have to crank up an entirely new set of fully patched servers and move all workers running an application in one go. Solving these kinds of problems for developers is why we started AppHarbor, of course, but some problems we can't solve. There's no way to prevent users logged in using forms auth from having to log back in after an update like this, for example.

The best way to get around this problem is to not use forms authentication at all: If you control how authentication cookies are generated, then no Microsoft patch is going to come along and mess things up. Indeed, this is how AppHarbor.com works and we hope to soon provide a sample of how you can do the same in your application.

To debug forms authentication problems we have used this simple application which lets you log in using forms authentication and determine cookie status on the various servers that run the application.

As always, we're committed to making AppHarbor the best platform for deploying and scaling your .NET applications. This post described some of the steps we take to do just that and provided guidance on ways you can adapt you application to ensure maximum reliability and continuity for your users. If you have questions on how to best scale your application on AppHarbor, then feel free to get in touch.