Introducing Realtime Logging

Today we're incredibly excited to announce the public beta of our brand new logging module. Starting immediately all new applications created on AppHarbor will have logging enabled. You can enable it for your existing apps on the new "Logging" page.

We know all too well that running applications on a PaaS like AppHarbor sometimes can feel like a black box. So far we haven't had a unified, simple and efficient way to collate, present and distribute log events from the platform and your apps.

That's exactly what we wanted to address with our logging solution, and based on the amazing feedback from private beta users we feel confident that you'll find it useful for getting insight about your application and AppHarbor. A big thanks to all the beta testers who have helped us refine and test these new features.

The new logging module collates log messages from multiple sources, including almost all AppHarbor infrastructure component and your applications - API changes, load balancer request logs, build, deploy and stdout/stderr from your background workers and more can now be accessed and sent to external services in real time.

Consider yourself lucky we're not that much into skeuomorphism

Interfaces

We're providing two interfaces "out of the box" - a convenient web-interface can be accessed on the Logging page and a new log command has been added to the CLI. [Get the installer directly from here or install with Chocolatey cinst appharborcli.install. To start a "tailing" log session with the CLI, you can for instance run appharbor log -t -s appharbor. Type appharbor log -h to see all options.

The web interface works a bit differently, but try it out and let us know what you think - it's heavily inspired by the log.io project who have built a great client side interface for viewing, filtering, searching and splitting logs into multiple "screens".

Integration

One of the most useful and interesting aspects of today's release is the flexible integration points it provides. Providing access to your logs in realtime is one thing, but AppHarbor will only store the last 1500 log messages for your application. Storing, searching, viewing and indexing logs can be fairly complex and luckily many services already exists that helps you make more sense of your log data.

We've worked with Logentries to provide a completely automated and convenient way for sending AppHarbor logs to them when you add their add-on. When you add the Logentries add-on your application can automatically be configured to send logs to Logentries, and Logentries will be configured to display log messages in AppHarbor's format.

You can also configure any syslog (TCP), HTTP and HTTPS endpoint you like with log "drains". You can use this to integrate with services like Loggly and Splunk, or even your own syslog server or HTTP service. More details about log drains are available in the this knowledge base article and the drain API documentation.

Finally there's a new new Log session API endpoint that you can use to create sessions similar to the ones used by the interfaces we provide.

Logplex

If you've ever used Heroku you'll find most of these features very familiar. That's no coincidence - the backend is based on Heroku's awesome distributed syslog router, Logplex. Integrating with Logplex makes it a lot easier for add-on providers who already support Heroku's Logplex to integrate with AppHarbor, while giving us a scalable and proven logging backend to support thousands of deployed apps.

Logplex is also in rapid, active development, and a big shout-out to the awesome people at Heroku who are building this incredibly elegant solution. If you're interested in learning more about Logplex we encourage you to check out the project on Github and try it for yourself. We've built a client library for interacting with Logplex's HTTP API and HTTP log endpoints from .NET apps - let us know if you'd like to use this and we'll be happy to open source the code. The Logplex documentation on stream management is also useful for a high-level overview of how Logplex works.

Next steps

With this release we've greatly improved the logging experience for our customers. We're releasing this public beta since we know it'll be useful to many of you as it is, but we're by no means finished. We want to add even more log sources, provide more information from the various infrastructure components and integrate with more add-on providers. Also note that request logs are currently only available on shared load balancers, but it will be rolled out to all load balancers soon. If you find yourself wanting some log data that is not currently available please let us know. We now have a solid foundation to provide you with the information you need when you need it, and we couldn't be more excited about that.

We'll provide you with some examples and more documentation for these new features over the next couple of weeks, but for now we hope you'll take it for a spin and test the waters for yourself. Have fun!