Using Papertrail from Docker/Tutum

In my search for a comprehensive logging solution for MuzHack that works in a Dockerized environment, more specifically Tutum, I came across Papertrail.

Papertrail is a fairly simple Web application for log aggregation, that receives log data over the Internet through the syslog protocol. As a registered user of Papertrail, you receive a syslog address that serves as your logging destination. Multiple sources (or "systems" as per Papertrail lingo) can use the same logging destination, as Papertrail will automatically discern between them (based on their hostnames). In my case, this corresponds to my staging and production deployments.

In order to funnel log data from my Docker containers on Tutum, I employ logspout. When running as a Docker container, it simply forwards the stdout and stderr streams of all Docker containers on a host to its configured destination, i.e. Papertrail. In my experience so far, it works pretty well. For each Tutum stack where I’ve installed logspout (i.e. MuzHack production and staging), there’s a corresponding "system" in Papertrail, where I can read the logs of the involved Docker containers. It’s nothing fancy, but it works. I have yet to e.g. configure pre-defined log filters in Papertrail, but the functionality is supposedly there.

Setting up Tutum

Setting up Tutum for logging to Papertrail is surprisingly easy. For me it was just a matter of editing my Tutum stack definitions, for example the production one, in order to include a logspout container:

logspout-production:
  image: 'gliderlabs/logspout:latest'
  command: 'syslog://logs3.papertrailapp.com:19740'
  deployment_strategy: every_node
  restart: always
  tags:
    - production
  volumes:
    - '/var/run/docker.sock:/tmp/docker.sock'

There are three main aspects to note from this stack definition. First, the command directive must tell logspout the address to forward logs to (syslog://logs3.papertrailapp.com:19740). Second, the deployment_strategy must be every node to ensure that containers on every Tutum node get their logs forwarded. Third, the container requires access to the Docker socket (/var/run/docker/sock), which is done by mounting it as one of the volumes.