Zero Downtime Docker Deployment with Tutum

In this post I will detail how to achieve (almost) zero downtime deployment of Docker containers with the Tutum Docker hosting service. Tutum is a service that really simplifies deploying with Docker, and it even has special facilities for enabling zero downtime deployment, i.e. the Tutum team has a version of HAProxy that can switch seamlessly between Docker containers in tandem with Tutum’s API. There is the caveat though, that at the time of writing, there is slight downtime involved with HAProxy’s switching of containers, typically a few seconds. I have been assured though, that this will be improved upon before Tutum goes into General Availability.

Basically, the officially sanctioned approach to implement zero downtime deployment on top of Tutum is so-called Blue-Green Deployment as detailed in this blogpost at Tutum. The concept underpinning blue-green deployment is pretty simple, the idea is to have a “blue” and a “green” version of your service, which are both behind a reverse proxy, which exposes exactly one at a time. When upgrading to a new version of your service, you deploy it to the container that is currently hidden, and make the reverse proxy switch to it. Then the new version will be publicly visible and the old version hidden.

Stack Definition

So how does one do this in practice? Basically you define a Tutum stack, with two app services, e.g. blue and green, and an HAProxy service to reside in front of the former two. The app services should have different deployment tags, so they deploy to different nodes. The HAProxy service needs to be linked to one of the two app services, i.e. the one that should initially be exposed. Later on, that link will be updated dynamically.

This is my current stack definition, wherein HAProxy is configured to enable HTTPS and redirect HTTP requests to HTTPS:

muzhack-green:
  image: quay.io/aknuds1/muzhack:omniscient
  tags:
    - muzhack-green
  environment:
    - FORCE_SSL=yes
  restart: always
  deployment_strategy: high_availability
muzhack-blue:
  image: quay.io/aknuds1/muzhack:omniscient
  tags:
    - muzhack-blue
  environment:
    - FORCE_SSL=yes
  restart: always
  deployment_strategy: high_availability
lb:
  image: tutum/haproxy
  tags:
    - muzhack-lb
  environment:
    - EXTRA_BIND_SETTINGS=redirect scheme https code 301 if !{ ssl_fc }
    - DEFAULT_SSL_CERT
    - HEALTH_CHECK=check inter 10s fall 1 rise 2
    - MODE=http
    - OPTION=redispatch, httplog, dontlognull, forwardfor
    - TCP_PORTS=80,443
    - TIMEOUT=connect 10s, client 1020s, server 1020s
    - VIRTUAL_HOST=https://*
  restart: always
  ports:
    - "443:443"
    - "80:80"
  links:
    - muzhack-green
  roles:
    - global

I’m not 100% certain yet about the HAProxy settings, which I’ve based off others’ advice. It appears to work however. The DEFAULT_SSL_CERT environment variable should contain the contents of an SSL .pem file.

Making the Switch

Implementing the deployment process itself caused me some bother initially, since it involves f.ex. knowing what is the currently active service (blue or green). I found I was able to script everything though, thankfully, through Tutum’s comprehensive API. In the end I wrote a Python script to deploy a new version to the currently inactive service, and make HAProxy switch to it:

#!/usr/bin/env python3
import argparse
import subprocess
import json
import sys


parser = argparse.ArgumentParser()
args = parser.parse_args()


def _info(msg):
    sys.stdout.write('* {}\n'.format(msg))
    sys.stdout.flush()


def _run_tutum(args):
    try:
        subprocess.check_call(['tutum',] + args, stdout=subprocess.PIPE)
    except subprocess.CalledProcessError as err:
        sys.stderr.write('{}\n'.format(err))
        sys.exit(1)


_info('Determining current production details...')
output = subprocess.check_output(['tutum', 'service', 'inspect', 'lb.muzhack-staging']).decode(
    'utf-8')
data = json.loads(output)
linked_service = data['linked_to_service'][0]['name']
_info('Currently linked service is \'{}\''.format(linked_service))

if linked_service == 'muzhack-green':
    link_to = 'muzhack-blue'
else:
    assert linked_service == 'muzhack-blue'
    link_to = 'muzhack-green'

_info('Redeploying service \'{}\'...'.format(link_to))
_run_tutum(['service', 'redeploy', '--sync', link_to,])

_info('Linking to service \'{}\'...'.format(link_to))
_run_tutum(['service', 'set', '--link-service', '{0}:{0}'.format(link_to),
    '--sync', 'lb.muzhack-staging',])
_info('Successfully switched production service to {}'.format(link_to))

This script does the following:

  1. Find out which service is currently active (green or blue)
  2. Redeploy inactive service so that it updates to the newest version
  3. Link HAProxy service to inactive service, thus making it the active one

Help Functionality Added to MuzHack’s Markdown editors

Markdown Help

The Markdown editors in MuzHack are now sporting a help button, to display documentation about the supported Markdown syntax. Early user feedback suggested this was a much desired feature, as many users aren’t familiar with the Markdown syntax.

Markdown Help

Markdown Help

MuzHack Presentation at Notam

So I gave the first presentation ever on MuzHack today, at the Norwegian sound research centre Notam. Feedback was positive and I got lots of great leads on who I should connect with further, and elsewhere I might present, both nationally and internationally.

Another presentation elsewhere in Norway might be in the works, later this year.

Using Papertrail from Docker/Tutum

In my search for a comprehensive logging solution for MuzHack that works in a Dockerized environment, more specifically Tutum, I came across Papertrail.

Papertrail is a fairly simple Web application for log aggregation, that receives log data over the Internet through the syslog protocol. As a registered user of Papertrail, you receive a syslog address that serves as your logging destination. Multiple sources (or “systems” as per Papertrail lingo) can use the same logging destination, as Papertrail will automatically discern between them (based on their hostnames). In my case, this corresponds to my staging and production deployments.

In order to funnel log data from my Docker containers on Tutum, I employ logspout. When running as a Docker container, it simply forwards the stdout and stderr streams of all Docker containers on a host to its configured destination, i.e. Papertrail. In my experience so far, it works pretty well. For each Tutum stack where I’ve installed logspout (i.e. MuzHack production and staging), there’s a corresponding “system” in Papertrail, where I can read the logs of the involved Docker containers. It’s nothing fancy, but it works. I have yet to e.g. configure pre-defined log filters in Papertrail, but the functionality is supposedly there.

Setting up Tutum

Setting up Tutum for logging to Papertrail is surprisingly easy. For me it was just a matter of editing my Tutum stack definitions, for example the production one, in order to include a logspout container:

logspout-production:
  image: 'gliderlabs/logspout:latest'
  command: 'syslog://logs3.papertrailapp.com:19740'
  deployment_strategy: every_node
  restart: always
  tags:
    - production
  volumes:
    - '/var/run/docker.sock:/tmp/docker.sock'

There are three main aspects to note from this stack definition. First, the command directive must tell logspout the address to forward logs to (syslog://logs3.papertrailapp.com:19740). Second, the deployment_strategy must be every node to ensure that containers on every Tutum node get their logs forwarded. Third, the container requires access to the Docker socket (/var/run/docker/sock), which is done by mounting it as one of the volumes.

MuzHack Has a Forum

I have just started a forum for MuzHack, based on the amazing Discourse. Stay tuned for useful content.