Official Elasticsearch/Fluentd/Kibana Add-On for Kubernetes updated to 5.5

My pull request for updating the official Elasticsearch/Fluentd/Kibana logging add-on for Kubernetes to version 5.5.1 of Elasticsearch and Kibana was recently approved and merged into the master branch! Users of the popular EFK/ELK stack can now enjoy the latest version with their Kubernetes clusters!

Installing Elasticsearch/Kibana 5.5 within Kubernetes cluster on AWS

Installing Elasticsearch/Kibana 5.5 within Kubernetes cluster on AWS

In my previous blog post I showed how to use the Kops tool to create a production ready Kubernetes cluster on Amazon Web Services (AWS). In this follow-up post I will show how to install Elasticsearch and its graphical counterpart Kibana in the cluster, in order to be able to collect and store logs from your cluster and search/read them. We will also install Fluentd as this component is responsible for transmitting the standard Kubernetes logs to Elasticsearch. This is generally known as the ELK stack, which stands for Elasticsearch, Logstash (precursor to Fluentd) and Kibana.

First of all, it should be mentioned that there is a standard addon for installing ELK in Kubernetes clusters, as part of the Kubernetes repository. The solution I’m going to present in this blog post is derived from that addon, but I’ve rewritten it in order to upgrade to Elasticsearch and Kibana 5.5.0 as the addon at the time of writing is based on version 5.4.0, as well as to include the repository-s3 Elasticsearch plugin. We need the latter plugin in order to be able to back up Elasticsearch to S3 storage in AWS. I have made a pull request in order to try to consolidate my modified version of the ELK Kubernetes addon with the original.

I will go through each part of the stack in succession and explain shortly how it functions. In order to install the logging stack in your Kubernetes cluster, apply the manifests via kubectl: kubectl apply -f *.yaml.

Elasticsearch

These manifests install Elasticsearch itself as a StatefulSet of two pods that will allocate an AWS EBS volume of 20 GB per pod (the size was chosen somewhat at random, pick a size that make sense for your workload). In order to configure Elasticsearch’s repository-s3 plugin, environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_KEY also have to be set, via a Kubernetes secret called bootstrap-environment. A Service is created in front of the StatefulSet pods to load balance the StatefulSet pods.

Elasticsearch is also configured to run under the service account elasticsearch-logging, which gets bound to the role elasticsearch-logging in order for it to have the right permissions.

We use our own custom Elasticsearch image that inherits from the official one and which includes a binary from the original Kubernetes addon in order to discover necessary Elasticsearch parameters at runtime, to make it integrate with the Kubernetes cluster.

es-clusterrole.yaml:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: elasticsearch-logging
  labels:
    k8s-app: elasticsearch-logging
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
  - ""
  resources:
  - "services"
  - "namespaces"
  - "endpoints"
  verbs:
  - "get"

es-serviceaccount.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: elasticsearch-logging
  namespace: kube-system
  labels:
    k8s-app: elasticsearch-logging
    version: v1
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile

es-clusterrolebinding.yaml:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  namespace: kube-system
  name: elasticsearch-logging
  labels:
    k8s-app: elasticsearch-logging
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
subjects:
- kind: ServiceAccount
  name: elasticsearch-logging
  namespace: kube-system
  apiGroup: ""
roleRef:
  kind: ClusterRole
  name: elasticsearch-logging
  apiGroup: ""

es-statefulset.yaml:


apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: elasticsearch-logging-v1
  namespace: kube-system
  labels:
    k8s-app: elasticsearch-logging
    version: v1
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  serviceName: elasticsearch-logging
  replicas: 2
  selector:
    matchLabels:
      k8s-app: elasticsearch-logging
      version: v1
  template:
    metadata:
      labels:
        k8s-app: elasticsearch-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccountName: elasticsearch-logging
      containers:
      - image: aknudsen/elasticsearch-k8s-s3:5.5.0-14
        name: elasticsearch-logging
        resources:
          # need more cpu upon initialization, therefore burstable class
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        ports:
        - containerPort: 9200
          name: db
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        volumeMounts:
        - name: elasticsearch-logging
          mountPath: /data
        env:
        - name: "NAMESPACE"
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: bootstrap-environment
              key: es-s3-access-key
        - name: AWS_SECRET_KEY
          valueFrom:
            secretKeyRef:
              name: bootstrap-environment
              key: es-s3-secret-key
  volumeClaimTemplates:
  - metadata:
      name: elasticsearch-logging
    spec:
      storageClassName: gp2
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 20Gi

es-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-logging
  namespace: kube-system
  labels:
    k8s-app: elasticsearch-logging
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "Elasticsearch"
spec:
  ports:
  - port: 9200
    protocol: TCP
    targetPort: db
  selector:
    k8s-app: elasticsearch-logging

env-secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-environment
  namespace: kube-system
type: Opaque
data:
  es-s3-access-key: example
  es-s3-secret-key: example

Fluentd

Fluentd is installed as a DaemonSet, which means that a corresponding pod will run on every Kubernetes worker machine in order to collect its logs (and send them to Elasticsearch). Furthermore, the pods run as the service account fluentd-es which is bound to the cluster role fluentd-es in order to have the necessary permissions.

Our Fluentd DaemonSet is configured to use my custom Docker image for Fluentd as it configures Fluentd to use environment variables FLUENT_ELASTICSEARCH_USER and FLUENT_ELASTICSEARCH_PASSWORD as credentials towards Elasticsearch.

fluentd-es-clusterrole.yaml

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: fluentd-es
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
  - ""
  resources:
  - "namespaces"
  - "pods"
  verbs:
  - "get"
  - "watch"
  - "list"

fluentd-es-serviceaccount.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd-es
  namespace: kube-system
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile

fluentd-es-clusterrolebinding.yaml:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: fluentd-es
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
subjects:
- kind: ServiceAccount
  name: fluentd-es
  namespace: kube-system
  apiGroup: ""
roleRef:
  kind: ClusterRole
  name: fluentd-es
  apiGroup: ""

fluentd-es-ds.yaml

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd-es
  namespace: kube-system
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    version: v1.23
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-es
        kubernetes.io/cluster-service: "true"
        version: v1.23
      # This annotation ensures that fluentd does not get evicted if the node
      # supports critical pod annotation based priority scheme.
      # Note that this does not guarantee admission on the nodes (#40573).
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      serviceAccountName: fluentd-es
      containers:
      - name: fluentd-es
        image: aknudsen/fluentd-k8s-es:1.23
        command:
          - '/bin/sh'
          - '-c'
          - '/usr/sbin/td-agent $FLUENTD_ARGS'
        env:
        - name: FLUENTD_AGRS
          value: -q
        # These are default ElasticSearch credentials
        - name: FLUENT_ELASTICSEARCH_USER
          value: "elastic"
        - name: FLUENT_ELASTICSEARCH_PASSWORD
          value: "changeme"
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      nodeSelector:
        beta.kubernetes.io/fluentd-ds-ready: "true"
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

Kibana

Unlike the original Kubernetes addon, we don’t use a custom image for Kibana, just the official one as that works out of the box. There’s not much to say about our Kibana manifests; we install a Deployment, which ensures that one pod is always running, and a Service in front of it (which is capable of load balancing in case there should be several pods in parallel).

kibana-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kibana-logging
  namespace: kube-system
  labels:
    k8s-app: kibana-logging
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kibana-logging
  template:
    metadata:
      labels:
        k8s-app: kibana-logging
    spec:
      containers:
      - name: kibana-logging
        image: docker.elastic.co/kibana/kibana:5.5.0
        resources:
          # keep request = limit to keep this container in guaranteed class
          limits:
            cpu: 100m
          requests:
            cpu: 100m
        env:
          - name: "ELASTICSEARCH_URL"
            value: "http://elasticsearch-logging:9200"
        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP

kibana-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: kibana-logging
  namespace: kube-system
  labels:
    k8s-app: kibana-logging
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "Kibana"
spec:
  ports:
  - port: 5601
    protocol: TCP
    targetPort: ui
  selector:
    k8s-app: kibana-logging

Creating a Highly Available Secured Kubernetes Cluster on AWS with Kops

Creating a Highly Available Secured Kubernetes Cluster on AWS with Kops

Today I will be talking about Kops, which is an official tool for creating Kubernetes clusters on AWS, with support for GCE and VMware vSphere in alpha. It takes a whole lot of the pain out of setting up a Kubernetes cluster yourself, but still presents many challenges to overcome and a great degree of freedom in how you configure the cluster.

I have recently created a Kubernetes cluster on AWS for a client, where I used the Kops tool for the very first time and I will here present what I learnt about implementing best practices with this technology stack. Given the rapid development of Kubernetes itself, and how relatively young Kops is, it proved to be far from a walk in the park to create a production-grade cluster. Documentation is often relatively poor, or just plain missing. The intention of this blog post is to make it easier for others going down the same route.

What we set out to do on this project, is to produce a highly available Kubernetes cluster (on AWS) which you can only SSH into via a dedicated host (a so-called bastion), and that secures the control plane/API via client certificate based authentication and RBAC authorization.

Creating the Cluster Itself

The very first step, which is thankfully quite simple thanks to the awesome power of Kops, is to bring the cluster up. The below kops invocation creates a highly available cluster on AWS, with 5 worker nodes spread among three availability zones and 3 master nodes in the same AZs. For security, all master/worker nodes are in a private subnet and not exposed to the Internet. We also instantiate a bastion host as the sole entry point into the cluster via SSH, and the cluster is configured to enable RBAC as its authorization mode. I chose Flannel as the networking system as I have some experience with it from prior work with Kubernetes and have a good impression of it.

kops --state s3://example.com create cluster --zones \
eu-central-1a,eu-central-1b,eu-central-1c --master-zones \
eu-central-1a,eu-central-1b,eu-central-1c --topology private --networking flannel \
--master-size m4.large --node-size m4.large --node-count 5 \
--bastion --cloud aws --ssh-public-key id_rsa.pub --authorization RBAC --yes \
example.com

On the topic of high availability, since the cluster has three master nodes within three different availability zones, the cluster is protected from the outage of individual AZs and is still available if a master goes down. The same goes for worker nodes.

Exporting the Kubectl Configuration

After creating the cluster, we would like to generate a configuration file to use in order to have kubectl operate against our cluster. We do this with the following command:

KUBECONFIG=$CLUSTER.kubeconfig kops export kubecfg $CLUSTER

Configuring Cluster Components for RBAC

In order for certain cluster components to function with RBAC enabled, some configuration is required. Basically, what we need to do is to bind the right roles to service accounts to allow the latter to perform certain tasks on behalf of pods.

Configure the cluster when it’s ready by applying the configuration files in the sections below:

kubectl --kubeconfig=$CLUSTER.kubeconfig apply -f kube-system-rbac.yaml
kubectl --kubeconfig=$CLUSTER.kubeconfig apply -f kube-flannel-rbac.yaml

Default System Service Account

The default service account in the kube-system namespace must be bound to the cluster-admin role:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: system:default-sa
subjects:
  - kind: ServiceAccount
    name: default
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

Flannel Service Account

The flannel service account in the kube-system namespace must be given a role with certain permissions in order to enable the Flannel networking component to do its job:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
rules:
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes/status
    verbs:
      - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system

Client Certificate Based Authentication

I decided to implement user authentication via TLS certificates, as this is directly supported in the kubectl tool and ties easily in with RBAC authorization. The trick here is to get hold of the certificate authority (CA) certificate and key that Kops used when creating the cluster, as it will allow us to generate valid user certificates. Luckily, these files are stored in Kops’ S3 bucket. The following commands copies the CA key and certificate to the local directory:

aws s3 cp s3://$BUCKET/$CLUSTER/pki/private/ca/$KEY ca.key
aws s3 cp s3://$BUCKET/$CLUSTER/pki/issued/ca/$CERT ca.crt

Now that we have the CA key and certificate, we can generate a user certificate with the openssl command line tool. The procedure consists of first generating a private key, then with the previously generated key generating a signing request for a certificate representing a user with username $USERNAME and finally signing the certificate with the help of the CA key and certificate. The below commands will produce a key and certificate named user.key and user.crt, respectively:

openssl genrsa -out user.key 4096
openssl req -new -key user.key -out user.csr -subj '/CN=$USERNAME/O=developer'
openssl x509 -req -in user.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out user.crt \
-days 365

Granting Cluster Administrator Rights to User

We would like for our user, as represented by the certificate, to have cluster administrator rights, meaning that they are basically permitted any operation on the cluster. The way to do this is to create a ClusterRoleBinding that gives the cluster-admin role to the new user, as in the following command:

kubectl --kubeconfig $CLUSTER.kubeconfig create clusterrolebinding \
$USERNAME-cluster-admin-binding --clusterrole=cluster-admin --user=$USERNAME

After granting your user this role, we can start using it instead of the default admin user, as you’ll see in the next section.

Identifying User through Certificate via Kubectl

Given the certificate we created for the user previously, and having assigned the user the cluster-admin role, we can now identify towards Kubernetes by modifying the kubectl configuration. We configure kubectl to authenticate with the certificate towards the cluster via the following commands:

kubectl --kubeconfig=$CLUSTER.kubeconfig config set-credentials $USERNAME \
--client-key=user.key --client-certificate=user.crt
kubectl --kubeconfig=$CLUSTER.kubeconfig config set-context $CLUSTER --user $USERNAME
kubectl --kubeconfig=$CLUSTER.kubeconfig config use-context $CLUSTER

After configuring kubectl with the previous commands, you should be able to operate on the cluster as the new user. Try f.ex. listing all pods:

kubectl --kubeconfig $CLUSTER.kubeconfig get pods --all-namespaces

If the above command worked, you should now have a fully functional cluster which you can deploy your applications within – have fun!

Route Handling Framework for Choo

I’ve made a complementary route handling framework for Yoshua Wuyts’ JavaScript SPA (Single Page Application) framework Choo: choo-routehandler. For those unfamiliar with Choo, it’s a view framework similar to React and Marko, although with a strong focus on simplicity and hackability and working directly with the DOM instead of a virtual DOM (unlike React).

Basically, the motivation for making the framework was that I found myself implementing the same pattern in my Choo apps: To load data before rendering a view corresponding to a route and to require authentication before accessing certain routes. I didn’t find any good way to handle this baked into Choo, even though it has a rather good routing system built into it. It was also tricky to use Choo’s standard effects/reducers system to implement handling of route change, since these are asynchronously triggered and I could end up handling the same route change several times as a result.

After some trial and failure and good old-fashioned experimentation, I found that I could use the standard MutationObserver API to detect that a new route has been rendered, and take necessary action (i.e. require authentication/load data) subsequently.

The flow of events after a route change has been detected by the framework depends on whether the user is logged in or not and whether the route requires authentication. If the route does require authentication and the user is not logged in, the user is redirected to the login page. Otherwise, the data loading flow commences so that the view’s function for loading data is invoked asynchronously and a loading view is rendered until data is fully loaded. Then the view corresponding to the route is rendered, given the loaded data. It may also be the case that the view has no function for loading data, in which case it’s rendered directly and no data loading takes place.

Event Catalogue for Berlin Released

I’m finally ready to share what I’ve been working on for the last few months, an online catalogue of underground cultural events in the city of Berlin: Experimental Berlin. My best effort to date I think! For this project I’ve been using Yoshua Wuyts’ excellent JavaScript framework Choo.

Maintainership of html-to-react

I’m happy to announce that I’ve taken over maintainership of the popular NPM package html-to-react, which currently has about 3600 downloads per month. This package has the ability to translate HTML markup into a React DOM structure.

An example should give you an idea of what html-to-react brings to the table:

var React = require('react');
var HtmlToReactParser = require('html-to-react').Parser;
 
var htmlInput = '<div><h1>Title</h1><p>A paragraph</p></div>';
var htmlToReactParser = new HtmlToReactParser();

// Turn HTML into a bona fide React component that we can integrate into our React tree
var reactComponent = htmlToReactParser.parse(htmlInput);

MuzHack has a blog

MuzHack now has its own blog!