Installing Elasticsearch/Kibana 5.5 within Kubernetes cluster on AWS

In my previous blog post I showed how to use the Kops tool to create a production ready Kubernetes cluster on Amazon Web Services (AWS). In this follow-up post I will show how to install Elasticsearch and its graphical counterpart Kibana in the cluster, in order to be able to collect and store logs from your cluster and search/read them. We will also install Fluentd as this component is responsible for transmitting the standard Kubernetes logs to Elasticsearch. This is generally known as the ELK stack, which stands for Elasticsearch, Logstash (precursor to Fluentd) and Kibana.

First of all, it should be mentioned that there is a standard addon for installing ELK in Kubernetes clusters, as part of the Kubernetes repository. The solution I’m going to present in this blog post is derived from that addon, but I’ve rewritten it in order to upgrade to Elasticsearch and Kibana 5.5.0 as the addon at the time of writing is based on version 5.4.0, as well as to include the repository-s3 Elasticsearch plugin. We need the latter plugin in order to be able to back up Elasticsearch to S3 storage in AWS. I have made a pull request in order to try to consolidate my modified version of the ELK Kubernetes addon with the original.

I will go through each part of the stack in succession and explain shortly how it functions. In order to install the logging stack in your Kubernetes cluster, apply the manifests via kubectl: kubectl apply -f *.yaml.

Elasticsearch

These manifests install Elasticsearch itself as a StatefulSet of two pods that will allocate an AWS EBS volume of 20 GB per pod (the size was chosen somewhat at random, pick a size that make sense for your workload). In order to configure Elasticsearch’s repository-s3 plugin, environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_KEY also have to be set, via a Kubernetes secret called bootstrap-environment. A Service is created in front of the StatefulSet pods to load balance the StatefulSet pods.

Elasticsearch is also configured to run under the service account elasticsearch-logging, which gets bound to the role elasticsearch-logging in order for it to have the right permissions.

We use our own custom Elasticsearch image that inherits from the official one and which includes a binary from the original Kubernetes addon in order to discover necessary Elasticsearch parameters at runtime, to make it integrate with the Kubernetes cluster.

es-clusterrole.yaml:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: elasticsearch-logging
  labels:
    k8s-app: elasticsearch-logging
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
  - ""
  resources:
  - "services"
  - "namespaces"
  - "endpoints"
  verbs:
  - "get"

es-serviceaccount.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: elasticsearch-logging
  namespace: kube-system
  labels:
    k8s-app: elasticsearch-logging
    version: v1
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile

es-clusterrolebinding.yaml:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  namespace: kube-system
  name: elasticsearch-logging
  labels:
    k8s-app: elasticsearch-logging
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
subjects:
- kind: ServiceAccount
  name: elasticsearch-logging
  namespace: kube-system
  apiGroup: ""
roleRef:
  kind: ClusterRole
  name: elasticsearch-logging
  apiGroup: ""

es-statefulset.yaml:

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: elasticsearch-logging-v1
  namespace: kube-system
  labels:
    k8s-app: elasticsearch-logging
    version: v1
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  serviceName: elasticsearch-logging
  replicas: 2
  selector:
    matchLabels:
      k8s-app: elasticsearch-logging
      version: v1
  template:
    metadata:
      labels:
        k8s-app: elasticsearch-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccountName: elasticsearch-logging
      containers:
      - image: aknudsen/elasticsearch-k8s-s3:5.5.0-14
        name: elasticsearch-logging
        resources:
          # need more cpu upon initialization, therefore burstable class
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        ports:
        - containerPort: 9200
          name: db
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        volumeMounts:
        - name: elasticsearch-logging
          mountPath: /data
        env:
        - name: "NAMESPACE"
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: bootstrap-environment
              key: es-s3-access-key
        - name: AWS_SECRET_KEY
          valueFrom:
            secretKeyRef:
              name: bootstrap-environment
              key: es-s3-secret-key
  volumeClaimTemplates:
  - metadata:
      name: elasticsearch-logging
    spec:
      storageClassName: gp2
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 20Gi

es-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-logging
  namespace: kube-system
  labels:
    k8s-app: elasticsearch-logging
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "Elasticsearch"
spec:
  ports:
  - port: 9200
    protocol: TCP
    targetPort: db
  selector:
    k8s-app: elasticsearch-logging

env-secret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-environment
  namespace: kube-system
type: Opaque
data:
  es-s3-access-key: example
  es-s3-secret-key: example

Fluentd

Fluentd is installed as a DaemonSet, which means that a corresponding pod will run on every Kubernetes worker machine in order to collect its logs (and send them to Elasticsearch). Furthermore, the pods run as the service account fluentd-es which is bound to the cluster role fluentd-es in order to have the necessary permissions.

Our Fluentd DaemonSet is configured to use my custom Docker image for Fluentd as it configures Fluentd to use environment variables FLUENT_ELASTICSEARCH_USER and FLUENT_ELASTICSEARCH_PASSWORD as credentials towards Elasticsearch.

fluentd-es-clusterrole.yaml

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: fluentd-es
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
  - ""
  resources:
  - "namespaces"
  - "pods"
  verbs:
  - "get"
  - "watch"
  - "list"

fluentd-es-serviceaccount.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd-es
  namespace: kube-system
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile

fluentd-es-clusterrolebinding.yaml:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: fluentd-es
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
subjects:
- kind: ServiceAccount
  name: fluentd-es
  namespace: kube-system
  apiGroup: ""
roleRef:
  kind: ClusterRole
  name: fluentd-es
  apiGroup: ""

fluentd-es-ds.yaml

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd-es
  namespace: kube-system
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    version: v1.23
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-es
        kubernetes.io/cluster-service: "true"
        version: v1.23
      # This annotation ensures that fluentd does not get evicted if the node
      # supports critical pod annotation based priority scheme.
      # Note that this does not guarantee admission on the nodes (#40573).
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      serviceAccountName: fluentd-es
      containers:
      - name: fluentd-es
        image: aknudsen/fluentd-k8s-es:1.23
        command:
          - '/bin/sh'
          - '-c'
          - '/usr/sbin/td-agent $FLUENTD_ARGS'
        env:
        - name: FLUENTD_AGRS
          value: -q
        # These are default ElasticSearch credentials
        - name: FLUENT_ELASTICSEARCH_USER
          value: "elastic"
        - name: FLUENT_ELASTICSEARCH_PASSWORD
          value: "changeme"
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      nodeSelector:
        beta.kubernetes.io/fluentd-ds-ready: "true"
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

Kibana

Unlike the original Kubernetes addon, we don’t use a custom image for Kibana, just the official one as that works out of the box. There’s not much to say about our Kibana manifests; we install a Deployment, which ensures that one pod is always running, and a Service in front of it (which is capable of load balancing in case there should be several pods in parallel).

kibana-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kibana-logging
  namespace: kube-system
  labels:
    k8s-app: kibana-logging
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kibana-logging
  template:
    metadata:
      labels:
        k8s-app: kibana-logging
    spec:
      containers:
      - name: kibana-logging
        image: docker.elastic.co/kibana/kibana:5.5.0
        resources:
          # keep request = limit to keep this container in guaranteed class
          limits:
            cpu: 100m
          requests:
            cpu: 100m
        env:
          - name: "ELASTICSEARCH_URL"
            value: "http://elasticsearch-logging:9200"
        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP

kibana-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: kibana-logging
  namespace: kube-system
  labels:
    k8s-app: kibana-logging
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "Kibana"
spec:
  ports:
  - port: 5601
    protocol: TCP
    targetPort: ui
  selector:
    k8s-app: kibana-logging