Creating a Highly Available Secured Kubernetes Cluster on AWS with Kops

Today I will be talking about Kops, which is an official tool for creating Kubernetes clusters on AWS, with support for GCE and VMware vSphere in alpha. It takes a whole lot of the pain out of setting up a Kubernetes cluster yourself, but still presents many challenges to overcome and a great degree of freedom in how you configure the cluster.

I have recently created a Kubernetes cluster on AWS for a client, where I used the Kops tool for the very first time and I will here present what I learnt about implementing best practices with this technology stack. Given the rapid development of Kubernetes itself, and how relatively young Kops is, it proved to be far from a walk in the park to create a production-grade cluster. Documentation is often relatively poor, or just plain missing. The intention of this blog post is to make it easier for others going down the same route.

What we set out to do on this project, is to produce a highly available Kubernetes cluster (on AWS) which you can only SSH into via a dedicated host (a so-called bastion), and that secures the control plane/API via client certificate based authentication and RBAC authorization.

Creating the Cluster Itself

The very first step, which is thankfully quite simple thanks to the awesome power of Kops, is to bring the cluster up. The below kops invocation creates a highly available cluster on AWS, with 5 worker nodes spread among three availability zones and 3 master nodes in the same AZs. For security, all master/worker nodes are in a private subnet and not exposed to the Internet. We also instantiate a bastion host as the sole entry point into the cluster via SSH, and the cluster is configured to enable RBAC as its authorization mode. I chose Flannel as the networking system as I have some experience with it from prior work with Kubernetes and have a good impression of it.

kops --state s3://example.com create cluster --zones \
eu-central-1a,eu-central-1b,eu-central-1c --master-zones \
eu-central-1a,eu-central-1b,eu-central-1c --topology private --networking flannel \
--master-size m4.large --node-size m4.large --node-count 5 \
--bastion --cloud aws --ssh-public-key id_rsa.pub --authorization RBAC --yes \
example.com

On the topic of high availability, since the cluster has three master nodes within three different availability zones, the cluster is protected from the outage of individual AZs and is still available if a master goes down. The same goes for worker nodes.

Exporting the Kubectl Configuration

After creating the cluster, we would like to generate a configuration file to use in order to have kubectl operate against our cluster. We do this with the following command:

KUBECONFIG=$CLUSTER.kubeconfig kops export kubecfg $CLUSTER

Configuring Cluster Components for RBAC

In order for certain cluster components to function with RBAC enabled, some configuration is required. Basically, what we need to do is to bind the right roles to service accounts to allow the latter to perform certain tasks on behalf of pods.

Configure the cluster when it’s ready by applying the configuration files in the sections below:

kubectl --kubeconfig=$CLUSTER.kubeconfig apply -f kube-system-rbac.yaml
kubectl --kubeconfig=$CLUSTER.kubeconfig apply -f kube-flannel-rbac.yaml

Default System Service Account

The default service account in the kube-system namespace must be bound to the cluster-admin role:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: system:default-sa
subjects:
  - kind: ServiceAccount
    name: default
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

Flannel Service Account

The flannel service account in the kube-system namespace must be given a role with certain permissions in order to enable the Flannel networking component to do its job:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
rules:
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes/status
    verbs:
      - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system

Client Certificate Based Authentication

I decided to implement user authentication via TLS certificates, as this is directly supported in the kubectl tool and ties easily in with RBAC authorization. The trick here is to get hold of the certificate authority (CA) certificate and key that Kops used when creating the cluster, as it will allow us to generate valid user certificates. Luckily, these files are stored in Kops' S3 bucket. The following commands copies the CA key and certificate to the local directory:

aws s3 cp s3://$BUCKET/$CLUSTER/pki/private/ca/$KEY ca.key
aws s3 cp s3://$BUCKET/$CLUSTER/pki/issued/ca/$CERT ca.crt

Now that we have the CA key and certificate, we can generate a user certificate with the openssl command line tool. The procedure consists of first generating a private key, then with the previously generated key generating a signing request for a certificate representing a user with username $USERNAME and finally signing the certificate with the help of the CA key and certificate. The below commands will produce a key and certificate named user.key and user.crt, respectively:

openssl genrsa -out user.key 4096
openssl req -new -key user.key -out user.csr -subj '/CN=$USERNAME/O=developer'
openssl x509 -req -in user.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out user.crt \
-days 365

Granting Cluster Administrator Rights to User

We would like for our user, as represented by the certificate, to have cluster administrator rights, meaning that they are basically permitted any operation on the cluster. The way to do this is to create a ClusterRoleBinding that gives the cluster-admin role to the new user, as in the following command:

kubectl --kubeconfig $CLUSTER.kubeconfig create clusterrolebinding \
$USERNAME-cluster-admin-binding --clusterrole=cluster-admin --user=$USERNAME

After granting your user this role, we can start using it instead of the default admin user, as you’ll see in the next section.

Identifying User through Certificate via Kubectl

Given the certificate we created for the user previously, and having assigned the user the cluster-admin role, we can now identify towards Kubernetes by modifying the kubectl configuration. We configure kubectl to authenticate with the certificate towards the cluster via the following commands:

kubectl --kubeconfig=$CLUSTER.kubeconfig config set-credentials $USERNAME \
--client-key=user.key --client-certificate=user.crt
kubectl --kubeconfig=$CLUSTER.kubeconfig config set-context $CLUSTER --user $USERNAME
kubectl --kubeconfig=$CLUSTER.kubeconfig config use-context $CLUSTER

After configuring kubectl with the previous commands, you should be able to operate on the cluster as the new user. Try f.ex. listing all pods:

kubectl --kubeconfig $CLUSTER.kubeconfig get pods --all-namespaces

If the above command worked, you should now have a fully functional cluster which you can deploy your applications within - have fun!