How to perform Kubernetes pentesting and secure it?

Kubernetes is a great platform for container management that has shown a breakthrough lately, both in terms of functionality and in terms of security and resiliency. Specialists claim that Kubernetes’ architecture makes it easy to survive different types of outages and stay active despite everything, making it a great option for pentesting.

On this occasion, pentesting experts from the International Institute of Cyber Security (IICS) will show you how to perform multiple hacking tasks on Kubernetes, including cluster disruption, certificate deletion, and node connection, all without downtime for the services being run.

Before continuing, as usual we remind you that this article was prepared for informational purposes only and should not be taken with a call to action. IICS is not responsible for the misuse of the information contained herein.

To get started, let’s remember that the main Kubernetes control panel consists of a few components:

  • etcd: Used as a database
  • kube-apiserver: API and heart of the cluster
  • kube-controller-manager: For the deployment of operations on Kubernetes resources
  • kube-scheduler: Main planner
  • kubelets: To launch containers directly on hosts

Each of these components is protected by a set of TLS, client and server certificates, whose purpose is to authenticate and authorize components between them, mention pentesting experts. These resources are not stored anywhere in the Kubernetes database, except in certain cases, but are presented as normal files:

# tree /etc/kubernetes/pki/ /etc/kubernetes/pki/ ├── apiserver.crt ├── apiserver-etcd-client.crt ├── apiserver-etcd-client.key ├── apiserver.key ├── apiserver-kubelet-client.crt ├── apiserver-kubelet-client.key ├── ca.crt ├── ca.key ├── CTNCA.pem ├── etcd │ ├── ca.crt │ ├── ca.key │ ├── healthcheck-client.crt │ ├── healthcheck-client.key │ ├── peer.crt │ ├── peer.key │ ├── server.crt │ └── server.key ├── front-proxy-ca.crt ├── front-proxy-ca.key ├── front-proxy-client.crt ├── front-proxy-client.key ├── sa.key └── sa.pub

The components themselves are described and run on the masters as static pods from the /etc/kubernetes/manifests/ directory.

The most important thing is to know how to make a functional cluster of all this; let’s imagine that the Kubernetes components mentioned above somehow interact with each other. The basic diagram looks like this:

To communicate, they need TLS certificates, which, in principle, can be taken to a separate level of abstraction and completely rely on their distribution tool, be it kubeadm, kubespray or something else, pentesting experts mention. In this example, we’ll look at kubeadm because it’s the most common Kubernetes deployment tool and is often used as part of other solutions.

Let’s say we already have a cluster in place. Let’s start with the fun part:

rm -rf /etc/kubernetes/

In masters, this directory contains:

  • A set of certificates and CAs for etcd (c/etc/kubernetes/pki/etcd)
  • A set of certificates and CAs for Kubernetes (c/etc/kubernetes/pki)
  • Kubeconfig for cluster-admin, kube-controller-manager, kube-Scheduler and kubelet (each also has a base64-coded CA certificate for our /etc/kubernetes/*.conf cluster)
  • A set of static manifests for etcd, kube-apiserver, kube-Scheduler and kube-controller-manager (c/etc/kubernetes/manifests)

Suppose we lost everything at once.    

Fixing the control plane

To avoid confusion, the researchers recommend making sure that all of our control plane pods are also stopped:

crictl rm `crictl ps -aq`

It should be remembered that kubeadm, by default, does not overwrite existing certificates and kubeconfigs, so to reissue them; you must first manually delete them.

Let’s start by restoring etcd, because if we had a quorum (3 or more master nodes), the etcd cluster would not start without most of them present.

kubeadm init phase certs etcd-ca

The above command will generate a new CA for our etcd cluster. Since all other certificates must be signed by it, we will copy it along with the private key to the rest of the master nodes:

/etc/kubernetes/pki/etcd/ca.{key,crt}

Now, let’s regenerate the rest of the etcd certificates and static manifests on all nodes in the control plane:

kubeadm init phase certs etcd-healthcheck-client
kubeadm init phase certs etcd-peer
kubeadm init phase certs etcd-server
kubeadm init phase etcd local

At this stage, we should already have an etcd cluster working, mention the experts in pentesting:

# crictl ps
CONTAINER ID        IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID
ac82b4ed5d83a       0369cf4303ffd       2 seconds ago       Running             etcd                0                   bc8b4d568751b

Now let’s do the same, but for Kubernetes, on one of the master nodes, run:

kubeadm init phase certs all
kubeadm init phase kubeconfig all
kubeadm init phase control-plane all
cp -f /etc/kubernetes/admin.conf ~/.kube/config

The above commands will generate all SSL certificates for our Kubernetes cluster, as well as statistics for the manifests and kubeconfigs for Kubernetes services. If you are using kubeadm to join kubeletes, you will also need to update the cluster information configuration in the kube-public namespace that still contains the hash of your old CA.

kubeadm init phase bootstrap-token

Since all certificates from other instances must also be signed by a CA, copy them to the other nodes in the control plane and repeat the above commands on each of them, pentesting experts recommend.

/etc/kubernetes/pki/{ca,front-proxy-ca}.{key,crt}
/etc/kubernetes/pki/sa.{key,pub}

As an alternative to manual copying certificates, you can now use the Kubernetes interface, for example, the following command:

kubeadm init phase upload-certs --upload-certs

Certificates will then be encrypted and uploaded to Kubernetes for 2 hours, so you can rejoin the masters as follows:

kubeadm join phase control-plane-prepare all kubernetes-apiserver:6443 --control-plane --token cs0etm.ua7fbmwuf1jz946l     --discovery-token-ca-cert-hash sha256:555f6ececd4721fed0269d27a5c7f1c6d7ef4614157a18e56ed9a1fd031a3ab8 --certificate-key 385655ee0ab98d2441ba8038b4e8d03184df1806733eac131511891d1096be73
kubeadm join phase control-plane-join all

It’s worth noting that the Kubernetes API has another configuration that stores the CA certificate for the front proxy client, used to authenticate apiserver requests to webhooks and other aggregation layer services. Fortunately, kube-apiserver updates it automatically. However, you may want to manually clean it from the old certificates:

kubectl get cm -n kube-system extension-apiserver-authentication -o yaml

In any case, at this stage we already have a fully functioning control plane.

Fixing the workers

This command will list all the nodes in the cluster, although they will now all be in the NotReady state:

kubectl get node

This happens because they are still using the old certificates and waiting for server requests signed by the old CA. To work around this issue, we’ll use kubeadm and create a logging node in the cluster.

When both teachers have access to the CA and can connect locally:

systemctl stop kubelet
rm -rf /var/lib/kubelet/pki/ /etc/kubernetes/kubelet.conf
kubeadm init phase kubeconfig kubelet
kubeadm init phase kubelet-start

Then, for the union workers, we will generate a new token:

kubeadm token create --print-join-command

And in each of them we execute:

systemctl stop kubelet
rm -rf /var/lib/kubelet/pki/ /etc/kubernetes/pki/ /etc/kubernetes/kubelet.conf 
kubeadm join phase kubelet-start kubernetes-apiserver:6443  --token cs0etm.ua7fbmwuf1jz946l     --discovery-token-ca-cert-hash sha256:555f6ececd4721fed0269d27a5c7f1c6d7ef4614157a18e56ed9a1fd031a3ab8

Attention: /etc/kubernetes/pki/ it is not necessary to delete the directory in the masters, since it contains all the necessary certificates, say the experts in pentesting.

The previous procedure will reconnect all your cublets to the cluster without affecting the containers that are already running on them. However, if you have many nodes in the cluster and you are doing this at the same time, you might have a situation where Controller-Manager starts recreating containers with NotReady nodes and tries to start them on the active nodes in the cluster.

To avoid this, we can temporarily stop the controller-administrator, on the masters:

rm /etc/kubernetes/manifests/kube-controller-manager.yaml
crictl rmp `crictl ps --name kube-controller-manager -q`

The last command is only necessary to make sure that the controller-administrator is not actually running. Once all the nodes in the cluster are connected, we can generate a static manifest for the back of the controller-administrator.

To do this, in all teachers they will have to execute:

kubeadm init phase control-plane controller-manager

Note that you need to do this at the stage where you have already generated the join token; otherwise, the connection operation will crash when trying to read the cluster-info token.

If the kubelet is configured to receive a certificate signed by your CA (optional TLSBootstrap server: true), you will also need to reconfirm the csr of your kubelets:

kubectl get csr
kubectl certificate approve <csr>

Fixing ServiceAccounts

There is one more thing. Since we lost /etc/kubernetes/pki/sa.key, pentesting experts mention that this is the same key that signed the jwt tokens for all our ServiceAccounts, so we need to recreate the tokens for each of them.

This can be done by simply removing the token field from all secrets as kubernetes.io/service-account-token:

kubectl get secret --all-namespaces | awk '/kubernetes.io\/service-account-token/ { print "kubectl patch secret -n " $1 " " $2 " -p {\\\"data\\\":{\\\"token\\\":null}}"}' | sh –x

After that, kube-controller-manager will automatically generate new tokens signed with a new key, pentesting experts mention. Unfortunately, not all microservices can re-read the token on the fly, and you’ll most likely need to manually restart the containers where they’re used:

kubectl get pod --field-selector 'spec.serviceAccountName!=default' --no-headers --all-namespaces | awk '{print "kubectl delete pod -n " $1 " " $2 " --wait=false --grace-period=0"}'

For example, this command will generate a list of commands to delete all modules using a non-standard service account. I recommend starting with the kube system namespace. You have kube-proxy and the CNI plugin installed, which are vital for configuring your microservices to communicate.

To learn more about information security risks, malware variants, vulnerabilities and information technologies, feel free to access the International Institute of Cyber Security (IICS) websites.