Is Your etcd an Open Door for Cyber Attacks? How to Secure Your Kubernetes Clusters & Nodes

Kubernetes has become the de facto orchestration platform for managing containerized applications, but with its widespread adoption, the security of Kubernetes clusters has come under greater scrutiny. Central to Kubernetes’ architecture is etcd, a highly-available key-value store used to persist the cluster’s state and its configuration details. While etcd is essential for the Kubernetes cluster’s functionality, it also presents a tantalizing target for attackers. A new research shows how a compromised etcd can lead to full control over the cluster, allowing unauthorized changes to resources, tampering with operations, and potentially leading to data breaches or service disruptions. Kubernetes architecture is divided into two main parts: the control-plane and the nodes. The control-plane acts as the central hub and includes components like the kube-apiserver (the brain of the cluster), scheduler (which assigns pods to nodes), control manager (which manages the status of various cluster elements), and etcd (a key-value store for cluster data). Nodes contain components like kubelet (which ensures pods are running correctly) and kube-proxy (which connects services to the network).

Etcd is more than just a storage component in Kubernetes; it’s a critical part of the architecture. It’s a key-value database that stores all the cluster’s information. The data in etcd is stored using a serialization format called Protobuf, developed by Google for efficient data exchange between systems. Kubernetes uses Protobuf to serialize different kinds of data, such as pods and roles, each requiring different parameters and definitions.

The research describes a tool called auger, which can serialize and deserialize data stored in etcd into more readable formats like YAML and JSON. NCC Group has developed a wrapper for auger called kubetcd to showcase how a compromised etcd can be critical.

However, exploiting etcd has limitations. You’d need root access to the host running etcd and have the necessary certificates for authentication. Moreover, this technique mainly applies to self-managed Kubernetes environments, not managed ones offered by cloud providers.

Top 8 Free Tools for security testing and audit of your Kubernetes cluster in 2022

Direct access to etcd could be used to tamper with Kubernetes resources, like changing the startup date of a pod or creating inconsistencies that make pods difficult to manage.

Direct access to etcd, the key-value store for Kubernetes, could allow an attacker to make unauthorized modifications to the cluster state, which could lead to various security issues. Here are some examples of how this could be exploited:

Changing Pod Timestamps:

• Attackers with access to etcd could alter the creation timestamps of pods. This could be used to disguise malicious pods as long-running, trusted processes.
• Example:
  bash kubetcd create pod nginx -t nginx --time 2000-01-31T00:00:00Z
  This command sets the timestamp of an nginx pod to January 31, 2000, making it appear as if it has been running for over 20 years.

Gaining Persistence:

• By changing the path where a pod’s data is stored in etcd, an attacker could prevent the pod from being deleted by the regular Kubernetes API commands.
• Example:
  bash kubetcd create pod maliciouspod -t nginx -p randomentry
  This command creates a pod and stores its data under a different path, making it difficult for Kubernetes to manage or delete it.

Creating Semi-hidden Pods:

• Attackers could manipulate the namespace entries in etcd to run pods in a namespace that does not match their manifest. This can cause confusion and make pods difficult to manage.
• Example:
  bash kubetcd create pod hiddenpod -t nginx -n invisible --fake-ns
  This command creates a pod that appears to run in the default namespace but is actually associated with the invisible namespace in etcd. This pod will not be listed under the default namespace, making it semi-hidden.

Bypassing Admission Controllers:

• Admission Controllers enforce security policies in Kubernetes. By injecting resources directly into etcd, an attacker can bypass these controls and deploy privileged pods that could compromise the cluster.
• Example:
  bash kubetcd create pod privilegedpod -t nginx -n restricted-ns -P
  This command injects a privileged pod into a namespace that is supposed to be restricted by Pod Security Admission (PSA) policies.

Tampering with Cluster Roles and Role Bindings:

• Attackers can modify roles and role bindings directly in etcd to escalate privileges.
• Example:
  bash kubetcd modify rolebinding admin-binding --clusterrole=cluster-admin --user=attacker
  This hypothetical command changes a role binding to give the attacker user cluster-admin privileges.

These examples show the potential dangers if an attacker gains direct access to etcd. They can make changes that are not subject to the usual Kubernetes API checks and balances, leading to unauthorized control over the cluster’s resources. This is why securing etcd access is critical in a Kubernetes environment.

Mitigation

To mitigate the risks associated with etcd and prevent the kinds of tampering mentioned earlier, several steps and best practices should be implemented:

Access Control:

• Restrict access to etcd by implementing strong authentication and authorization mechanisms. Use TLS client certificates to secure communication with etcd.
• Regularly rotate etcd access credentials and audit access logs to detect unauthorized access attempts.

Network Policies:

• Limit network access to etcd servers, ensuring that only specific, authorized components can communicate with etcd.
• Implement firewall rules or Kubernetes network policies to control the traffic towards etcd servers.

Etcd Encryption:

• Enable encryption at rest for etcd to protect sensitive data. Even if attackers gain physical access to the etcd storage, they should not be able to read the data without the encryption keys.

Regular Backups:

• Regularly back up the etcd data store. In case of a breach, this allows the cluster to be restored to a known good state.
• Ensure backup integrity by verifying and testing backups periodically.

Monitoring and Auditing:

• Implement monitoring to detect abnormal activities, such as unexpected changes in etcd.
• Use tools like etcd’s built-in audit capabilities, Falco, or other intrusion detection systems to alert on suspicious behavior.

Least Privilege Principle:

• Apply the principle of least privilege to etcd access. Ensure that only the necessary components have the minimum required access level to perform their functions.

Patch Management:

• Regularly update etcd to the latest version to mitigate vulnerabilities caused by software defects.

Admission Controllers:

• Use Admission Controllers like OPA Gatekeeper or Kyverno to define and enforce policies that can help prevent the creation of unauthorized resources within Kubernetes.

Security Contexts and Policies:

• Apply Security Contexts and Pod Security Policies (or their successors, like Pod Security Admission) to enforce security-related settings in pods and prevent privilege escalation.

Disaster Recovery Plan:

• Have a disaster recovery plan in case etcd is compromised. This plan should include steps to isolate affected systems, revoke compromised credentials, and restore from backups.

Education and Training:

• Train the operations team to understand the security risks associated with etcd and Kubernetes, and how to apply best practices for securing the cluster.

By implementing these mitigations, organizations can significantly reduce the risk of unauthorized access and tampering with etcd, thus securing their Kubernetes clusters. Mitigating the risks associated with etcd ensures the integrity and reliability of Kubernetes clusters. By implementing industry best practices for security and maintaining a proactive stance on potential vulnerabilities, organizations can confidently deploy and manage their containerized workloads, keeping them secure in an ever-evolving threat landscape.