- Kubernetes logs: Pod logs, container logs, Docker logs, kubelet logs, and master node logs - Mon, Sep 25 2023
- Kubernetes DaemonSets - Wed, Sep 6 2023
- Static Pods in Kubernetes - Fri, Sep 1 2023
Before I jump into the configuration, let me give a brief overview of how encryption at rest will be achieved.
“Encrypted at rest” refers to the encryption of data when it is stored in a persistent state as opposed to "encrypted in transit." You might already know that Kubernetes maintains everything as configuration files (also known as manifests), which are nothing but simple YAML or JSON files.
To encrypt Secrets in etcd (database that stores the Kubernetes configuration), we will create a new configuration file with the kind set to EncryptionConfiguration. Under the encryption config file, we will define an encryption provider (discussed later in this post) and specify an encryption key that can be either manually generated or managed by an external key provider (e.g., Google or AWS KMS) to carry out the encryption.
The encryption configuration file is then passed as a startup parameter to the kube-apiserver. Once this is done, every time you create a Kubernetes Secret, it is first encrypted based on the encryption configuration and then stored in etcd. The diagram below provides a high-level overview of how secrets are stored with and without encryption in the etcd database.
Understanding how secrets are stored with and without encryption
The kube-apiserver performs the encoding and decoding automatically, as Secrets are read from and written to the etcd database. The important point to note is that Secret values in etcd are stored in clear text unless encryption is enabled.
Determine whether encryption is enabled
Let's quickly create a Secret named db-secret in the default namespace with the following command:
kubectl create secret generic db-secret \ --from-literal username=blueadmin \ --from-literal password=P@ssw0rd123
To determine whether etcd encryption is already enabled in the Kubernetes cluster, install the etcd-client package on the master node, as shown below:
sudo apt install etcd-client
Install the etcd client tool in Ubuntu Linux
The etcd-client package gives you a tool called etcdctl (similar to kubectl), which you can use to communicate with the etcd datastore. The Kubeadm tool, which I used to set up my Kubernetes cluster, creates Pod manifest files for various control plane components in the /etc/kubernetes/manifests directory. Kubelet watches this directory to create static Pods during startup. You will learn about static Pods in the next post. For now, just remember that Kubeadm configures all control plane components as static Pods. Let's take a look at this directory on the control plane (master) node.
sudo ls -l /etc/kubernetes/manifests/
View the static manifest files for various control plane components
At this time, we are only interested in the etcd.yaml file, which defines the etcd datastore for our Kubernetes cluster. Let's see what the etcd.yaml file looks like.
Viewing the etcd Pod manifest file in Kubernetes
It is a regular Pod configuration file, as we saw in the Kubernetes Pod post. The kubeadm tool generates all the TLS certificates for secure communication in the /etc/kubernetes/pki/ directory. If we take a look at the spec section, you will notice a /etc/kubernetes/pki/etcd/ subdirectory in particular, which holds TLS certificates for the etcd Pod. To view an unencrypted Secret stored in the etcd datastore, run the etcdctl command, as shown below:
sudo ETCDCTL_API=3 etcdctl \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ get /registry/secrets/default/db-secret
Viewing a Secret stored in the etcd datastore using the etcdctl utility
Here, we set the ETCDCTL_API environment variable to version 3, used the etcdctl command, and passed the TLS certificate files for authentication. In the etcd datastore, Secrets are stored under /registry/secrets/<namespace>/<secret-name>, so we mentioned the exact path, including the secret name (db-secret, in our case). The <default> here stands for the namespace. If your Secret exists in the production namespace, you will type production instead of default. As you can see in the screenshot, Secret values are stored in clear text, which means encryption at rest is not yet enabled.
If you don't want to use the etcdctl tool and TLS certificates for some reason, you can open the etcd database file directly with a text editor (like nano or vim), which is available at /var/lib/etcd/member/snap/db on the control plane node. Remember, this is a binary file so it will not look pretty, but you can search for the Secret name and see the values in clear text. See the screenshot below for reference.
sudo nano /var/lib/etcd/member/snap/db
Opening the etcd database file with a text editor and searching the Secret name
Do not make any changes to this file directly, as it could corrupt the database and cause potential issues with your Kubernetes cluster. My point was just to show how Secrets are stored without encryption.
Encryption providers
Encryption providers allow you to encrypt confidential data at rest in a Kubernetes cluster. Some providers that can be used with Kubernetes include:
- Identity: This is the default provider in Kubernetes, which does not provide any encryption at all. When it is used, the resources are written to the etcd database in clear text.
- AES-CBC: This stands for Advanced Encryption Standard–Cipher Block Chaining. It encrypts each data block with a key and an initialization vector. The output of each block is then used as input for the next block, which creates a chain of dependencies. It is no longer considered safe due to a padding Oracle vulnerability. Furthermore, it stores keys on the control plane node itself so it can only protect against an etcd compromise but doesn't offer any protection from host compromise.
- AES-GCM: This stands for Advanced Encryption Standard–Galois/Counter Mode. It is better than AES-CBC as it offers padding Oracle attack resistance, inherent authenticated encryption capabilities, better parallelism, and overall improved security. It also stores keys on the control plane node, so it fails to offer any protection in the event of host compromise.
- Secretbox: This uses the NaCl secretbox package to encrypt and decrypt data. The NaCl secretbox is a function that encrypts and authenticates a message using a secret key and a nonce. A nonce is a number that is used only once for each message to prevent replay attacks. To use this provider, you need to specify a 32-byte-long base64-encoded value as the secret parameter in the encryption config file. It offers strong encryption but doesn't help in the event of host compromise, since the keys are stored on the master node itself.
- KMS v1 (deprecated): The Key Management Service version 1 provider used the concept of envelope encryption, in which data was encrypted using data encryption keys (DEKs). The DEKs, in turn, were encrypted using key encryption keys (KEKs). It provided the strongest encryption but was slower, so it was deprecated beginning with Kubernetes v1.28.
- KMS v2 (beta): This uses the same concept of envelope encryption with improved performance, key rotation, and health checks. Kubernetes uses a KMS plugin to support this provider, and the plugin implements a gRPC server over a Unix socket on the control plane node to communicate with the remote KMS. The important thing to note is that key encryption keys (KEKs) are stored in an external KMS service (like AWS KMS or the Azure key vault) outside of the Kubernetes cluster.
Encrypt your secrets
To enable encryption, we first need to create an encryption configuration file. For this demo, I will use the secretbox provider, which offers strong encryption as compared to AES-CBC or AES-GCM. The secretbox provider requires a 32-byte-long key in base64 encoded form, so let's create one with the following command:
head -c 32 /dev/urandom | base64
Remember, all control plane components run as static Pods. So, when you create a file on the master node, it will not be directly accessible by the kube-apiserver Pod unless we define a persistent volume and mount it. If we use the kubectl describe command on the kube-apiserver Pod, you will notice that the /etc/kubernetes/pki/ directory is already mounted as a HostPath volume, as shown in the screenshot below:
kubectl describe pods kube-apiserver-kube-srv1.testlab.local -n kube-system
Describing the kube apiserver Pod in Kubernetes
Don't forget to check the name of your kube-apiserver using the kubectl get pods -n kube-system command. To avoid all the hassle of creating and mounting a new volume, we can create the encryption config YAML file under the /etc/kubernetes/pki/ directory so it will automatically become available for the Pod. Let's create the enc-config.yaml file.
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- secretbox:
keys:
- name: key1
secret: <base64_encoded_key>
- identity: {} # default encryption provider to allow reading unencrypted secrets
Creating the encryption configuration file in Kubernetes
Notice that the resources field accepts multiple resources to encrypt. For example, you can write configmaps under secrets in the same way to encrypt all the ConfigMaps as well. Also, don't forget to paste the base64-encoded value generated by the head -c 32 /dev/urandom | base64 command under the secret field. After saving this file, you need to open the kube-apiserver Pod manifest file in a text editor and specify the encryption config file using an --encryption-provider-config flag.
sudo nano /etc/kubernetes/manifests/kube-apiserver.yaml
Specify encryption provider config in the kube apiserver Pod manifest
I know we haven't discussed the static Pods yet. As soon as a static Pod manifest is changed, the Kubelet will recreate the Pod based on the updated configuration. As a result, your kube-apiserver will go down for a few minutes, but if your configuration is right, a new Pod will be created soon enough.
Verify the kube apiserver Pod is recreated once the manifest is changed
To confirm whether the updated configuration is applied, you could run this command:
sudo ps -ef | grep kube-apiserver | grep "encryption-provider-config"
Determine whether encryption is enabled in the Kubernetes cluster
If you see the --encryption-provider-config parameter passed to the kube-apiserver, it essentially means that the etcd encryption is enabled. Any new Secrets we create after enabling the encryption will now be encrypted before being stored in the etcd database. But what about the old Secrets that already existed in the Kubernetes cluster? Well, those are not encrypted yet. To prove this, let's view the Secret (db-secret) that we created at the beginning.
Viewing an unencrypted Secret using the etcdctl command
To encrypt all the old Secrets, you can run the following command:
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
Encrypting all existing Kubernetes Secrets
This command grabs all the Secrets from all namespaces and then recreates them. Since our kube-apiserver now supports encryption, all the Secrets are recreated and stored in encrypted form. If we repeat the previous command to view the db-secret again, you will notice that there is no plain-text value stored in etcd.
Viewing an encrypted Secret using the etcdctl command
That's it. All your Kubernetes Secrets are now encrypted at rest.
Conclusion
The procedure discussed in this post only helps in situations in which an attacker gains access to the etcd datastore for offline use. He cannot simply view the Secrets stored within etcd since they are now encrypted. But if the attacker gains access to the master node, he can view the encryption key you specified in the encryption config file and use it to decrypt all your Secrets. Therefore, it is highly recommended to use the KMSv2 encryption provider whenever possible, since it stores the keys away from the Kubernetes cluster. Also, it is important that you properly secure your Kubernetes cluster with role-based access controls (RBAC) and always follow the principle of least privilege.
Subscribe to 4sysops newsletter!
Furthermore, in highly available Kubernetes clusters, there are usually multiple instances of the etcd datastore. In this case, the encryption configuration file should be the same, and you should also configure TLS communication between instances to ensure that the data remains encrypted during the transit. For more information, you can read the best practices around Kubernetes Secrets.
