- Kubernetes logs: Pod logs, container logs, Docker logs, kubelet logs, and master node logs - Mon, Sep 25 2023
- Kubernetes DaemonSets - Wed, Sep 6 2023
- Static Pods in Kubernetes - Fri, Sep 1 2023
The Kubernetes control plane manages normal (nonstatic) Kubernetes Pods and comprises several components, such as kube-apiserver (API server), etcd (key–value store), kube-scheduler (assigns Pods to nodes), and kube-controller-manager (centralized control of processes).
kubelet is a service (daemon) that runs on each node in a Kubernetes cluster and communicates with the Kubernetes control plane. You can think of kubelet as a captain on a ship, which receives instructions from the Kubernetes API server to manage the Pods and containers that are assigned to a node and reports the status of a node back to the API server.
Why Static Pods?
You might be wondering what the point is of having static Pods when the API server is already in a Kubernetes cluster. Well, there are situations in which an API server isn't available, for instance, when a control plane node starts up. In that case, kubelet must know how to operate without an API server or even a Kubernetes cluster. This is where the concept of static Pods comes into the picture.
Static Pods are created from configuration files (a.k.a. manifests) that are placed in a designated directory that kubelet checks periodically. When you make any changes to a manifest file, kubelet recreates the Pod with the updated configuration. If you delete a manifest file, kubelet deletes the corresponding Pod, too. The kubeadm tool leverages this kubelet feature to bootstrap various control plane components to give you a fully functional Kubernetes cluster under the kube-system namespace.
Viewing static Pods
To view static Pods, you can use the docker ps command if using Docker runtime or the crictl ps command if using ContainerD runtime. If the Kubernetes API server is available, you can also use the traditional kubectl get pods command on the master node to view the static Pods, but remember that the Pods you see in the Kubernetes cluster are just mirror Pods. When kubelet creates a static Pod, it also tries to create a mirror object in the Kubernetes cluster if the API server is available. Mirror Pods are essentially read-only Pods that are visible to the Kubernetes cluster but cannot be controlled by it. Instead, they are controlled and managed independently by the kubelet service running on the node.
How would you differentiate between a static Pod and a regular Pod? A quick way is to look at the Pod name, and if you see the node name appended to the Pod name, it is a static Pod. See the screenshot below for reference:
kubectl get pods -n kube-system
Distinguishing static Pods from regular Pods by looking at their names
The screenshot shows four Pods where the name of the control plane node (srv1.testlab.local) is appended to their names: etcd server, kube-apiserver, kube-controller-manager, and kube-scheduler. Notice that all the Pods are in the kube-system namespace, since these are the major control plane components for a Kubernetes cluster. In a highly available Kubernetes cluster, you might see different output. You can also use the kubectl describe pods command. If you see a Controlled By: Node/<node-name> field, as shown in the screenshot below, it is a static Pod.
kubectl describe pod etcd-kube-srv1.testlab.local -n kube-system
Distinguishing static Pods from regular Pods by looking at their properties
Determine the designated directory for kubelet
When you set up a Kubernetes cluster using Kubeadm, you install kubelet and a container runtime (Docker or ContainerD) on each node. The designated directory that we discussed above can be configured in multiple ways. You could pass the --pod-manifest-path option in the kubelet.service file to specify the designated directory. Alternatively, you could use the --config option in the kubelet.service file to specify the path of another configuration (YAML) file, which uses a staticPodPath field to define the designated directory.
The kubeadm tool uses the latter approach to define a designated directory. Since my Kubernetes cluster is configured using kubeadm, let's view the kubelet.service file.
sudo systemctl cat kubelet.service
Viewing the kubelet.service file on a node
You can see that there is no --pod-manifest-path option, but we can see a --config option that specifies the path of a YAML file. Let's view that YAML file and see if we can find the staticPodPath option in it.
sudo cat /var/lib/kubelet/config.yaml | grep staticPodPath
Finding out the designated directory for the kubelet service
You can see in the screenshot that the designated directory is defined using staticPodPath: /etc/kubernetes/manifests. Let's take a look at this directory.
ls -l /etc/kubernetes/manifests
Viewing the static Pod manifests in the designated directory
This directory contains the definition files for the control plane components that we saw above. You can use this procedure to determine the designated directory on each worker node and start defining your own static Pods. This allows you to run system daemons or agents (such as log collectors or metrics agents) on each node independently of the Kubernetes cluster.
Create a static Pod
Let's assume you deployed a business-critical web app in your Kubernetes cluster. Now, you want to run a logging agent on each worker node to monitor your web app. A static Pod is perfect for deploying such logging agents. To deploy the monitoring agent, create a Pod definition file named webapp-logger.yaml in the designated directory (/etc/kubernetes/manifests, in our case) on worker nodes (kube-srv2 and kube-srv2).
apiVersion: v1
kind: Pod
metadata:
name: webapp-logger
namespace: kube-system
spec:
containers:
- name: webapp-logger-container
image: harbor.testlab.local/library/webapp-logger:latest
#other options
Creating a static Pod definition file in the designated directory on worker nodes
The screenshot shows that I created a static Pod definition file in the designated directory on the kube-srv2 worker node. You could copy the same Pod definition file to other worker nodes in your Kubernetes cluster. As soon as kubelet detects a new definition file, it spins up a new static Pod based on the settings you defined in the file.
To view the static Pods on the worker node itself, you can use the docker ps command. If you are using ContainerD runtime instead of Docker, you can use the crictl ps command. Remember, there is no kubectl on the worker nodes, so you have to use the native commands that are available with the container runtime.
Viewing the static Pod on a worker node
To view the mirror Pod in the Kubernetes cluster, you can use the kubectl get pods -all-namespaces command.
Viewing the mirror Pod on the control plane master node
The static Pod is also visible as a mirror object in the Kubernetes cluster. Notice that the worker node name is appended to the Pod name. The kubelet service running on the worker node is solely responsible for managing this static Pod. If the Pod crashes for any reason, kubelet will restart it. If you delete the Pod manually (either by running the docker rm or kubectl delete pods command), kubelet will recreate it automatically.
The only way to modify or delete this static Pod is by modifying or deleting the original Pod definition file on the worker node itself. To delete the static Pod, simply remove the Pod definition file (/etc/kubernetes/manifests/webapp-logger.yaml) from the worker node, as shown in the screenshot below.
Subscribe to 4sysops newsletter!
sudo rm -f /etc/kubernetes/manifests/webapp-logger.yaml
Deleting a static Pod from the worker node
Conclusion
By creating a static Pod, you made sure that every node has a logging agent running all the time, even if the node is disconnected from the Kubernetes cluster or the API server is unavailable. This was a fairly simple use case for static Pods. You can use static Pods to deploy essential system services across all nodes of the Kubernetes cluster. This approach provides independent management, consistency, and simplicity for deploying critical services that need to run on every node.
