Are you ready for some Kubernetes fun? You know a few minutes on a Kubernetes tutorial wouldn’t be a waste.
Alright, let’s start with a Kubernetes buzzword – Pods.
Pods are the most granular resource you can manage in a Kubernetes cluster by using Kubernetes control plane. Pods can contain one or more containers, where one will deploy applications. You may also mount storage to your pods. And the pods have their own ephemeral IPs. And more. But let’s not get lost in the details of a Pod definition. Our goal is to understand what Static Pods are.
First, let’s take a look at a simple example of creating a pod using kubectl.
root:/home/ubuntu/pods# cat test-pod.yaml
apiVersion: v1
kind: Pod
metadata:
labels:
run: test-pod
name: test-pod
spec:
containers:
- args:
- sleep
- "3600"
image: busybox
name: test-pod
root:/home/ubuntu/pods# kubectl apply -f test-pod.yaml
pod/test-pod created
root:/home/ubuntu/pods# kubectl get pods
NAME READY STATUS RESTARTS AGE
test-pod 1/1 Running 0 20sNothing fancy going on here – a pod resource was created using a simple yaml file which has the pod specifications.
But what is not obvious is that kubectl has interacted with Kubernetes API Server (also referred to as Kube-apiserver) which in turn communicates with a kube-scheduler component to find the best node to create the pod resource on and then instructs a kubelet agent sitting on that particular node to create the pod. Then all the task details are stored in an etcd database. This is a typical workflow that happens mostly.
By the way, a couple of ways to validate me 😉
Subscribe to my YouTube channel
The components like kube-apiserver, kube-scheduler, kube-controller-manager, and etcd database together form the control plane of a Kubernetes cluster.
While the (normal) pods are created in response to the instruction from kube-apiserver by kubelet, kubelet itself can create and manage pods on its own. The type of pods that are managed by a kubelet is called static pods.
Kubelet Service
To understand further, we will start by looking at how a kubelet service is configured. Then we will see how to get a kubelet agent to create a static pod.
My test/lab Kubernetes cluster runs on Ubuntu operating system. I believe any linux distribution should look pretty similar to what we are going to see here. Just to be clear, we are observing one of my worker nodes, where Kubernetes resources are commonly created.
root:/home/ubuntu# ps -ef|grep kubelet
root 820 1 2 May21 ? 00:12:57 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --hostname-override=node1 --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.2/var/lib/kubelet/config.yaml file is particular important because it contains a line that will help us to create static pods. More specifically, a key-value pair. I am referring to staticPodPath highlighted below.
root:/home/ubuntu# cat /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 0s
cacheUnauthorizedTTL: 0s
cgroupDriver: cgroupfs
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0sOne way to create a static pod
We can simply create a pod yaml file, which has the static pod definition, in the directory ‘/etc/kubernetes/manifests‘. Kubelet will create the static pod shortly. Let’s try that.
# list the pods prior to creating a pod yaml file
root:/etc/kubernetes/manifests# kubectl get pods
NAME READY STATUS RESTARTS AGE
test-pod 1/1 Running 0 21m
# this is the file I created in the directory we found before.
root:/etc/kubernetes/manifests# cat static-pod.yaml
apiVersion: v1
kind: Pod
metadata:
labels:
run: static-pod
name: static-pod
spec:
containers:
- args:
- sleep
- "3600"
image: busybox
name: static-pod
# static pod gets created. kubelet appends the node name to the pod name. Interesting.
root:/etc/kubernetes/manifests# kubectl get pods
NAME READY STATUS RESTARTS AGE
static-pod-node1 1/1 Running 0 26s
test-pod 1/1 Running 0 21mWhat if we removed the yaml file in the manifests directory?
root:/etc/kubernetes/manifests# rm static-pod.yaml
root:/etc/kubernetes/manifests# kubectl get pods
NAME READY STATUS RESTARTS AGE
static-pod-node1 1/1 Running 0 8m51s
test-pod 1/1 Running 0 30m
root:/etc/kubernetes/manifests# kubectl get pods
NAME READY STATUS RESTARTS AGE
test-pod 1/1 Running 0 30mWell, the static pod gets removed automatically.
Another way to create a static pod
Alternatively, we can create the pod yaml file in a different directory as shown below.
root:/etc/kubelet.d# cat static-pod.yaml
apiVersion: v1
kind: Pod
metadata:
labels:
run: static-pod
name: static-pod
spec:
containers:
- args:
- sleep
- "3600"
image: busybox
name: static-podIf we observe the status of kubelet service using a systemd command, we will see an additional configuration file being used from a Drop-In directory. I think of this directory as a place that contains the override configuration of particularly kubelet service.
root@ip-172-31-27-42:/etc/kubelet.d# systemctl status kubelet.service
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Sat 2022-05-21 19:02:44 UTC; 10h ago
Docs: https://kubernetes.io/docs/home/Let’s dig this file – /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
root:/etc/kubelet.d# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGSLet’s add an extra argument to “/etc/default/kubelet” file such that kubelet will be started with a new manifest path. And restart kubelet service. You will see ‘static-pod-node1’ gets created.
#Do this on a worker node
#Add new pod manifest path
root:/etc/kubelet.d# cat /etc/default/kubelet
KUBELET_EXTRA_ARGS="--pod-manifest-path=/etc/kubelet.d/"
#list pods
root:/etc/kubelet.d# kubectl get pods
NAME READY STATUS RESTARTS AGE
test-pod 1/1 Running 0 47m
#restart kubelet service
root:/etc/kubelet.d# systemctl restart kubelet.service
root:/etc/kubelet.d# kubectl get pods
NAME READY STATUS RESTARTS AGE
static-pod-node1 1/1 Running 0 14s
test-pod 1/1 Running 0 52m
#Observe kubelet was restarted with the new argument you added.
root:/etc/kubelet.d# ps -ef|grep kubelet
root 20820 1 13 06:13 ? 00:00:00 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --hostname-override=node1 --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.2 --pod-manifest-path=/etc/kubelet.d/By the way, do you know the control plane components we looked at before run as pods as well in a namespace called kube-system.
root:/home/ubuntu# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
etcd-master 1/1 Running 8 (5d11h ago) 53d
kube-apiserver-master 1/1 Running 8 (5d11h ago) 53d
kube-controller-manager-master 1/1 Running 8 (5d11h ago) 53d
kube-scheduler-master 1/1 Running 8 (5d11h ago) 53dAt this point we understand the normal pods are created by kubelet as per the instruction from kube-apiserver. But then who told kubelet to create kube-apiserver pod? Obviously it would not have been kube-apiserver, because it didn’t exist.
Well, kube-apiserver and other control plane components’ pods are indeed created as static pods by kubelet. Because during the installation, the YAML files of these components are created in /etc/kubernetes/manifests directory.
root:/home/ubuntu/pods# ls -l /etc/kubernetes/manifests/
total 16
-rw------- 1 root root 2209 Mar 29 14:49 etcd.yaml
-rw------- 1 root root 3854 Mar 29 14:49 kube-apiserver.yaml
-rw------- 1 root root 3251 Mar 29 14:49 kube-controller-manager.yaml
-rw------- 1 root root 1435 Mar 29 14:49 kube-scheduler.yamlNow you know why we need static pods.
