Kubernetes is an open-source system for running and managing containerized applications across physical machines, virtual machines, and cloud instances.
The reason it exists is simple: once an application is split into many microservices, containers become easy to create but hard to operate. Kubernetes gives us a consistent way to deploy them, recover from failures, scale them, and keep the whole system available.
This note is both a conceptual map of Kubernetes and a record of my first high-availability cluster setup with kubeadm, HAProxy, containerd, and Cilium.
What Kubernetes Tries to Solve
A production system usually needs three things:
- High availability: the application should keep running when one machine or process fails.
- Scalability: the system should support adding or removing workload capacity.
- Disaster recovery: important state should survive node or process failures.
Kubernetes does not make these problems disappear, but it gives us the primitives to manage them explicitly.
Core Building Blocks
Node
A Node is a physical machine, virtual machine, or cloud instance that runs Kubernetes workloads.
Pod
A Pod is the smallest deployable unit in Kubernetes. It wraps one or more containers. In many common applications, one Pod roughly maps to one application container.
Each Pod gets its own IP address, but that IP is not something we should treat as permanent.
Service
Because Pod IP addresses are temporary, Kubernetes uses a Service to provide a stable network endpoint in front of one or more Pods. The lifecycle of a Service is independent from the lifecycle of any individual Pod.
Ingress
An Ingress routes external HTTP or HTTPS requests to internal Services. It is the layer that lets different hostnames or paths reach different services inside the cluster.
ConfigMap and Secret
A ConfigMap stores non-sensitive configuration, such as a database URL or application setting. This avoids rebuilding a container image every time configuration changes.
A Secret is used for sensitive values such as passwords, tokens, and API keys. These should not be placed in a ConfigMap.
Volume
A Pod is disposable. If a Pod is deleted, local data inside that Pod disappears with it. A Volume makes data persistent by connecting the Pod to local or remote storage.
This is especially important for stateful systems such as databases.
Deployment and StatefulSet
A Deployment manages stateless Pods. It creates them, keeps the desired number running, replaces failed Pods, and supports rolling updates.
A StatefulSet manages Pods that need stable identity and persistent storage, such as PostgreSQL, Redis, or other stateful systems. The key difference is that Kubernetes must preserve the relationship between a Pod and its storage.
Namespace
A Namespace is a logical space inside a Kubernetes cluster. It helps isolate and organize resources so that different applications, environments, or teams do not interfere with each other.
Namespaces are useful for:
- Structuring cluster resources.
- Avoiding naming conflicts between teams.
- Separating environments while still allowing selected services to be shared.
- Applying access control and resource limits at the namespace level.
Worker Node Processes
A worker node mainly needs three components:
- kubelet: manages Pods on the node and makes the actual state match what Kubernetes wants.
- container runtime: runs containers. In this setup, I used
containerd. - kube-proxy: handles service networking. In many production setups, this responsibility is increasingly handled by Cilium.
Control Plane Processes
The control plane manages the desired state of the cluster and coordinates worker nodes.
Its core components are:
- API Server: the entry point of the entire Kubernetes control plane.
- Scheduler: decides which node should run a newly created Pod.
- Controller Manager: watches the API Server and continuously reconciles actual state with desired state.
- etcd: the database that stores cluster state.
In a production-like setup, there are usually three control plane nodes. The API Server instances can all serve requests, typically behind a load balancer. For the Scheduler and Controller Manager, only one instance acts as the leader at a time while the others remain on standby.
Each control plane node also runs etcd, and the etcd members synchronize state with one another as a distributed database.
For user-side traffic, the path is different. A request usually enters through an external load balancer, reaches a Kubernetes LoadBalancer Service and Ingress Controller, and is then routed to internal Services.
My Lab Environment
I built the cluster locally with:
- macOS
- UTM
- Ubuntu Server 24.04 LTS
- ARM64 virtual machines
| Role | CPU | Memory | Disk |
|---|---|---|---|
| HAProxy | 2 | 2 GB | 20 GB |
| Control Plane | 2 | 4 GB | 30 GB |
| Worker | 2 | 4 GB | 30 GB |
The final target topology was:
Build Steps
1. Install Ubuntu
For each VM, I installed Ubuntu Server without a GUI and enabled OpenSSH Server.
sudo apt update
sudo apt upgrade -y
2. Set Hostnames
sudo hostnamectl set-hostname <hostname>
Example names:
lb1
cp1
cp2
cp3
worker1
worker2
3. Install containerd
sudo apt install -y containerd
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
In /etc/containerd/config.toml, set:
SystemdCgroup = true
Then restart and enable containerd:
sudo systemctl restart containerd
sudo systemctl enable containerd
4. Disable Swap
sudo swapoff -a
Then comment out the swap entry in /etc/fstab so it does not return after reboot.
5. Configure Kernel Modules and sysctl
Load these modules in /etc/modules-load.d/k8s.conf:
overlay
br_netfilter
Enable the required networking settings:
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
Apply them:
sudo sysctl --system
6. Install Kubernetes Tools
Install kubeadm, kubelet, and kubectl, then enable kubelet:
sudo systemctl enable kubelet
7. Clone Virtual Machines
After preparing the base VM, I cloned it to create cp1, cp2, cp3, worker1, and worker2.
For each cloned VM, I updated the hostname and machine ID.
8. Configure HAProxy
On the load balancer node:
sudo apt update
sudo apt install -y haproxy
Example configuration:
global
log /dev/log local0
daemon
defaults
log global
mode tcp
timeout connect 10s
timeout client 1m
timeout server 1m
frontend kubernetes-api
bind *:6443
mode tcp
option tcplog
default_backend kubernetes-control-plane
backend kubernetes-control-plane
mode tcp
balance roundrobin
option tcp-check
server cp1 192.168.2.191:6443 check
server cp2 192.168.2.192:6443 check
server cp3 192.168.2.193:6443 check
Validate and reload:
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy
During bootstrap, I only added cp1 to the backend first. After cp2 and cp3 successfully joined the cluster, I added them back into HAProxy.
9. Initialize the Cluster
On the first control plane node:
sudo kubeadm init --control-plane-endpoint="<LB-IP>:6443" --upload-certs
Save the generated certificate key, control-plane join command, and worker join command.
10. Configure kubectl
mkdir -p ~/.kube
sudo cp /etc/kubernetes/admin.conf ~/.kube/config
sudo chown $$(id -u):$$(id -g) ~/.kube/config
Repeat this on every control-plane node where kubectl is needed.
11. Install Cilium
cilium install
cilium status
12. Join Additional Control Plane Nodes
Generate or refresh the certificate key and join command:
sudo kubeadm init phase upload-certs --upload-certs
kubeadm token create --print-join-command
On cp2 and cp3:
sudo kubeadm join <LB-IP>:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash> --control-plane --certificate-key <certificate-key>
13. Join Worker Nodes
On each worker node:
sudo kubeadm join <LB-IP>:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>
14. Validate the Cluster
kubectl get nodes
kubectl get pods -A
cilium status
The Most Important Lesson
Do Not Add Unjoined Control Planes to HAProxy Too Early
The most confusing issue I hit was not caused by Cilium, even though it first looked like a Cilium problem.
After installing Cilium, this command:
cilium status --wait
returned errors like:
unable to retrieve cilium status
error sending request
EOF
kubectl also became unstable:
Unable to connect to the server: EOF
TLS handshake timeout
Some Kubernetes API requests succeeded, while others failed.
Root Cause
I had configured HAProxy with all three control plane nodes before cp2 and cp3 had actually joined the cluster.
HAProxy
├── cp1 ✅
├── cp2 ❌
└── cp3 ❌
At that point, only cp1 was serving the Kubernetes API. cp2 and cp3 did not yet have working API Server instances. HAProxy still routed some requests to them, which caused intermittent API failures.
Because the Cilium CLI talks to the Kubernetes API Server, the symptom appeared during Cilium installation. The real problem was the unstable API endpoint.
Fix
Bootstrap the cluster incrementally.
Start with only cp1 in HAProxy:
HAProxy
└── cp1
After cp2 joins successfully:
HAProxy
├── cp1
└── cp2
After cp3 becomes healthy:
HAProxy
├── cp1
├── cp2
└── cp3
After every HAProxy change, validate and reload:
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy
Takeaway
Building Kubernetes from scratch makes the control plane feel much less magical. The key mental model is that Kubernetes is a reconciliation system built around a stable API Server and a persistent state store.
In a high-availability setup, the load balancer in front of the API Server is part of the control plane reliability story. If that endpoint is unstable, everything above it becomes noisy: kubectl, Cilium, node joins, and even basic health checks.
The practical rule I will remember is simple: during bootstrap, only route traffic to control plane nodes that are already serving the Kubernetes API successfully.