🏠 ☸️Vagrant-KubeADM

Every VM is an ordinary Debian box with SSH and apt, and every step the scripts take is a kubeadm command you could type yourself; §5 shows exactly which ones. What the repo adds is the error-prone part: the VIP that must exist before kubeadm init, the node-ip every Vagrant lab gets wrong, the containerd 2.x config, the certificate SANs you cannot add later.

git clone --recurse-submodules https://github.com/OPS-NC/Vagrant-kubeadm.git
cd Vagrant-kubeadm
cp lab.env.example lab.env      # pick the topology
vagrant up                      # creates and PREPARES the VMs (no cluster yet)
./kubeadm/cluster-up.sh         # kubeadm init + join + kubeconfig
./_k8s/platform-up.sh           # CNI, Envoy Gateway, metrics-server, wildcard TLS

⚠️ --recurse-submodules is not optional. _k8s/ is a git submodule; a plain git clone leaves it empty and ./_k8s/platform-up.sh returns No such file or directory. On a clone already made: git submodule update --init --recursive.

ℹ️ There is a twin lab, Vagrant-Talos: same IP plan, same application layer, opposite operating model: Talos is immutable, has no SSH and no package manager, and is driven entirely through an API. Here you get a normal distribution and you drive kubeadm yourself: more moving parts, and seeing them work is the point of the lab.

🧰 1. Prerequisites (on the host)#

That is the whole list: no cluster-specific binary on your machine. kubeadm, kubelet, kubectl and containerd live inside the VMs, installed by kubeadm/provision.sh during vagrant up. The bento/debian-13 box is downloaded by Vagrant on first use; no plugin required.

Managing the submodule:

git submodule update --init --recursive     # fills _k8s/ on an existing clone
git submodule update --remote _k8s          # move it to the latest upstream commit

⚠️ git pull does not update the submodule. It moves this repo only, leaving _k8s/ on the commit pinned before, so you would run the documented commands against an older application layer. git status showing modified: _k8s (new commits) just means the checkout no longer matches the pin.

⚠️ VirtualBox and KVM cannot share VT-x. With the KVM module loaded, vagrant up dies on VERR_VMX_IN_VMX_ROOT_MODE. Unload it first (sudo modprobe -r kvm_intel kvm, or kvm_amd). See TROUBLESHOOTING.md.

💡 Keep host kubectl within one minor of the cluster (1.35 → 1.37 for a 1.36 cluster), or fall back to the in-VM one: vagrant ssh k8s-cp1 -c 'kubectl get nodes -o wide'.

🗺️ 2. IP plan (host-only network 192.168.56.0/24)#

Pod network 10.244.0.0/16, Service network 10.96.0.0/12. Node IPs are static, assigned by the Vagrantfile; it refuses a node IP landing on .1, .2, .100 or on the VIP, and refuses duplicates.

Every VM has 2 NICs: NIC1 = VirtualBox NAT (Internet, 10.0.2.15 on every VM) and NIC2 = host-only 192.168.56.x (cluster, API, etcd, pods). The default route goes through the NAT so the VMs can reach apt and the registries; what must be host-only is the node's identity, never its default route (see node-ip in §8).

ℹ️ The host-only interface name is never hard-coded. Debian 13 usually names it enp0s8, some box builds still give eth1. provision.sh finds the interface carrying the node's IP, writes it to /etc/kubeadm-lab/node.env, and cluster-up.sh copies it into _out/cluster.env as HOSTONLY_IF. keepalived binds VRRP to it and Cilium announces LoadBalancer IPs on it.

ℹ️ Name resolution depends on neither DNS nor boot order: the Vagrantfile pushes an identical /etc/hosts block to every node, and provision.sh deletes Debian's 127.0.1.1 <hostname> line. Left in place, it makes the kubelet resolve its own name to loopback, and the node registers as unreachable.

⚙️ 3. Pick the topology — lab.env#

lab.env is the single source read by the Vagrantfile, by kubeadm/cluster-up.sh and by the _k8s/*-up.sh scripts. Copy the versioned template (lab.env itself is gitignored):

cp lab.env.example lab.env

Format is strict: one KEY=value per line, no spaces around =. A real environment variable always wins, so one-off overrides work: WORKERS=5 vagrant up.

Two more are read by cluster-up.sh without being in the template: OUT (_out) and WAIT_API (600, seconds to wait for the apiserver on the VIP).

What each topology costs. Default (1 CP + 2 workers): 7 GB of RAM, 6 vCPU. Full HA (CONTROL_PLANES=3, WORKERS=3): 3 × 3072 + 3 × 2048 = 15.4 GB, 12 vCPU. Disks are linked clones, so the box is stored roughly once.

Three constraints worth knowing before you edit:

• Control planes must be odd — the Vagrantfile and cluster-up.sh both refuse an even number. etcd holds quorum at (n/2)+1: 2 members cost twice one CP and tolerate zero failures.
• CP_MEM ≥ 3072. kubeadm's preflight demands ~1700 MiB, so 2048 passes and then starves the stacked etcd as soon as addons pile up. _k8s/observability/ wants 4096.
• K8S_VERSION and K8S_APT_MINOR must agree. pkgs.k8s.io repositories are per-minor, and a mismatch fails in apt with an error that never mentions it. That pair is what you bump for an upgrade; see kubeadm/UPGRADE.md.

🚀 4. Start the cluster#

vagrant up                      # VMs + packages + containerd + kubeadm + keepalived
./kubeadm/cluster-up.sh         # init + joins + kubeconfig

vagrant up bootstraps nothing. It creates the VMs and runs provision.sh in each, which lays down, in order: /etc/hosts · swap off + kernel modules + sysctl · base packages (conntrack, socat, ethtool, open-iscsi, nfs-common…) · containerd with SystemdCgroup = true · kubelet/kubeadm/kubectl pinned and held · pre-pulled images · and, on control planes, keepalived carrying the VIP. Each VM ends ready to receive a kubeadm init or join, and nothing more.

cluster-up.sh then prints five steps:

Before touching anything it validates the config and checks that all expected VMs are running: cheap up front, versus a vagrant ssh timing out mid-join.

export KUBECONFIG="$PWD/kubeconfig"
kubectl get nodes -o wide

The kubeconfig needs no editing: its server: is the VIP, reachable from the host.

⚠️ The nodes will be NotReady, and that is normal. kubeadm never installs a CNI. Until a pod network exists the kubelet reports cni plugin not initialized, CoreDNS stays Pending and the nodes stay NotReady. The fix is the next command: ./_k8s/platform-up.sh (§6).

💡 cluster-up.sh is idempotent. node-init.sh refuses to re-run kubeadm init if /etc/kubernetes/admin.conf exists, node-join.sh skips a node that already has kubelet.conf. Re-running it is also how you grow the lab (§7.1). Join credentials are regenerated on every run, because the bootstrap token expires after 24 h and the certificate key after 2 h, so a run three days later just works.

For another topology, edit lab.env, or override on the spot for both commands, since each re-reads its own environment:

CONTROL_PLANES=3 WORKERS=3 vagrant up
CONTROL_PLANES=3 WORKERS=3 ./kubeadm/cluster-up.sh

🎓 5. Doing it by hand#

This is what the lab is for. The scripts exist so you do not retype these commands at every rebuild, not to hide them. Below is the same path by hand, on a lab that has been vagrant up-ed.

5.1 What vagrant up already left you#

vagrant ssh k8s-cp1
sudo -i
kubeadm version -o short                 # v1.36.3, held by apt-mark
containerd --version                     # 2.x when CONTAINERD_SOURCE=docker
crictl ps                                # talks to /run/containerd/containerd.sock
ip -4 addr show | grep 192.168.56.5      # the VIP is ALREADY there, before any init
cat /etc/kubeadm-lab/node.env            # NODE_IP, HOSTONLY_IF, VIP…

The VIP being up before kubeadm init is the whole reason keepalived is used here rather than kube-vip (§8.1).

5.2 kubeadm init on the first control plane#

The repo's way, with cluster-up.sh having already rendered the config into _out/, visible from the VM through the synced folder:

sudo kubeadm init --config /vagrant/_out/kubeadm-init.yaml --upload-certs \
     --skip-phases=addon/kube-proxy          # only when KUBE_PROXY_REPLACEMENT=true

The flag-only equivalent, to see it without a config file:

sudo kubeadm init \
  --control-plane-endpoint 192.168.56.5:6443 \
  --apiserver-advertise-address 192.168.56.10 \
  --pod-network-cidr 10.244.0.0/16 \
  --service-cidr 10.96.0.0/12 \
  --cri-socket unix:///run/containerd/containerd.sock \
  --apiserver-cert-extra-sans 192.168.56.5,192.168.56.10,192.168.56.20,192.168.56.30 \
  --upload-certs \
  --skip-phases=addon/kube-proxy

The two addresses are not the same thing: --apiserver-advertise-address is the real IP this apiserver listens on, --control-plane-endpoint is the shared VIP baked into the certificates and into every kubeconfig.

⚠️ The flag form cannot set node-ip, which is why the repo uses --config. With flags alone the kubelet picks the default-route NIC: the NAT, 10.0.2.15, identical on every VM. Every node then registers with the same address: kubectl get nodes -o wide looks plausible while logs, exec, probes and inter-node traffic go to the wrong place. The setting only exists as nodeRegistration.kubeletExtraArgs.

Two more things that cannot be fixed afterwards: --upload-certs stores the cluster CAs in the kubeadm-certs Secret (without it, a second control plane can only join after you copy /etc/kubernetes/pki by hand), and certSANs, which need regenerating the API certificate to change, hence the 5 control-plane IPs declared up front, including nodes that do not exist yet.

5.3 Joining nodes#

# on the control plane — prints a ready-to-paste command, token valid 24 h
sudo kubeadm token create --print-join-command
# on the worker
sudo kubeadm join 192.168.56.5:6443 --token <t> --discovery-token-ca-cert-hash sha256:<h>

A second control plane needs two more ingredients: --control-plane and the certificate key, which decrypts the kubeadm-certs Secret.

# on cp1 — re-encrypts the Secret and prints a NEW key on the last line
sudo kubeadm token create --print-join-command \
  --certificate-key "$(sudo kubeadm init phase upload-certs --upload-certs | tail -n1)"

Four things bite here:

• That printed join line is exactly what this lab does not use. It cannot carry node-ip (§5.2), so a node joined this way registers with 10.0.2.15. The repo renders a JoinConfiguration file instead and runs kubeadm join --config /vagrant/_out/join-<node>.yaml. Every node sharing one INTERNAL-IP is this, every time.
• The certificate key expires after 2 hours, the token after 24. Both are cheap to regenerate; a stale one gives a decryption error that never mentions expiry.
• --config and --certificate-key are mutually exclusive. With a config file the key goes under controlPlane.certificateKey, not at the document root, unlike InitConfiguration.
• Join control planes one at a time. Each join adds an etcd member, and etcd accepts a single membership change at a time; two in parallel fail on an unreadable quorum error.

Getting a kubeconfig needs no scp: the synced folder is right there, and server: already points at the VIP:

vagrant ssh k8s-cp1 -c 'sudo cat /etc/kubernetes/admin.conf' > kubeconfig
chmod 0600 kubeconfig && export KUBECONFIG="$PWD/kubeconfig"

📦 6. What comes next: the application layer#

A bare cluster does nothing useful; here it is not even Ready. Cilium, Envoy Gateway, cert-manager, metrics-server, Longhorn, Vault, CloudNativePG, Prometheus/Loki, Kyverno, Trivy, MinIO, Argo CD… all come from k8s-playground, mounted here as _k8s/ and shared with the Talos twin. Its documentation is published separately: https://ops-nc.github.io/k8s-playground/.

./_k8s/platform-up.sh                       # CNI → Envoy Gateway → metrics-server → TLS
./_k8s/install.sh longhorn vault argocd     # opt-in addons
./_k8s/install.sh list                      # the full catalogue
./_k8s/install.sh all                       # platform + every addon, in dependency order
./_k8s/longhorn/longhorn-up.sh              # one addon on its own

Nothing to declare: the lab is the directory containing _k8s/ that carries the Vagrantfile (so lab.env, _out/ and kubeconfig are found there), and the distribution is read off its contents: a kubeadm/cluster-up.sh next to the Vagrantfile means the kubeadm lab. That works straight from the clone, before any vagrant up. An explicit ./_k8s/install.sh kubeadm platform, --distro=kubeadm or K8S_DISTRO still wins, and LAB_DIR is the escape hatch for an unusual layout; neither is needed here.

platform-up.sh installs the CNI first; the nodes go Ready a minute or two later.

⚠️ This layer assumes CNI=cilium (the default). It needs a LoadBalancer Service that really gets an IP, which on a host-only network only Cilium's L2/ARP announcement provides. Otherwise the Gateway stays at EXTERNAL-IP <pending> and no UI is reachable. See §9.

6.1 The two manual prerequisites#

Nothing in the cluster can do these for you.

a) Make *.<LAB_DOMAIN> resolve to the Gateway IP. Every lab UI is served through Envoy's LoadBalancer Service, which takes the first IP of LB_POOL_START (192.168.56.200 by default). With SELF_SIGNED=true an /etc/hosts line is enough and no public record is needed:

kubectl -n envoy-gateway-system get svc -o wide | grep LoadBalancer   # the actual IP
# /etc/hosts
# 192.168.56.200  argo.kubeadm.lab.example.io grafana.kubeadm.lab.example.io

With SELF_SIGNED=false you need a real wildcard A record *.<LAB_DOMAIN> → the Gateway IP, DNS-only (a CDN proxy cannot reach a private 192.168.56.x origin).

b) Choose the TLS mode with SELF_SIGNED. true: platform-up.sh builds a local CA and a wildcard with openssl: no cert-manager, no token, no public domain, and a browser warning until you import _out/self-signed/ca.crt. false: cert-manager + Let's Encrypt over ACME DNS-01, which needs a real domain, CLOUDFLARE_API_TOKEN, and respect for the 5 certificates per week production quota (LAB_ACME_ISSUER=staging is the default for that reason). Both paths fill the same wildcard-<LAB_DOMAIN with dashes>-tls Secret, so no addon has to know which one you picked.

♻️ 7. Lifecycle#

vagrant status                 # VM state
vagrant halt                   # power off (the cluster comes back on the next `up`)
vagrant up                     # power back on
vagrant destroy -f             # delete every VM
rm -rf _out kubeconfig         # clear host-side state before rebuilding

Keeping the repo current takes two commands, since git pull leaves _k8s/ where it was:

git pull
git submodule update --init --recursive   # _k8s/ back onto the commit this repo pins
git submodule update --remote _k8s        # or: jump to the latest k8s-playground

7.1 Growing the lab#

cluster-up.sh being idempotent is the procedure:

1. raise WORKERS (or CONTROL_PLANES, keeping it odd) in lab.env;
2. vagrant up — only the new VMs get created and provisioned;
3. ./kubeadm/cluster-up.sh — skips what is in place, joins the new nodes with fresh credentials.

No certificate regeneration: the certSANs already cover 5 control-plane IPs (§5.2).

Removing a worker means draining first, so the cluster stops scheduling onto a machine about to vanish:

kubectl drain k8s-w3 --ignore-daemonsets --delete-emptydir-data
vagrant destroy -f k8s-w3
kubectl delete node k8s-w3

then lower WORKERS in lab.env.

7.2 Undoing the cluster without destroying the VMs#

./kubeadm/cluster-reset.sh          # asks for confirmation
./kubeadm/cluster-reset.sh --yes    # unattended

It runs kubeadm reset on every node (workers first, so they deregister while the API still answers), then removes _out/ and kubeconfig. The VMs keep their packages, containerd and keepalived, so a rebuild is ./kubeadm/cluster-up.sh alone: minutes instead of a full vagrant up. Prefer it to vagrant destroy to replay a failed bootstrap, or to change POD_CIDR, SERVICE_CIDR, the CNI or the VIP: all four are frozen at kubeadm init.

⚠️ Destructive: etcd, the certificates and every workload are lost, PersistentVolumes on node disks included.

ℹ️ Why a dedicated reset. kubeadm reset deliberately leaves behind what it did not create: CNI interfaces, Cilium's pinned eBPF programs under /sys/fs/bpf (which survive the DaemonSet and keep intercepting traffic for a cluster that no longer exists), and kube-proxy's iptables rules. node-reset.sh cleans all of it; without that pass the next init inherits a ghost datapath and the pod network misbehaves with nothing in any log.

🔍 8. Design notes#

8.1 The VIP is carried by keepalived, not kube-vip#

The most structural decision in the repo. controlPlaneEndpoint points at the VIP and is frozen into the certificates and every kubeconfig at kubeadm init time, so the VIP must exist before the init.

kube-vip, the usual answer in kubeadm HA guides, runs as a static pod and elects its leader through the Kubernetes API, that is, through the very VIP it is supposed to carry. The documented way out is --k8sConfigPath /etc/kubernetes/super-admin.conf, itself fragile since Kubernetes 1.29 moved admin.conf out of system:masters (kube-vip#684, still open).

keepalived has none of that: a plain VRRP daemon, knows nothing about Kubernetes, brings the VIP up at VM boot. provision.sh configures it with unicast VRRP (multicast is the first thing to misbehave on a VirtualBox host-only switch, and every control-plane IP is known anyway), priorities cp1 = 100 / cp2 = 90 / cp3 = 80, and a vrrp_script polling https://127.0.0.1:6443/livez/ping every 3 s with weight -30, so a CP whose apiserver is dead drops to 70 and a healthy cp2 at 90 takes over. /livez/ping is readable anonymously via the system:public-info-viewer binding kubeadm creates, so no credential has to reach a health script. There is no authentication block: VRRPv2 sends its password in clear text and buys nothing here, the trust boundary being the host-only network. VRRP_ROUTER_ID is the knob to coexist with another keepalived lab.

While no cluster exists the check fails on every CP: they all lose 30 points, the relative order holds, and the VIP is carried anyway, which is what kubeadm init needs. kube-vip remains a good option once the cluster runs (--services mode); it is the bootstrap role that fails here.

The VIP is used even with a single control plane, for the same reason: pointing controlPlaneEndpoint at cp1's real IP would turn "1 CP → 3 CPs" into regenerating every certificate and redistributing every kubeconfig, instead of a plain join.

8.2 containerd 2.x from the Docker repo#

Debian 13 ships containerd 1.7.24. Only the 2.x branch implements the CRI RuntimeConfig method kubeadm uses to read the runtime's cgroup driver. On 1.36 its absence is a preflight warning; the fallback disappears in 1.37, and the backport to 1.7 was refused (containerd#11346, closed without merge). CONTAINERD_SOURCE=debian stays available for an offline lab, and is a dead end for upgrades.

SystemdCgroup = true matters more than the kubelet's cgroupDriver field: Debian 13 is cgroup v2 with systemd as the manager, and leaving containerd on cgroupfs puts two managers on one hierarchy, and nodes then get unstable under load.

⚠️ The 1.7 → 2.x trap: the pause image key changed name and location. Config v2 has sandbox_image under [plugins."io.containerd.grpc.v1.cri"]; config v3 has sandbox under [plugins.'io.containerd.cri.v1.images'.pinned_images]. A config copied over as-is silently loses the setting, so provision.sh regenerates it from containerd config default on every run and patches whichever key is present. The tag itself comes from kubeadm config images list, never hard-coded: a mismatch is invisible online and fatal offline.

8.3 kubeadm API v1beta4#

Default since Kubernetes 1.31; v1beta3 is deprecated. The breaking change to know: extraArgs and kubeletExtraArgs are no longer maps but lists of {name, value}, so a flag can be repeated. Any file written before 1.31 is invalid as-is, and kubeadm's error does not point at the shape.

# v1beta3:  extraArgs: {bind-address: "0.0.0.0"}
# v1beta4:  extraArgs: [{name: bind-address, value: "0.0.0.0"}]

make validate-kubeadm catches exactly this, in CI, without a cluster.

8.4 What kubeadm does not do, and cluster-up.sh does#

• Worker role labels — kubeadm sets none, so kubectl get nodes shows <none> and node-role.kubernetes.io/worker selectors match nothing.
• The control-plane taint: UNTAINT_CP=auto removes it only when WORKERS=0, which is what makes a 1-VM lab usable.
• Control-plane metrics: bind-address: 0.0.0.0 on controllerManager and scheduler, which otherwise listen on loopback and give Prometheus two DOWN targets with no explanation.
• Pre-pulled images, during vagrant up and in parallel across VMs, so kubeadm init downloads nothing, the biggest source of bootstrap timeouts. Workers pull only pause and kube-proxy, saving ~500 MiB each.
• Swap off and masked, systemd swap units included (/etc/fstab does not describe those).

The /vagrant synced folder is a mechanism, not a convenience: cluster-up.sh renders configs on the host and the VMs read them at /vagrant/_out/, so nothing needs scp and no secret is passed on a command line where it would land in shell history. _out/join.env does hold the bootstrap token and the certificate key, readable from every VM: fine for a lab, not a pattern for production.

🌐 9. CNI: Cilium, Calico or Flannel#

kubeadm installs no CNI, ever. Unlike the Talos twin (where flannel can be laid down by the OS at bootstrap), the pod network here is always installed afterwards by ./_k8s/platform-up.sh. CNI is read by cluster-up.sh (for the kube-proxy decision and _out/cluster.env) and by the platform step (which chart to install).

In practice: keep cilium. It is the only value that gives Services an EXTERNAL-IP on a host-only network, and therefore the only one that gets you the HTTPS UIs. calico is there to compare CNIs and work on NetworkPolicy (its page); flannel for a deliberately bare cluster.

⚠️ KUBE_PROXY_REPLACEMENT=true requires CNI=cilium, and cluster-up.sh refuses any other combination. With --skip-phases=addon/kube-proxy and no replacement, no ClusterIP answers at all, not even CoreDNS reaching the API. The error message offers the two ways out: CNI=cilium, or KUBE_PROXY_REPLACEMENT=false.

ℹ️ Cilium needs k8sServiceHost/k8sServicePort when kube-proxy is gone: nothing provisions the apiserver's ClusterIP, so the agent cannot bootstrap through kubernetes.default. The lab points it at the VIP, which also means the agents survive the loss of any single CP.

⚠️ POD_CIDR must be the CIDR the CNI really announces. Cilium in cluster-pool mode defaults to 10.0.0.0/8, unrelated to what kubeadm was told; cilium-up.sh passes POD_CIDR back to it explicitly. Two divergent values give a broken pod network that looks configured.

⚠️ Switching CNI on a live cluster is not supported. ./kubeadm/cluster-reset.sh (or vagrant destroy) first: two CNIs fight over the pod network, and the leftover datapath is exactly what node-reset.sh exists to clean.

🛠️ 10. Validating a change#

Everything validates without booting a cluster:

make validate       # shell + YAML + Vagrantfile + kubeadm templates + doc links
make docs           # regenerates docs/index.html from every README (EN + FR)
make help           # lists the targets

validate-kubeadm earns its keep: it catches a real v1beta4 schema error instead of letting you discover it ten minutes into a vagrant up. On CI, where kubeadm is installed, the schema check always runs.

The ci workflow calls these same make targets on every pull request, so a check cannot pass in CI and fail on your machine. It also asserts the guard rails actually fire: CONTROL_PLANES=2 vagrant validate must be rejected. Nothing in the Makefile touches a running cluster or regenerates secrets: make validate is safe on a lab that is up.

ℹ️ validate-shell and validate-yaml only cover files tracked by this repo. The _k8s/ submodule is one pointer, so none of its scripts are checked here; they are validated in k8s-playground's own CI.

📄 11. License#

Apache License 2.0. See LICENSE. Use it, modify it, redistribute it, including commercially, as long as you keep the copyright notice and state your changes. No warranty: this is a lab, do not run it in production.

It covers what this repo contains: the Vagrantfile, the kubeadm/ scripts, the templates, the manifests, the docs. It does not extend to the third-party components those scripts download (Kubernetes, containerd, keepalived, Cilium, Envoy Gateway, Longhorn, Vault…), nor to the _k8s/ submodule: k8s-playground carries its own LICENSE.

🏠 ☸️Vagrant-KubeADM

Chaque VM est une Debian ordinaire avec SSH et apt, et chaque étape des scripts est une commande kubeadm que tu pourrais taper toi-même ; le §5 montre exactement lesquelles. Ce que le dépôt ajoute, c'est la partie ingrate : la VIP qui doit exister avant kubeadm init, le node-ip que tous les labs Vagrant ratent, la config containerd 2.x, les SAN de certificat qu'on ne peut pas ajouter après coup.

git clone --recurse-submodules https://github.com/OPS-NC/Vagrant-kubeadm.git
cd Vagrant-kubeadm
cp lab.env.example lab.env      # choisir la topologie
vagrant up                      # crée et PRÉPARE les VM (aucun cluster encore)
./kubeadm/cluster-up.sh         # kubeadm init + join + kubeconfig
./_k8s/platform-up.sh           # CNI, Envoy Gateway, metrics-server, TLS wildcard

⚠️ --recurse-submodules n'est pas optionnel. _k8s/ est un sous-module git ; un git clone simple le laisse vide et ./_k8s/platform-up.sh répond No such file or directory. Sur un clone déjà fait : git submodule update --init --recursive.

ℹ️ Il existe un lab jumeau, Vagrant-Talos : même plan d'adressage, même couche applicative, modèle d'exploitation opposé : Talos est immuable, sans SSH ni gestionnaire de paquets, et se pilote entièrement par API. Ici tu as une distribution normale et tu conduis kubeadm toi-même : plus de pièces mobiles, et c'est ce qui rend le lab intéressant à lire.

🧰 1. Prérequis (sur l'hôte)#

C'est toute la liste : aucun binaire propre au cluster sur ta machine. kubeadm, kubelet, kubectl et containerd vivent dans les VM, installés par kubeadm/provision.sh pendant vagrant up. La box bento/debian-13 est téléchargée par Vagrant au premier usage ; aucun plugin nécessaire.

Gérer le sous-module :

git submodule update --init --recursive     # remplit _k8s/ sur un clone existant
git submodule update --remote _k8s          # le déplace sur le dernier commit amont

⚠️ git pull ne met pas le sous-module à jour. Il ne déplace que ce dépôt, _k8s/ reste sur le commit épinglé avant, et tu exécuterais les commandes documentées contre une couche applicative plus ancienne. Un git status qui affiche modified: _k8s (new commits) signifie juste que le checkout ne correspond plus à l'épingle.

⚠️ VirtualBox et KVM ne peuvent pas partager VT-x. Module KVM chargé, vagrant up meurt sur VERR_VMX_IN_VMX_ROOT_MODE. Décharge-le d'abord (sudo modprobe -r kvm_intel kvm, ou kvm_amd). Voir DEPANNAGE.md.

💡 Garde le kubectl de l'hôte à un minor près du cluster (1.35 → 1.37 pour un cluster 1.36), ou rabats-toi sur celui de la VM : vagrant ssh k8s-cp1 -c 'kubectl get nodes -o wide'.

🗺️ 2. Plan d'adressage (réseau host-only 192.168.56.0/24)#

Réseau des pods 10.244.0.0/16, réseau des Services 10.96.0.0/12. Les IP des nodes sont statiques, posées par le Vagrantfile ; il refuse une IP de node qui tombe sur .1, .2, .100 ou sur la VIP, et refuse les doublons.

Chaque VM a 2 cartes : NIC1 = NAT VirtualBox (Internet, 10.0.2.15 sur toutes les VM) et NIC2 = host-only 192.168.56.x (cluster, API, etcd, pods). La route par défaut passe par le NAT pour que les VM atteignent apt et les registres ; ce qui doit être host-only, c'est l'identité du node, jamais sa route par défaut (voir node-ip au §8).

ℹ️ Le nom de l'interface host-only n'est jamais codé en dur. Debian 13 la nomme habituellement enp0s8, certaines box donnent encore eth1. provision.sh trouve l'interface qui porte l'IP du node, l'écrit dans /etc/kubeadm-lab/node.env, et cluster-up.sh la recopie dans _out/cluster.env sous HOSTONLY_IF. keepalived y attache VRRP et Cilium y annonce les IP de LoadBalancer.

ℹ️ La résolution de noms ne dépend ni du DNS ni de l'ordre de démarrage : le Vagrantfile pousse un bloc /etc/hosts identique sur chaque node, et provision.sh supprime la ligne 127.0.1.1 <hostname> de Debian ; laissée en place, le kubelet résout son propre nom en loopback et le node s'enregistre comme injoignable.

⚙️ 3. Choisir la topologie — lab.env#

lab.env est la source unique lue par le Vagrantfile, par kubeadm/cluster-up.sh et par les scripts _k8s/*-up.sh. Copie le modèle versionné (lab.env est gitignoré) :

cp lab.env.example lab.env

Le format est strict : un KEY=value par ligne, pas d'espace autour du =. Une vraie variable d'environnement gagne toujours, ce qui rend les surcharges ponctuelles possibles : WORKERS=5 vagrant up.

Deux autres sont lues par cluster-up.sh sans figurer dans le modèle : OUT (_out) et WAIT_API (600, secondes d'attente de l'apiserver sur la VIP).

Ce que coûte chaque topologie. Par défaut (1 CP + 2 workers) : 7 Go de RAM, 6 vCPU. HA complète (CONTROL_PLANES=3, WORKERS=3) : 3 × 3072 + 3 × 2048 = 15,4 Go, 12 vCPU. Les disques sont des clones liés, donc la box est stockée à peu près une fois.

Trois contraintes à connaître avant d'éditer :

• Les control planes doivent être en nombre impair — le Vagrantfile et cluster-up.sh refusent tous les deux un nombre pair. etcd tient son quorum à (n/2)+1 : 2 membres coûtent deux fois un CP et ne tolèrent aucune panne.
• CP_MEM ≥ 3072. Le preflight de kubeadm exige ~1700 Mio : 2048 passe, puis affame l'etcd empilé dès que les addons s'accumulent. _k8s/observability/ demande 4096.
• K8S_VERSION et K8S_APT_MINOR doivent concorder. Les dépôts pkgs.k8s.io sont par minor, et l'écart échoue dans apt sur une erreur qui ne le mentionne jamais. C'est cette paire qu'on incrémente pour une montée de version ; kubeadm/MISE-A-JOUR.md.

🚀 4. Démarrer le cluster#

vagrant up                      # VM + paquets + containerd + kubeadm + keepalived
./kubeadm/cluster-up.sh         # init + jonctions + kubeconfig

vagrant up ne bootstrape rien. Il crée les VM et exécute provision.sh dans chacune, qui pose, dans l'ordre : /etc/hosts · swap coupé + modules noyau + sysctl · paquets de base (conntrack, socat, ethtool, open-iscsi, nfs-common…) · containerd avec SystemdCgroup = true · kubelet/kubeadm/kubectl épinglés et gelés · images pré-tirées · et, sur les control planes, keepalived portant la VIP. Chaque VM finit prête à recevoir un kubeadm init ou join, rien de plus.

cluster-up.sh affiche ensuite cinq étapes :

Avant de toucher à quoi que ce soit, il valide la config et vérifie que toutes les VM attendues sont running : une seconde en amont, contre un vagrant ssh qui expire au milieu d'un join.

export KUBECONFIG="$PWD/kubeconfig"
kubectl get nodes -o wide

Le kubeconfig ne demande aucune retouche : son server: est la VIP, joignable depuis l'hôte.

⚠️ Les nodes seront NotReady, et c'est normal. kubeadm n'installe jamais de CNI. Sans réseau de pods, le kubelet signale cni plugin not initialized, CoreDNS reste Pending et les nodes restent NotReady. Le remède est la commande suivante : ./_k8s/platform-up.sh (§6).

💡 cluster-up.sh est idempotent. node-init.sh refuse de rejouer kubeadm init si /etc/kubernetes/admin.conf existe, node-join.sh saute un node qui a déjà kubelet.conf. Le relancer est aussi la manière d'agrandir le lab (§7.1). Les identifiants de jonction sont régénérés à chaque exécution, parce que le token expire au bout de 24 h et la clé de certificats au bout de 2 h ; un lancement trois jours plus tard fonctionne donc directement.

Pour une autre topologie, édite lab.env, ou surcharge sur place pour les deux commandes, chacune relisant son propre environnement :

CONTROL_PLANES=3 WORKERS=3 vagrant up
CONTROL_PLANES=3 WORKERS=3 ./kubeadm/cluster-up.sh

🎓 5. Faire la même chose à la main#

C'est la raison d'être du lab. Les scripts existent pour ne pas retaper ces commandes à chaque reconstruction, pas pour les cacher. Voici le même parcours à la main, sur un lab déjà vagrant up.

5.1 Ce que vagrant up t'a déjà laissé#

vagrant ssh k8s-cp1
sudo -i
kubeadm version -o short                 # v1.36.3, gelé par apt-mark
containerd --version                     # 2.x quand CONTAINERD_SOURCE=docker
crictl ps                                # parle à /run/containerd/containerd.sock
ip -4 addr show | grep 192.168.56.5      # la VIP est DÉJÀ là, avant tout init
cat /etc/kubeadm-lab/node.env            # NODE_IP, HOSTONLY_IF, VIP…

La VIP debout avant kubeadm init est toute la raison pour laquelle keepalived est utilisé ici plutôt que kube-vip (§8.1).

5.2 kubeadm init sur le premier control plane#

La façon du dépôt : cluster-up.sh a déjà rendu la config dans _out/, visible depuis la VM par le dossier synchronisé :

sudo kubeadm init --config /vagrant/_out/kubeadm-init.yaml --upload-certs \
     --skip-phases=addon/kube-proxy          # seulement si KUBE_PROXY_REPLACEMENT=true

L'équivalent en options seules, pour le voir sans fichier de config :

sudo kubeadm init \
  --control-plane-endpoint 192.168.56.5:6443 \
  --apiserver-advertise-address 192.168.56.10 \
  --pod-network-cidr 10.244.0.0/16 \
  --service-cidr 10.96.0.0/12 \
  --cri-socket unix:///run/containerd/containerd.sock \
  --apiserver-cert-extra-sans 192.168.56.5,192.168.56.10,192.168.56.20,192.168.56.30 \
  --upload-certs \
  --skip-phases=addon/kube-proxy

Les deux adresses ne sont pas la même chose : --apiserver-advertise-address est l'IP réelle sur laquelle cet apiserver écoute, --control-plane-endpoint est la VIP partagée gravée dans les certificats et dans chaque kubeconfig.

⚠️ La forme en options ne peut pas poser node-ip, d'où le --config du dépôt. Avec les seules options, le kubelet prend l'interface de la route par défaut : le NAT, 10.0.2.15, identique sur toutes les VM. Tous les nodes s'enregistrent alors avec la même adresse : kubectl get nodes -o wide paraît crédible pendant que les logs, exec, les sondes et le trafic inter-nodes partent au mauvais endroit. Le réglage n'existe que sous nodeRegistration.kubeletExtraArgs.

Deux autres choses irréparables après coup : --upload-certs stocke les AC du cluster dans le Secret kubeadm-certs (sans lui, un second control plane ne peut joindre qu'après une copie manuelle de /etc/kubernetes/pki), et les certSANs, qui exigent de régénérer le certificat de l'API pour changer, d'où les 5 IP de control plane déclarées d'emblée, y compris pour des nodes qui n'existent pas encore.

5.3 Joindre des nodes#

# sur le control plane — imprime une commande prête à coller, token valable 24 h
sudo kubeadm token create --print-join-command
# sur le worker
sudo kubeadm join 192.168.56.5:6443 --token <t> --discovery-token-ca-cert-hash sha256:<h>

Un second control plane demande deux ingrédients de plus : --control-plane et la clé de certificats, qui déchiffre le Secret kubeadm-certs.

# sur cp1 — rechiffre le Secret et imprime une NOUVELLE clé en dernière ligne
sudo kubeadm token create --print-join-command \
  --certificate-key "$(sudo kubeadm init phase upload-certs --upload-certs | tail -n1)"

Quatre choses mordent ici :

• Cette ligne de jonction imprimée est précisément ce que le lab n'utilise pas. Elle ne peut pas porter node-ip (§5.2), donc un node joint comme ça s'enregistre avec 10.0.2.15. Le dépôt rend un fichier JoinConfiguration à la place et lance kubeadm join --config /vagrant/_out/join-<node>.yaml. Tous les nodes avec la même INTERNAL-IP, c'est ça, chaque fois.
• La clé de certificats expire au bout de 2 heures, le token au bout de 24. Les deux se régénèrent pour rien ; une clé périmée donne une erreur de déchiffrement qui ne parle jamais d'expiration.
• --config et --certificate-key sont mutuellement exclusifs. Avec un fichier de config, la clé va sous controlPlane.certificateKey, pas à la racine du document, contrairement à InitConfiguration.
• Joins les control planes un par un. Chaque jonction ajoute un membre etcd, et etcd n'accepte qu'un changement d'appartenance à la fois ; deux en parallèle échouent sur une erreur de quorum illisible.

Récupérer un kubeconfig ne demande aucun scp : le dossier synchronisé est là, et server: pointe déjà la VIP :

vagrant ssh k8s-cp1 -c 'sudo cat /etc/kubernetes/admin.conf' > kubeconfig
chmod 0600 kubeconfig && export KUBECONFIG="$PWD/kubeconfig"

📦 6. La suite : la couche applicative#

Un cluster nu ne sert à rien ; ici il n'est même pas Ready. Cilium, Envoy Gateway, cert-manager, metrics-server, Longhorn, Vault, CloudNativePG, Prometheus/Loki, Kyverno, Trivy, MinIO, Argo CD… viennent tous de k8s-playground, monté ici en _k8s/ et partagé avec le jumeau Talos. Sa documentation est publiée à part : https://ops-nc.github.io/k8s-playground/.

./_k8s/platform-up.sh                       # CNI → Envoy Gateway → metrics-server → TLS
./_k8s/install.sh longhorn vault argocd     # addons opt-in
./_k8s/install.sh list                      # le catalogue complet
./_k8s/install.sh all                       # plateforme + tous les addons, dans l'ordre
./_k8s/longhorn/longhorn-up.sh              # un addon seul

Rien à déclarer : le lab est le dossier contenant _k8s/ qui porte le Vagrantfile (donc lab.env, _out/ et kubeconfig s'y trouvent), et la distribution se lit sur son contenu : un kubeadm/cluster-up.sh à côté du Vagrantfile signifie le lab kubeadm. Ça marche dès le clone, avant tout vagrant up. Un ./_k8s/install.sh kubeadm platform explicite, --distro=kubeadm ou K8S_DISTRO gagnent toujours, et LAB_DIR est la porte de sortie pour une arborescence inhabituelle ; aucun des deux n'est nécessaire ici.

platform-up.sh installe le CNI en premier ; les nodes passent Ready une à deux minutes après.

⚠️ Cette couche suppose CNI=cilium (le défaut). Elle a besoin d'un Service LoadBalancer qui obtienne réellement une IP, ce que seule l'annonce L2/ARP de Cilium fournit sur un réseau host-only ; sinon le Gateway reste en EXTERNAL-IP <pending> et aucune UI n'est joignable. Voir §9.

6.1 Les deux prérequis manuels#

Rien dans le cluster ne peut les faire à ta place.

a) Faire résoudre *.<LAB_DOMAIN> vers l'IP du Gateway. Toutes les UI du lab passent par le Service LoadBalancer d'Envoy, qui prend la première IP de LB_POOL_START : 192.168.56.200 par défaut. Avec SELF_SIGNED=true, une ligne /etc/hosts suffit et aucun enregistrement public n'est nécessaire :

kubectl -n envoy-gateway-system get svc -o wide | grep LoadBalancer   # l'IP réelle
# /etc/hosts
# 192.168.56.200  argo.kubeadm.lab.example.io grafana.kubeadm.lab.example.io

Avec SELF_SIGNED=false, il faut un vrai enregistrement A wildcard *.<LAB_DOMAIN> → l'IP du Gateway, en DNS-only (un proxy CDN ne peut pas joindre une origine privée 192.168.56.x).

b) Choisir le mode TLS avec SELF_SIGNED. true : platform-up.sh fabrique une AC locale et un wildcard avec openssl : pas de cert-manager, pas de token, pas de domaine public, et un avertissement du navigateur jusqu'à l'import de _out/self-signed/ca.crt. false : cert-manager

• Let's Encrypt en ACME DNS-01, ce qui demande un vrai domaine, CLOUDFLARE_API_TOKEN, et le respect du quota de production de 5 certificats par semaine (LAB_ACME_ISSUER=staging est le défaut pour cette raison). Les deux chemins remplissent le même Secret wildcard-<LAB_DOMAIN avec tirets>-tls, donc aucun addon n'a à savoir lequel tu as choisi.

♻️ 7. Cycle de vie#

vagrant status                 # état des VM
vagrant halt                   # extinction (le cluster revient au `up` suivant)
vagrant up                     # rallumage
vagrant destroy -f             # supprime toutes les VM
rm -rf _out kubeconfig         # nettoyer l'état côté hôte avant de reconstruire

Garder le dépôt à jour prend deux commandes, git pull laissant _k8s/ où il était :

git pull
git submodule update --init --recursive   # _k8s/ revient sur le commit épinglé ici
git submodule update --remote _k8s        # ou : sauter au dernier k8s-playground

7.1 Agrandir le lab#

L'idempotence de cluster-up.sh est la procédure :

1. augmente WORKERS (ou CONTROL_PLANES, en restant impair) dans lab.env ;
2. vagrant up — seules les nouvelles VM sont créées et provisionnées ;
3. ./kubeadm/cluster-up.sh — saute ce qui est en place, joint les nouveaux nodes avec des identifiants frais.

Aucune régénération de certificat : les certSANs couvrent déjà 5 IP de control plane (§5.2).

Retirer un worker demande une vidange d'abord, pour que le cluster arrête de placer des pods sur une machine qui va disparaître :

kubectl drain k8s-w3 --ignore-daemonsets --delete-emptydir-data
vagrant destroy -f k8s-w3
kubectl delete node k8s-w3

puis baisse WORKERS dans lab.env.

7.2 Défaire le cluster sans détruire les VM#

./kubeadm/cluster-reset.sh          # demande confirmation
./kubeadm/cluster-reset.sh --yes    # sans interaction

Il lance kubeadm reset sur chaque node (les workers d'abord, pour qu'ils se désinscrivent pendant que l'API répond encore), puis supprime _out/ et kubeconfig. Les VM gardent leurs paquets, containerd et keepalived, donc la reconstruction se réduit à ./kubeadm/cluster-up.sh : des minutes au lieu d'un vagrant up complet. À préférer à vagrant destroy pour rejouer un bootstrap échoué, ou pour changer POD_CIDR, SERVICE_CIDR, le CNI ou la VIP : les quatre sont figés à kubeadm init.

⚠️ Destructif : etcd, les certificats et toutes les charges de travail sont perdus, y compris les PersistentVolumes sur disque de node.

ℹ️ Pourquoi un reset dédié. kubeadm reset laisse volontairement ce qu'il n'a pas créé : interfaces CNI, programmes eBPF épinglés sous /sys/fs/bpf (qui survivent au DaemonSet et continuent d'intercepter le trafic d'un cluster qui n'existe plus), et règles iptables de kube-proxy. node-reset.sh nettoie tout ça ; sans cette passe, l'init suivant hérite d'un datapath fantôme et le réseau de pods déraille sans que rien n'apparaisse dans les logs.

🔍 8. Notes de conception#

8.1 La VIP est portée par keepalived, pas par kube-vip#

La décision la plus structurante du dépôt. controlPlaneEndpoint pointe la VIP et est gravé dans les certificats et dans chaque kubeconfig au moment du kubeadm init : la VIP doit donc exister avant l'init.

kube-vip, la réponse habituelle des guides HA kubeadm, tourne en pod statique et élit son leader à travers l'API Kubernetes, c'est-à-dire à travers la VIP même qu'il est censé porter. La sortie documentée est --k8sConfigPath /etc/kubernetes/super-admin.conf, elle-même fragile depuis que Kubernetes 1.29 a sorti admin.conf du groupe system:masters (kube-vip#684, toujours ouverte).

keepalived n'a rien de tout ça : un simple démon VRRP, qui ignore Kubernetes et lève la VIP au démarrage de la VM. provision.sh le configure en VRRP unicast (le multicast est la première chose à mal se comporter sur un switch host-only VirtualBox, et on connaît de toute façon toutes les IP de control plane), avec les priorités cp1 = 100 / cp2 = 90 / cp3 = 80 et un vrrp_script qui interroge https://127.0.0.1:6443/livez/ping toutes les 3 s avec weight -30 : un CP dont l'apiserver est mort tombe à 70 et un cp2 sain à 90 reprend la VIP. /livez/ping est lisible anonymement grâce au binding system:public-info-viewer créé par kubeadm, donc aucun identifiant n'a besoin d'atteindre un script de santé. Il n'y a aucun bloc authentication : VRRPv2 envoie son mot de passe en clair et n'apporte rien ici, la frontière de confiance étant le réseau host-only. VRRP_ROUTER_ID est le bouton pour coexister avec un autre lab keepalived.

Tant qu'aucun cluster n'existe, le contrôle échoue sur chaque CP : tous perdent 30 points, l'ordre relatif tient, et la VIP est portée quand même, ce dont kubeadm init a besoin. kube-vip reste une bonne option une fois le cluster debout (mode --services) ; c'est le rôle au bootstrap qui ne marche pas ici.

La VIP est utilisée même avec un seul control plane, pour la même raison : pointer controlPlaneEndpoint sur l'IP réelle de cp1 transformerait « 1 CP → 3 CP » en régénération de tous les certificats et redistribution de tous les kubeconfig, au lieu d'un simple join.

8.2 containerd 2.x depuis le dépôt Docker#

Debian 13 livre containerd 1.7.24. Seule la branche 2.x implémente la méthode CRI RuntimeConfig que kubeadm utilise pour lire le pilote cgroup du runtime. En 1.36 son absence est un avertissement de preflight ; le repli disparaît en 1.37, et le backport vers 1.7 a été refusé (containerd#11346, fermée sans merge). CONTAINERD_SOURCE=debian reste disponible pour un lab hors-ligne, et c'est une impasse pour les montées de version.

SystemdCgroup = true compte plus que le champ cgroupDriver du kubelet : Debian 13 est en cgroup v2 avec systemd comme gestionnaire, et laisser containerd en cgroupfs met deux gestionnaires sur la même hiérarchie, et les nodes deviennent instables sous charge.

⚠️ Le piège 1.7 → 2.x : la clé de l'image pause a changé de nom et d'emplacement. La config v2 a sandbox_image sous [plugins."io.containerd.grpc.v1.cri"] ; la v3 a sandbox sous [plugins.'io.containerd.cri.v1.images'.pinned_images]. Une config recopiée telle quelle perd le réglage en silence, donc provision.sh la régénère depuis containerd config default à chaque passage et corrige la clé présente. Le tag lui-même vient de kubeadm config images list, jamais codé en dur : un écart est invisible en ligne et fatal hors-ligne.

8.3 API kubeadm v1beta4#

Défaut depuis Kubernetes 1.31 ; v1beta3 est déprécié. Le changement cassant à connaître : extraArgs et kubeletExtraArgs ne sont plus des dictionnaires mais des listes de {name, value}, pour qu'une option puisse être répétée. Tout fichier écrit avant 1.31 est invalide tel quel, et l'erreur de kubeadm ne désigne pas la forme.

# v1beta3 :  extraArgs: {bind-address: "0.0.0.0"}
# v1beta4 :  extraArgs: [{name: bind-address, value: "0.0.0.0"}]

make validate-kubeadm attrape exactement ça, en CI, sans cluster.

8.4 Ce que kubeadm ne fait pas, et que cluster-up.sh rattrape#

• Les étiquettes de rôle des workers — kubeadm n'en pose aucune, donc kubectl get nodes affiche <none> et les sélecteurs node-role.kubernetes.io/worker ne correspondent à rien.
• Le taint control-plane : UNTAINT_CP=auto ne le retire que si WORKERS=0, ce qui rend un lab à 1 VM utilisable.
• Les métriques du control plane : bind-address: 0.0.0.0 sur controllerManager et scheduler, qui sinon n'écoutent qu'en loopback et donnent à Prometheus deux cibles DOWN sans explication.
• Les images pré-tirées, pendant vagrant up et en parallèle entre VM, pour que kubeadm init ne télécharge rien, la première cause de timeout au bootstrap. Les workers ne tirent que pause et kube-proxy, ~500 Mio économisés chacun.
• Le swap coupé et masqué, unités systemd de swap incluses (/etc/fstab ne les décrit pas).

Le dossier synchronisé /vagrant est un rouage, pas un confort : cluster-up.sh rend les configs sur l'hôte et les VM les lisent dans /vagrant/_out/, donc rien n'a besoin de scp et aucun secret ne passe en ligne de commande où il finirait dans l'historique du shell. _out/join.env contient bien le token de jonction et la clé de certificats, lisibles depuis toutes les VM : acceptable pour un lab, pas un modèle pour la production.

🌐 9. CNI : Cilium, Calico ou Flannel#

kubeadm n'installe jamais de CNI. Contrairement au jumeau Talos (où flannel peut être posé par l'OS au bootstrap), le réseau de pods est ici toujours installé après, par ./_k8s/platform-up.sh. CNI est lu par cluster-up.sh (pour la décision kube-proxy et _out/cluster.env) et par l'étape plateforme (quel chart installer).

En pratique : garde cilium. C'est la seule valeur qui donne une EXTERNAL-IP aux Services sur un réseau host-only, donc la seule qui te donne les UI HTTPS. calico est là pour comparer les CNI et travailler sur NetworkPolicy (sa page) ; flannel pour un cluster délibérément nu.

⚠️ KUBE_PROXY_REPLACEMENT=true exige CNI=cilium, et cluster-up.sh refuse toute autre combinaison. Avec --skip-phases=addon/kube-proxy et sans remplacement, aucune ClusterIP ne répond, pas même CoreDNS joignant l'API. Le message d'erreur donne les deux sorties : CNI=cilium, ou KUBE_PROXY_REPLACEMENT=false.

ℹ️ Cilium a besoin de k8sServiceHost/k8sServicePort quand kube-proxy disparaît : plus rien ne provisionne la ClusterIP de l'apiserver, donc l'agent ne peut pas s'amorcer par kubernetes.default. Le lab le pointe sur la VIP, ce qui fait aussi survivre les agents à la perte d'un control plane.

⚠️ POD_CIDR doit être le CIDR que le CNI annonce vraiment. Cilium en mode cluster-pool vaut 10.0.0.0/8 par défaut, sans rapport avec ce qu'on a dit à kubeadm ; cilium-up.sh lui repasse POD_CIDR explicitement. Deux valeurs divergentes donnent un réseau de pods cassé qui a l'air configuré.

⚠️ Changer de CNI sur un cluster vivant n'est pas supporté. ./kubeadm/cluster-reset.sh (ou vagrant destroy) d'abord : deux CNI se disputent le réseau de pods, et le datapath résiduel est exactement ce que node-reset.sh existe pour nettoyer.

🛠️ 10. Valider une modification#

Tout se valide sans démarrer de cluster :

make validate       # shell + YAML + Vagrantfile + templates kubeadm + liens de doc
make docs           # régénère docs/index.html depuis tous les README (EN + FR)
make help           # liste les cibles

validate-kubeadm justifie son existence : elle attrape une vraie erreur de schéma v1beta4 au lieu de te la faire découvrir dix minutes après le début d'un vagrant up. En CI, où kubeadm est installé, le contrôle de schéma tourne toujours.

Le workflow ci appelle ces mêmes cibles make à chaque pull request, donc un contrôle ne peut pas passer en CI et échouer chez toi. Il vérifie aussi que les garde-fous se déclenchent réellement : CONTROL_PLANES=2 vagrant validate doit être rejeté. Rien dans le Makefile ne touche un cluster vivant ni ne régénère de secret : make validate est sans risque sur un lab debout.

ℹ️ validate-shell et validate-yaml ne couvrent que les fichiers suivis par ce dépôt. Le sous-module _k8s/ est un pointeur unique, donc aucun de ses scripts n'est vérifié ici ; ils le sont dans la CI de k8s-playground.

📄 11. Licence#

Apache License 2.0. Voir LICENSE. Utilise-le, modifie-le, redistribue-le, y compris commercialement, tant que tu conserves la notice de copyright et que tu signales tes modifications. Aucune garantie : c'est un lab, pas de production.

Elle couvre ce que ce dépôt contient : le Vagrantfile, les scripts kubeadm/, les templates, les manifestes, la doc. Elle ne s'étend pas aux composants tiers que ces scripts téléchargent (Kubernetes, containerd, keepalived, Cilium, Envoy Gateway, Longhorn, Vault…), ni au sous-module _k8s/ : k8s-playground porte sa propre LICENSE.

⬆️Upgrading Kubernetes

Reference at the time of writing: Kubernetes 1.36.3, apt repository v1.36, containerd 2.2.6, Cilium 1.20.0, CNI=cilium. Adapt node names and IPs to your topology (lab.env); the repo default is 1 control plane + 2 workers.

⚠️ Unlike the Talos sibling lab, this procedure has not been timed on a live run. It is the upstream kubeadm procedure transposed to this repo's variables and scripts; every command is quoted from the documentation linked in §6.

🎯 1. The two rules you cannot bend#

One MINOR version at a time. 1.36 → 1.37 → 1.38, never 1.36 → 1.38. This is not a kubeadm quirk: the API deprecation policy requires kube-apiserver not to skip minors, even on a single-instance cluster, and kubeadm upgrade apply refuses a target more than one minor above the current version. Patch versions inside a minor are free (1.36.3 → 1.36.7).

The kubelet must never be ahead of the apiserver.

That dictates the order of the whole procedure: control plane first, kubelet last. Upgrading a node's kubelet package before kubeadm upgrade apply has run puts a 1.37 kubelet in front of a 1.36 apiserver.

⚠️ Never run vagrant provision to "upgrade" the lab. provision.sh unholds the packages and installs kubelet/kubeadm/kubectl at K8S_VERSION with --allow-change-held-packages, on every node at once, without ever calling kubeadm upgrade. Bumping lab.env and re-provisioning would jump every kubelet to the new minor while the control plane is still on the old one. vagrant provision is for a fresh VM.

📦 2. Held packages, and one apt repository per MINOR#

provision.sh ends its package step with apt-mark hold kubelet kubeadm kubectl. An upgrade must be a deliberate act, never the side effect of an apt upgrade inside a VM — which would silently break the kubelet/apiserver skew. So every upgrade starts with apt-mark unhold and ends with apt-mark hold. Check with vagrant ssh k8s-cp1 -c "apt-mark showhold".

There is one apt repository per Kubernetes minor, and this is the step people miss:

https://pkgs.k8s.io/core:/stable:/v1.36/deb/

The v1.36 repository will never offer 1.37. Staying on it makes apt-get install kubeadm=1.37.x-* answer "Version '1.37.x-' for 'kubeadm' was not found"* — and people conclude the release does not exist.

Here the repository file is generated from K8S_APT_MINOR and the package version from K8S_VERSION. Both live in lab.env and must move together:

# lab.env
K8S_VERSION=1.37.0
K8S_APT_MINOR=v1.37

⚠️ Both also have fallback defaults duplicated in the Vagrantfile and in kubeadm/cluster-up.sh (K8S_VERSION only there), so that a lab without a lab.env still works. Bump them in the same commit as lab.env.example, or a lab built without lab.env restarts on the old version.

🧭 3. The lab shortcut: destroy and rebuild#

On a disposable lab, the fastest and safest path is not the upgrade at all:

# lab.env: K8S_VERSION=1.37.0 and K8S_APT_MINOR=v1.37
vagrant destroy -f
vagrant up
./kubeadm/cluster-up.sh
./_k8s/platform-up.sh

Clean cluster on the target version, no half-upgraded state, in roughly the time a careful rolling upgrade takes on three nodes. Use §4 instead when you want to practise the upgrade — that is the reason to run a kubeadm lab in the first place, and here a mistake costs a vagrant destroy.

⚡ 4. The real procedure, on a running cluster#

Everything runs inside the VMs (vagrant ssh <node>), except the kubectl commands, which run from the host with KUBECONFIG=$PWD/kubeconfig. 1.37.x stands for the exact target patch version; the -* suffix in the apt-get install lines is intentional, since the Debian revision is not always -1.1.

4.1 Pre-flight — never start from a degraded cluster#

export KUBECONFIG="$PWD/kubeconfig"
kubectl get nodes -o wide                 # every node Ready, all on the same version
kubectl get pods -A | grep -v Running     # nothing broken before you start
kubectl get --raw='/healthz/etcd'
vagrant ssh k8s-cp1 -c "sudo kubeadm certs check-expiration"

Read the target release's changelog, then check two lab-specific constraints:

💡 kubeadm upgrade pulls new control plane images. With REGISTRY_MIRROR set they come from the mirror; otherwise the node needs Internet access through its NAT NIC.

4.2 Every node starts with the same two steps#

On each node, in the order of §4.3 → §4.5:

# 1. Point apt at the NEW minor's repository
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] \
https://pkgs.k8s.io/core:/stable:/v1.37/deb/ /" \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list

# 2. Upgrade kubeadm ONLY
sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.37.x-*' && \
sudo apt-mark hold kubeadm
kubeadm version

And every node ends with the same four:

kubectl drain <node> --ignore-daemonsets
sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.37.x-*' kubectl='1.37.x-*' && \
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet
kubectl uncordon <node>

💡 If the drain stalls on a pod with an emptyDir, add --delete-emptydir-data. If it stalls on a PodDisruptionBudget (Longhorn is the usual suspect), fix the PDB rather than forcing; --disable-eviction is the blunt instrument of last resort.

What changes between node roles is only the middle step.

4.3 First control plane (k8s-cp1) — upgrade apply#

Between the two blocks of §4.2:

sudo kubeadm upgrade plan          # what would happen
sudo kubeadm upgrade apply v1.37.x # the step that upgrades the control plane

upgrade apply rewrites the static pod manifests for kube-apiserver, kube-controller-manager, kube-scheduler and etcd, and renews the certificates it manages on this node (§5).

⚠️ With CONTROL_PLANES=1 the API is unavailable while the static pods roll. Expected on a single control plane, and the best argument for practising this on a 3-CP topology.

kubectl get nodes                  # k8s-cp1 Ready, VERSION v1.37.x
kubectl get --raw='/healthz/etcd'

4.4 The other control planes (k8s-cp2, k8s-cp3) — upgrade node#

One node at a time, checking etcd between each: with 3 control planes the quorum is 2, and losing two at once freezes the API. The middle step becomes:

sudo kubeadm upgrade node

⚠️ The 192.168.56.5 VIP moves on its own while a control plane restarts — keepalived's health check (/livez/ping every 3 s, weight -30) drops the restarting node behind a healthy peer. Watch the failover happen:

while true; do curl -sk -o /dev/null -w '%{http_code} ' https://192.168.56.5:6443/livez; sleep 1; done

4.5 The workers (k8s-w1, k8s-w2, …)#

Same kubeadm upgrade node (on a worker it only updates the local kubelet config), one node at a time. Workers hold no etcd member, so nothing here can break quorum — but draining them all at once takes every workload down.

4.6 After the upgrade#

kubectl get nodes -o wide            # every node Ready, all on v1.37.x
kubectl get pods -A | grep -v Running
kubectl version

Then write the new version back into the repo, so a future rebuild starts where you left off:

Three of those four carry a duplicated default on purpose — a safety net when lab.env is missing. Two defaults that diverge give an incoherent lab: packages from one minor, generated configuration for another. make validate-defaults checks that pair, key by key.

🔐 5. Certificates, containerd and Cilium#

Certificates#

kubeadm issues client and serving certificates valid for 1 year, signed by a CA valid for 10 years.

vagrant ssh k8s-cp1 -c "sudo kubeadm certs check-expiration"

An upgrade renews them for you: kubeadm upgrade (both apply and node) renews the certificates it manages on that node, unless --certificate-renewal=false. A cluster upgraded at least once a year never sees an expired certificate — which is why the yearly expiry rarely bites in production and always bites on a lab VM left suspended for months.

Manual renewal, when no upgrade is due:

vagrant ssh k8s-cp1
sudo kubeadm certs renew all
sudo systemctl restart kubelet    # reloads the control plane static pods

⚠️ Renewing also renews admin.conf, which the host's kubeconfig was copied from. Refresh it, or kubectl keeps presenting the old client certificate:

vagrant ssh k8s-cp1 -c "sudo cp /etc/kubernetes/admin.conf /vagrant/_out/admin.conf"
cp -f _out/admin.conf kubeconfig && chmod 0600 kubeconfig

Two things kubeadm does not renew: the CA itself (10 years, beyond any lab's life) and the kubelet's own client certificate, which rotates automatically under /var/lib/kubelet/pki. None of this concerns the two short-lived items used for joining a node — the bootstrap token (24 h) and the certificate key (2 h), both regenerated on every cluster-up.sh run.

containerd#

Kubernetes, the container runtime and the CNI are three independent release trains. Bump one at a time and check the cluster in between.

containerd.io is not held by provision.sh, so it moves with a plain apt upgrade inside a VM — usually harmless, but it restarts every container on that node:

kubectl drain k8s-w1 --ignore-daemonsets
vagrant ssh k8s-w1 -c "sudo apt-get update && sudo apt-get install -y --only-upgrade containerd.io"
kubectl uncordon k8s-w1

provision.sh regenerates /etc/containerd/config.toml from containerd config default on every run and patches whichever pause key the format uses, so the 1.7 → 2.x rename cannot silently lose the setting — do not hand-edit that file and expect it to survive. Going back to CONTAINERD_SOURCE=debian is a downgrade to a dead end: containerd 1.7 will never implement RuntimeConfig and cannot carry you past 1.36.

Cilium#

# lab.env: CILIUM_VERSION=1.2x.y
./_k8s/cilium/cilium-up.sh

Run it from the repository root; the k8s-playground submodule finds the lab and the distribution on its own. The script is a helm upgrade --install, so it is the same command whether you install or upgrade. Read the Cilium upgrade notes first: a minor bump can require a one-off pre-flight step, and this lab depends on two Cilium features that must keep working — kubeProxyReplacement (there is no kube-proxy to fall back to) and the L2 announcement that gives the Envoy Gateway its IP.

kubectl -n kube-system exec ds/cilium -- cilium-dbg status --verbose
kubectl -n envoy-gateway-system get svc      # the Gateway must keep its EXTERNAL-IP

Everything else in the VMs (keepalived included) follows a plain apt upgrade, which is safe precisely because kubelet/kubeadm/kubectl are held.

📚 References#

• kubeadm — Upgrading kubeadm clusters
• Kubernetes — Version skew policy
• kubeadm — Certificate management
• Kubernetes — Installing kubeadm (apt repositories)
• Cilium — Upgrade guide
• ../TROUBLESHOOTING.md — symptoms and fixes

⬆️Monter Kubernetes de version

Référence au moment de l'écriture : Kubernetes 1.36.3, dépôt apt v1.36, containerd 2.2.6, Cilium 1.20.0, CNI=cilium. Adapte les noms de nodes et les IP à ta topologie (lab.env) ; le défaut du dépôt est 1 control plane + 2 workers.

⚠️ Contrairement au lab Talos jumeau, cette procédure n'a pas été chronométrée sur une exécution réelle. C'est la procédure kubeadm amont transposée aux variables et aux scripts de ce dépôt ; chaque commande est citée de la documentation liée au §6.

🎯 1. Les deux règles non négociables#

Un seul MINOR à la fois. 1.36 → 1.37 → 1.38, jamais 1.36 → 1.38. Ce n'est pas une bizarrerie de kubeadm : la politique de dépréciation de l'API interdit à kube-apiserver de sauter un minor, même sur un cluster à une seule instance, et kubeadm upgrade apply refuse une cible à plus d'un minor de la version courante. Les versions de patch dans un minor sont libres (1.36.3 → 1.36.7).

Le kubelet ne doit jamais être en avance sur l'apiserver.

C'est ce qui dicte l'ordre de toute la procédure : le control plane d'abord, le kubelet en dernier. Monter le paquet kubelet d'un node avant que kubeadm upgrade apply ait tourné place un kubelet 1.37 devant un apiserver 1.36.

⚠️ Ne lance jamais vagrant provision pour « mettre à jour » le lab. provision.sh dégèle les paquets et installe kubelet/kubeadm/kubectl à K8S_VERSION avec --allow-change-held-packages, sur tous les nodes d'un coup, sans jamais appeler kubeadm upgrade. Incrémenter lab.env puis reprovisionner ferait sauter tous les kubelets au nouveau minor pendant que le control plane est encore sur l'ancien. vagrant provision est fait pour une VM neuve.

📦 2. Paquets gelés, et un dépôt apt par MINOR#

provision.sh termine son étape paquets par apt-mark hold kubelet kubeadm kubectl. Une montée de version doit être un acte délibéré, jamais l'effet de bord d'un apt upgrade dans une VM — qui casserait en silence l'écart kubelet/apiserver. Toute montée commence donc par un apt-mark unhold et finit par un apt-mark hold. Vérification : vagrant ssh k8s-cp1 -c "apt-mark showhold".

Il y a un dépôt apt par minor de Kubernetes, et c'est l'étape que tout le monde rate :

https://pkgs.k8s.io/core:/stable:/v1.36/deb/

Le dépôt v1.36 n'offrira jamais la 1.37. Y rester fait répondre à apt-get install kubeadm=1.37.x-* : « Version '1.37.x-' for 'kubeadm' was not found »* — et on en conclut que la version n'existe pas.

Ici le fichier de dépôt est généré depuis K8S_APT_MINOR et la version du paquet depuis K8S_VERSION. Les deux vivent dans lab.env et doivent bouger ensemble :

# lab.env
K8S_VERSION=1.37.0
K8S_APT_MINOR=v1.37

⚠️ Les deux ont aussi des défauts de repli dupliqués dans le Vagrantfile et dans kubeadm/cluster-up.sh (K8S_VERSION seulement là), pour qu'un lab sans lab.env fonctionne quand même. Incrémente-les dans le même commit que lab.env.example, sinon un lab construit sans lab.env repart sur l'ancienne version.

🧭 3. Le raccourci du lab : détruire et reconstruire#

Sur un lab jetable, le chemin le plus rapide et le plus sûr n'est pas la montée de version :

# lab.env : K8S_VERSION=1.37.0 et K8S_APT_MINOR=v1.37
vagrant destroy -f
vagrant up
./kubeadm/cluster-up.sh
./_k8s/platform-up.sh

Cluster propre sur la version cible, sans état à moitié migré, à peu près dans le temps qu'une montée roulante prudente prend sur trois nodes. Utilise le §4 quand tu veux t'exercer à la montée de version — c'est la raison même de faire tourner un lab kubeadm, et ici une erreur coûte un vagrant destroy.

⚡ 4. La vraie procédure, sur un cluster vivant#

Tout se passe dans les VM (vagrant ssh <node>), sauf les commandes kubectl, qui tournent depuis l'hôte avec KUBECONFIG=$PWD/kubeconfig. 1.37.x représente la version de patch cible exacte ; le suffixe -* des lignes apt-get install est volontaire, la révision Debian n'étant pas toujours -1.1.

4.1 Pré-vol — ne jamais partir d'un cluster dégradé#

export KUBECONFIG="$PWD/kubeconfig"
kubectl get nodes -o wide                 # tous les nodes Ready, tous sur la même version
kubectl get pods -A | grep -v Running     # rien de cassé avant de commencer
kubectl get --raw='/healthz/etcd'
vagrant ssh k8s-cp1 -c "sudo kubeadm certs check-expiration"

Lis le changelog de la version cible, puis vérifie deux contraintes propres à ce lab :

💡 kubeadm upgrade tire de nouvelles images de control plane. Avec REGISTRY_MIRROR défini elles viennent du miroir ; sinon le node a besoin d'Internet par sa carte NAT.

4.2 Chaque node commence par les deux mêmes étapes#

Sur chaque node, dans l'ordre des §4.3 → §4.5 :

# 1. Pointer apt sur le dépôt du NOUVEAU minor
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] \
https://pkgs.k8s.io/core:/stable:/v1.37/deb/ /" \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list

# 2. Monter kubeadm SEULEMENT
sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.37.x-*' && \
sudo apt-mark hold kubeadm
kubeadm version

Et chaque node finit par les quatre mêmes :

kubectl drain <node> --ignore-daemonsets
sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.37.x-*' kubectl='1.37.x-*' && \
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet
kubectl uncordon <node>

💡 Si la vidange coince sur un pod avec un emptyDir, ajoute --delete-emptydir-data. Si elle coince sur un PodDisruptionBudget (Longhorn est le suspect habituel), corrige le PDB plutôt que de forcer ; --disable-eviction est l'instrument brutal de dernier recours.

Ce qui change entre les rôles de nodes, c'est seulement l'étape du milieu.

4.3 Premier control plane (k8s-cp1) — upgrade apply#

Entre les deux blocs du §4.2 :

sudo kubeadm upgrade plan          # ce qui se passerait
sudo kubeadm upgrade apply v1.37.x # l'étape qui monte le control plane

upgrade apply réécrit les manifestes de pods statiques de kube-apiserver, kube-controller-manager, kube-scheduler et etcd, et renouvelle les certificats qu'il gère sur ce node (§5).

⚠️ Avec CONTROL_PLANES=1, l'API est indisponible pendant que les pods statiques roulent. Attendu sur un control plane unique, et le meilleur argument pour s'exercer sur une topologie à 3 CP.

kubectl get nodes                  # k8s-cp1 Ready, VERSION v1.37.x
kubectl get --raw='/healthz/etcd'

4.4 Les autres control planes (k8s-cp2, k8s-cp3) — upgrade node#

Un node à la fois, en vérifiant etcd entre chaque : avec 3 control planes le quorum est 2, et en perdre deux d'un coup gèle l'API. L'étape du milieu devient :

sudo kubeadm upgrade node

⚠️ La VIP 192.168.56.5 se déplace toute seule pendant le redémarrage d'un control plane — le contrôle de santé de keepalived (/livez/ping toutes les 3 s, weight -30) fait passer le node qui redémarre derrière un pair sain. Regarde le basculement se produire :

while true; do curl -sk -o /dev/null -w '%{http_code} ' https://192.168.56.5:6443/livez; sleep 1; done

4.5 Les workers (k8s-w1, k8s-w2, …)#

Même kubeadm upgrade node (sur un worker il ne met à jour que la config locale du kubelet), un node à la fois. Les workers ne portent aucun membre etcd, donc rien ici ne peut casser le quorum — mais les vidanger tous en même temps met toutes les charges de travail à terre.

4.6 Après la montée de version#

kubectl get nodes -o wide            # tous les nodes Ready, tous en v1.37.x
kubectl get pods -A | grep -v Running
kubectl version

Puis réécris la nouvelle version dans le dépôt, pour qu'une reconstruction future reprenne là où tu t'es arrêté :

Trois de ces quatre fichiers portent un défaut dupliqué à dessein — un filet de sécurité quand lab.env manque. Deux défauts qui divergent donnent un lab incohérent : des paquets d'un minor, une configuration générée pour un autre. make validate-defaults vérifie cette paire, clé par clé.

🔐 5. Certificats, containerd et Cilium#

Certificats#

kubeadm émet des certificats client et serveur valables 1 an, signés par une AC valable 10 ans.

vagrant ssh k8s-cp1 -c "sudo kubeadm certs check-expiration"

Une montée de version les renouvelle pour toi : kubeadm upgrade (apply comme node) renouvelle les certificats qu'il gère sur ce node, sauf --certificate-renewal=false. Un cluster mis à jour au moins une fois par an ne voit donc jamais de certificat expiré — ce qui explique que l'expiration annuelle morde rarement en production et toujours sur une VM de lab laissée suspendue des mois.

Renouvellement manuel, quand aucune montée n'est prévue :

vagrant ssh k8s-cp1
sudo kubeadm certs renew all
sudo systemctl restart kubelet    # recharge les pods statiques du control plane

⚠️ Le renouvellement touche aussi admin.conf, dont le kubeconfig de l'hôte est une copie. Rafraîchis-le, sinon kubectl continue de présenter l'ancien certificat client :

vagrant ssh k8s-cp1 -c "sudo cp /etc/kubernetes/admin.conf /vagrant/_out/admin.conf"
cp -f _out/admin.conf kubeconfig && chmod 0600 kubeconfig

Deux choses que kubeadm ne renouvelle pas : l'AC elle-même (10 ans, au-delà de la vie de tout lab) et le certificat client du kubelet, qui tourne automatiquement sous /var/lib/kubelet/pki. Rien de tout ça ne concerne les deux éléments à courte vie utilisés pour joindre un node — le token de bootstrap (24 h) et la clé de certificats (2 h), tous deux régénérés à chaque exécution de cluster-up.sh.

containerd#

Kubernetes, le runtime de conteneurs et le CNI sont trois trains de versions indépendants. Monte-les un par un et vérifie le cluster entre chaque.

containerd.io n'est pas gelé par provision.sh : il bouge donc avec un simple apt upgrade dans une VM — généralement sans conséquence, mais ça redémarre tous les conteneurs du node :

kubectl drain k8s-w1 --ignore-daemonsets
vagrant ssh k8s-w1 -c "sudo apt-get update && sudo apt-get install -y --only-upgrade containerd.io"
kubectl uncordon k8s-w1

provision.sh régénère /etc/containerd/config.toml depuis containerd config default à chaque passage et corrige la clé pause que le format utilise, donc le renommage 1.7 → 2.x ne peut pas perdre le réglage en silence — n'édite pas ce fichier à la main en espérant que ça survive. Revenir à CONTAINERD_SOURCE=debian est une régression vers une impasse : containerd 1.7 n'implémentera jamais RuntimeConfig et ne peut pas te porter au-delà de la 1.36.

Cilium#

# lab.env : CILIUM_VERSION=1.2x.y
./_k8s/cilium/cilium-up.sh

Lance-le depuis la racine du dépôt ; le sous-module k8s-playground trouve le lab et la distribution tout seul. Le script est un helm upgrade --install, donc c'est la même commande à l'installation et à la montée. Lis d'abord les notes de montée de Cilium : un changement de minor peut demander une étape préalable unique, et ce lab dépend de deux fonctions Cilium qui doivent continuer de marcher — kubeProxyReplacement (il n'y a pas de kube-proxy sur lequel se rabattre) et l'annonce L2 qui donne son IP au Gateway Envoy.

kubectl -n kube-system exec ds/cilium -- cilium-dbg status --verbose
kubectl -n envoy-gateway-system get svc      # le Gateway doit garder son EXTERNAL-IP

Tout le reste dans les VM (keepalived compris) suit un simple apt upgrade, ce qui est sans risque précisément parce que kubelet/kubeadm/kubectl sont gelés.

📚 Références#

• kubeadm — Upgrading kubeadm clusters
• Kubernetes — Version skew policy
• kubeadm — Certificate management
• Kubernetes — Installing kubeadm (dépôts apt)
• Cilium — Upgrade guide
• ../DEPANNAGE.md — symptômes et remèdes

🚑Troubleshooting

This page covers the lab itself: the host, VirtualBox, keepalived, kubeadm and the Debian nodes. Addon problems (Longhorn, Vault, Calico…) are documented with the addons, in k8s-playground.

Unless stated otherwise, commands run from the repository root, with export KUBECONFIG="$PWD/kubeconfig".

🖥️ 1. Host, repository and VirtualBox#

vagrant up dies on VERR_VMX_IN_VMX_ROOT_MODE#

VBoxManage: error: VT-x is being used by another hypervisor (VERR_VMX_IN_VMX_ROOT_MODE).

VirtualBox and KVM cannot hold VT-x at the same time, and most Linux distributions load KVM at boot.

lsmod | grep kvm                    # Intel: kvm_intel — AMD: kvm_amd
sudo modprobe -r kvm_intel kvm      # fails if a KVM/libvirt VM is still running

💡 KVM comes back on every boot. If this host never runs KVM/libvirt, blacklist it once:

echo -e "blacklist kvm_intel\nblacklist kvm" | sudo tee /etc/modprobe.d/disable-kvm.conf

VirtualBox refuses the 192.168.56.0/24 host-only network#

VirtualBox 7 only allows explicitly permitted host-only ranges:

# /etc/vbox/networks.conf
* 192.168.56.0/21

The whole lab lives in that /24 (nodes, the .5 VIP, the .200–.230 LoadBalancer pool), so nothing works until VirtualBox accepts it.

vagrant up refuses an even number of control planes#

Vagrant-KubeADM: CONTROL_PLANES=2 is EVEN — etcd requires an odd number to hold a
useful quorum (1, 3, 5). With 2 members, losing a single node freezes the API.

A guard rail, not a bug: etcd holds quorum at (n/2)+1, so two members tolerate zero failures while costing twice as much as one. Use 1, 3 or 5. The Vagrantfile also refuses a node IP colliding with .1, .2, .100 or the VIP, and refuses duplicates; each error names the offending variable.

_k8s/ is empty — ./_k8s/platform-up.sh: No such file or directory#

_k8s/ is a git submodule. A plain git clone records it but does not check it out.

git submodule update --init --recursive     # fills _k8s/
git -C _k8s log --oneline -1                # sanity check

Clone correctly next time with git clone --recurse-submodules <url>. git pull does not update the submodule either: repeat the command above after every pull, or git submodule update --remote _k8s to jump to the latest upstream commit.

The _k8s/ scripts find neither lab.env nor the kubeconfig#

Symptoms: addons install into the wrong domain (lab.example.io instead of your LAB_DOMAIN), the wrong CNI is chosen, or kubectl inside the scripts fails with connection refused. The banner the scripts print at start-up shows lab.env: absent (defaults).

The lab was not located. k8s-playground has no Vagrantfile of its own: it takes the directory containing _k8s/ as the lab, provided that directory carries a Vagrantfile. That is where lab.env, _out/ and kubeconfig live. The same walk decides the distribution (kubeadm/cluster-up.sh next to the Vagrantfile = kubeadm lab), so a lab that is not found also means a distribution that is not detected.

ls Vagrantfile lab.env kubeadm/cluster-up.sh   # marker, config, distro signature
ls -d _k8s/lib                                 # _k8s/ really is INSIDE the lab

Typical causes: _k8s/ cloned on its own somewhere else, a lab.env never created from lab.env.example, or scripts invoked through a symlink landing outside the lab. The pointer always wins over detection:

LAB_DIR=/path/to/Vagrant-kubeadm ./_k8s/platform-up.sh

💡 LAB_ENV=/path/to/lab.env does the same when the file is elsewhere or named differently. LAB_DIR is the one to remember: it drives lab.env, _out/cluster.env and the default KUBECONFIG at once.

🌐 2. The API VIP and keepalived#

cluster-up.sh fails on "the apiserver does not answer on the VIP"#

    - waiting for https://192.168.56.5:6443 ....................... FAILED (600s)
ERROR: the apiserver does not answer on the VIP 192.168.56.5 after 600s.

kubeadm init has already run: the script is waiting for /readyz through the VIP, the address every other node will use to join. Two causes, by frequency.

Cause 1: keepalived is not carrying the VIP.

vagrant ssh k8s-cp1 -c "ip -4 addr show | grep 192.168.56.5"
vagrant ssh k8s-cp1 -c "sudo systemctl status keepalived"
vagrant ssh k8s-cp1 -c "sudo journalctl -u keepalived -n 50 --no-pager"

The interface is detected, never hard-coded. Check what provision.sh found:

vagrant ssh k8s-cp1 -c "cat /etc/kubeadm-lab/node.env"
vagrant ssh k8s-cp1 -c "sudo sed -n '/vrrp_instance/,\$p' /etc/keepalived/keepalived.conf"

If HOSTONLY_IF fell back to eth1 while the VM really uses enp0s8, keepalived binds to an interface that does not exist. Re-run vagrant provision k8s-cp1 once the VM has its host-only address.

Cause 2: the apiserver itself does not start.

vagrant ssh k8s-cp1 -c "sudo crictl ps -a | grep apiserver"
vagrant ssh k8s-cp1 -c "sudo journalctl -u kubelet -n 50 --no-pager"
vagrant ssh k8s-cp1 -c "sudo crictl logs \$(sudo crictl ps -a -q --name kube-apiserver | head -1)"

A CrashLoopBackOff apiserver is almost always etcd underneath (see section 5). Note that keepalived's health check only subtracts 30 priority points, it never drops the VIP, so the VIP being up proves nothing about the apiserver.

The VIP is held by TWO nodes at once (VRRP split-brain)#

kubectl behaves erratically: one request succeeds, the next times out. The journal shows Entering MASTER STATE on two nodes.

for n in k8s-cp1 k8s-cp2 k8s-cp3; do
  echo -n "$n: " ; vagrant ssh "$n" -c "ip -4 -o addr show | grep -c 192.168.56.5" -- -q
done
# healthy: exactly one node answers 1, the others 0

VRRP here is unicast (unicast_src_ip + unicast_peer), not multicast, because multicast is the first thing to misbehave on a VirtualBox host-only switch. A control plane that does not see its peers believes it is alone and promotes itself.

# the peer list must contain every OTHER control plane IP
vagrant ssh k8s-cp1 -c "sudo sed -n '/unicast/,/}/p' /etc/keepalived/keepalived.conf"

# the router ID must be IDENTICAL on all control planes
for n in k8s-cp1 k8s-cp2 k8s-cp3; do
  vagrant ssh "$n" -c "sudo sed -n 's/.*virtual_router_id //p' /etc/keepalived/keepalived.conf" -- -q
done

Three causes, in order of likelihood:

1. A missing unicast_peer block — keepalived does not reject such a config, it silently reverts to multicast, and the two modes are mutually deaf. vagrant provision <node> rewrites it (the config always lists the five control-plane IPs the addressing plan allows, so a config written for one CP is already correct for three).
2. Divergent VRRP_ROUTER_ID — nodes provisioned with different lab.env values. All control planes of one cluster must share the same ID.
3. Another keepalived lab on the same host-only network with the same ID (default 51). Change VRRP_ROUTER_ID, then vagrant provision.

ℹ️ There is no VRRP password on purpose: VRRPv2 authentication sends it in clear text and buys nothing. The trust boundary is the host-only network; the isolation knob is VRRP_ROUTER_ID.

☸️ 3. Nodes and kubeadm#

The nodes stay NotReady, CoreDNS stays Pending#

NAME      STATUS     ROLES           AGE   VERSION
k8s-cp1   NotReady   control-plane   2m    v1.36.3
k8s-w1    NotReady   worker          1m    v1.36.3

Normal between cluster-up.sh and the platform step. kubeadm installs no CNI, and a node with no pod network never reports Ready. CoreDNS follows: every node carries the node.kubernetes.io/not-ready taint, which it does not tolerate.

kubectl describe node k8s-cp1 | sed -n '/Conditions:/,/Addresses:/p'
# Ready False — KubeletNotReady — cni plugin not initialized

Fix: ./_k8s/platform-up.sh. With CNI=none nothing will ever install a network: that is what the setting means, and cluster-up.sh prints a different closing message in that case.

Still NotReady after the CNI install, or CoreDNS still Pending after the nodes are Ready:

kubectl -n kube-system get pods -l k8s-app=cilium -o wide
kubectl -n kube-system logs ds/cilium --tail=50
kubectl -n kube-system logs deploy/cilium-operator --tail=50

WORKERS=0 with UNTAINT_CP=false also leaves nowhere to schedule.

Every node shows the same IP, 10.0.2.15#

# NAME      INTERNAL-IP
# k8s-cp1   10.0.2.15
# k8s-w1    10.0.2.15

Each VM has two NICs: NIC1 = VirtualBox NAT (always 10.0.2.15, identical on every VM) and NIC2 = host-only (the real cluster address). Without kubeletExtraArgs: node-ip the kubelet picks the default-route interface, which is the NAT one. kubectl get nodes looks plausible, but logs, exec, probes and cross-node traffic all go to the wrong place.

The lab sets node-ip in all three templates, so you only hit this on a node joined by hand with the printed kubeadm join line, which cannot carry node-ip.

vagrant ssh k8s-w1 -c "cat /var/lib/kubelet/kubeadm-flags.env"
# expected: KUBELET_KUBEADM_ARGS="… --node-ip=192.168.56.101 …"

Supported fix: redo the join through the repo (./kubeadm/cluster-reset.sh && ./kubeadm/cluster-up.sh). To repair a single node, add --node-ip=<host-only IP> to /var/lib/kubelet/kubeadm-flags.env and systemctl restart kubelet; if INTERNAL-IP does not change, kubectl delete node k8s-w1 so the kubelet re-registers.

kubeadm join fails on an expired token or certificate key#

Easy fix: re-run ./kubeadm/cluster-up.sh. It is idempotent, and node-init.sh regenerates both elements on every run before rewriting _out/join.env. Joining a node days after the initial init is a supported path.

By hand, if you are driving kubeadm yourself (both are safe to replay on a running cluster):

vagrant ssh k8s-cp1 -c "sudo kubeadm init phase upload-certs --upload-certs \\
     --config /vagrant/_out/kubeadm-init.yaml"                                  # new certificate key
vagrant ssh k8s-cp1 -c "sudo kubeadm token create --print-join-command"        # new token + CA hash

⚠️ Run upload-certs with --config. Without it, kubeadm builds its API client from a LocalAPIEndpoint.AdvertiseAddress it detects off the default route (10.0.2.15 in any Vagrant VM), and TLS fails on x509: certificate is valid for …, not 10.0.2.15. The endpoint is what must be corrected: never add 10.0.2.15 to certSANs, it identifies no node at all.

kubeadm preflight complains about swap, CPU count or memory#

[ERROR Swap]: swap is enabled; production deployments should disable swap …
[ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
[ERROR Mem]: the system RAM (1024 MB) is less than the minimum 1700 MB

Swap is already handled by provision.sh: swapoff -a, the /etc/fstab line commented out, and any systemd swap unit masked (Debian 13 can provide swap through a unit /etc/fstab never mentions; that is how swap comes back after a reboot). The error showing up anyway means provisioning did not finish:

vagrant ssh k8s-cp1 -c "free -m ; swapon --show ; systemctl list-unit-files --type=swap"
vagrant provision k8s-cp1

CPU and memory thresholds are kubeadm's own: 2 vCPU and ~1700 MiB on a control plane. The repo defaults clear them, so this only bites after lowering them in lab.env. Resources change on a VM restart: vagrant reload k8s-cp1.

ℹ️ NodeSwap is GA since 1.34, but failSwapOn still defaults to true: the kubelet refuses to start with swap on until you configure it explicitly. On a lab, disabling swap is the shortest and best-tested path.

A preflight warning about RuntimeConfig or the cgroup driver#

A warning, not an error: kubeadm could not read the cgroup driver from the container runtime and fell back to the cgroupDriver field of KubeletConfiguration. Only containerd 2.x implements the CRI RuntimeConfig method it uses; Debian 13 ships 1.7.24, which never will (backport refused upstream, containerd#11346).

vagrant ssh k8s-cp1 -c "containerd --version"
vagrant ssh k8s-cp1 -c "sudo grep SystemdCgroup /etc/containerd/config.toml"   # must be true

With CONTAINERD_SOURCE=docker (the default) the warning disappears. With CONTAINERD_SOURCE=debian it is expected: harmless in 1.36, fatal in 1.37 where the fallback is removed, so that value is an offline-lab option and a dead end for upgrades.

⚠️ What really matters is SystemdCgroup = true. Debian 13 is cgroup v2 with systemd as the manager; leaving containerd on cgroupfs makes two managers fight over one hierarchy and the nodes go unstable under load.

🔌 4. Pod network and Services#

After a cluster-reset.sh, the pod network behaves inexplicably#

Pods get IPs but cross-node traffic dies; DNS fails while ping 1.1.1.1 works; the Cilium agent complains about pre-existing BPF maps.

kubeadm reset deliberately leaves behind what it did not lay down: CNI interfaces, pinned eBPF programs, and kube-proxy's iptables rules, so a later kubeadm init inherits a ghost datapath. kubeadm/node-reset.sh is the cleanup and cluster-reset.sh runs it everywhere. It removes /etc/cni/net.d/*, the cilium_*/flannel.1/cni0/vxlan.calico/kube-ipvs0/lxc*/cali* interfaces, the pinned programs under /sys/fs/bpf/tc/globals/cilium_*, the KUBE-/CILIUM_/ cali- chains and IPVS, then wipes /var/lib/etcd, /var/lib/cni, /run/flannel and restarts containerd.

Check what is left on a suspect node:

vagrant ssh k8s-w1 -c "ip -o link show | grep -E 'cilium|lxc|flannel|cali|cni0'"
vagrant ssh k8s-w1 -c "sudo ls /sys/fs/bpf/tc/globals/ 2>/dev/null"
vagrant ssh k8s-w1 -c "sudo iptables-save | grep -cE 'KUBE-|CILIUM_|cali-'"

Anything non-empty means the cleanup did not complete: the script prints partial reset on <node> — carrying on rather than stopping. Re-run it there: vagrant ssh k8s-w1 -c "sudo bash /vagrant/kubeadm/node-reset.sh". In doubt, vagrant destroy -f && vagrant up is the guaranteed clean slate.

ℹ️ cluster-reset.sh is also the right tool to change POD_CIDR, SERVICE_CIDR, the CNI or the VIP: all four are frozen at kubeadm init time.

A LoadBalancer Service stays <pending>#

Cause 1: the CNI is not Cilium. Only Cilium hands out Service IPs here (L2/ARP announcement). Calico needs BGP and there is no peer router on a host-only network (MetalLB required); flannel and none do nothing.

sed -n 's/^CNI=//p' _out/cluster.env      # what the cluster was actually built with

⚠️ _out/cluster.env is the truth (written at bootstrap); lab.env is only an intent and may have been edited afterwards.

Cause 2: the L2 pool is missing, exhausted or announced on the wrong interface.

kubectl get ciliumloadbalancerippool
kubectl get ciliuml2announcementpolicy
kubectl -n kube-system logs deploy/cilium-operator --tail=50
sed -n 's/^HOSTONLY_IF=//p' _out/cluster.env

The pool is 192.168.56.200–.230 by default, and the announcement interface comes from the detected HOSTONLY_IF. A pool overlapping the node range, or a policy pinned to an interface that does not exist, both give a permanent <pending>. Changing the pool is a re-run away: ./_k8s/cilium/cilium-up.sh.

🗄️ 5. etcd and cluster performance#

etcd loses its leader, or the whole cluster crawls#

etcdserver: request timed out
apply request took too long
waiting for ReadIndex response took too long, retrying
leader changed

kubectl takes seconds to answer, pods stay Pending, the apiserver restarts on its own.

kubectl get --raw='/healthz/etcd'
kubectl -n kube-system logs -l component=etcd --tail=50
vagrant ssh k8s-cp1 -c "free -m ; uptime"

Causes, in order of frequency on this lab:

1. fsync latency. etcd commits every write to disk before acknowledging it. On VirtualBox, a VM disk on a spinning drive (or on an SSD already saturated by the host) pushes fsync past etcd's tolerance and leader election starts flapping. Keep the VM disks on an SSD, and do not run a 3-control-plane topology next to a heavy build.
2. CP_MEM too low. A stacked etcd on 2048 MiB has ~350 MiB of headroom; the first addons eat it. 3072 is the real floor, _k8s/observability/ wants 4096.
3. Clock drift. etcd is very sensitive to it. The Vagrantfile lowers the guest additions' time-sync threshold to 1000 ms, which covers a suspend/resume cycle, but a VM left suspended for a long time is better off vagrant reload-ed.

⚠️ With 3 control planes etcd tolerates one failure. Do not stop two at the same time (during an upgrade included, see kubeadm/UPGRADE.md): the API freezes until quorum is back.

🔐 6. Lab UIs over HTTPS#

Work down the chain, in order; each step assumes the previous one.

1. Does the Gateway have an IP?

kubectl -n envoy-gateway-system get gateway main-gateway -o jsonpath='{.status.addresses[0].value}'; echo

Empty or <pending> → a LoadBalancer problem, see section 4. The expected address is the first IP of the pool, 192.168.56.200 by default.

2. Does the name resolve to that IP?

LAB_DOMAIN has no reason to resolve on your machine. platform-up.sh prints the line to add:

# /etc/hosts on the HOST
192.168.56.200  argo.kubeadm.lab.example.io grafana.kubeadm.lab.example.io

…or a wildcard A record *.<LAB_DOMAIN> → 192.168.56.200 if you own a DNS zone (DNS-only behind Cloudflare: the proxy cannot reach a private IP).

getent hosts argo.kubeadm.lab.example.io

⚠️ Do not test the Gateway IP with ping. A Service IP announced in L2 by Cilium answers ARP and TCP but not ICMP: no interface actually carries the address. A failing ping on .200 is normal and proves nothing, while ping on a node works, which makes the false negative convincing. The real proof that the announcement works is the ARP entry resolving to a node's MAC:

sudo ip neigh flush 192.168.56.200
curl -s -o /dev/null --max-time 5 http://192.168.56.200/    # 404 = Envoy answers
ip neigh show 192.168.56.200                                # lladdr = the announcing node

3. Is there an HTTPRoute for that hostname?

kubectl get httproute -A
kubectl -n <ns> describe httproute <name> | sed -n '/Status:/,$p'   # Accepted / ResolvedRefs

ℹ️ On the bare IP, http:// answers 404 (Envoy is listening, no route matches) but https:// answers nothing at all: the TLS listener is scoped by hostname, so a request without SNI matches no listener. Test with the name, short-circuiting DNS if needed: curl -sk --resolve argo.kubeadm.lab.example.io:443:192.168.56.200 https://argo.kubeadm.lab.example.io/.

4. Is the TLS mode the one you think it is?

sed -n 's/^SELF_SIGNED=//p' lab.env
kubectl -n envoy-gateway-system get secret | grep wildcard

A browser warning is therefore normal in two of the three modes. To trust the local CA:

sudo cp _out/self-signed/ca.crt /usr/local/share/ca-certificates/vagrant-kubeadm-lab.crt
sudo update-ca-certificates

5. Still nothing? Look at the proxy itself:

kubectl -n envoy-gateway-system get pods
kubectl -n envoy-gateway-system logs deploy/envoy-gateway --tail=50

🧰 7. Toolbox#

From the host#

vagrant status                       # which VMs exist and are running
vagrant ssh k8s-cp1                  # interactive shell
vagrant ssh k8s-cp1 -c "<command>" -- -q -o LogLevel=ERROR   # one shot, quiet (what the scripts use)
vagrant provision k8s-cp1            # replay provision.sh (idempotent)
vagrant reload k8s-cp1               # restart, applying new CPU/RAM from lab.env

export KUBECONFIG="$PWD/kubeconfig"
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -30
kubectl get --raw='/readyz?verbose'

cat _out/cluster.env                 # what the cluster was REALLY built with

⚠️ _out/join.env holds the join token and the certificate key. _out/ is gitignored, but readable by every VM through the /vagrant synced folder. Never paste its contents anywhere.

Inside a VM#

cat /etc/kubeadm-lab/node.env                 # role, node IP, detected host-only interface
ip -4 addr show                               # is the VIP here?
sudo systemctl status kubelet containerd keepalived

sudo journalctl -u kubelet -f
sudo journalctl -u containerd -n 50 --no-pager
sudo journalctl -u keepalived -n 50 --no-pager

sudo crictl ps -a                             # containers, including dead ones
sudo crictl logs <container-id>

sudo kubeadm certs check-expiration           # control planes only
sudo kubeadm config images list --kubernetes-version v1.36.3

💡 crictl talks to the same socket as the kubelet thanks to /etc/crictl.yaml, written by provision.sh. Without it, crictl goes looking for dockershim and prints confusing errors.

The nuclear options, from least to most destructive#

📚 References#

• kubeadm — Troubleshooting
• kubeadm — Configuring a cgroup driver
• Cilium — Troubleshooting
• etcd — Tuning
• kubeadm/UPGRADE.md — version bumps and certificate renewal
• k8s-playground — the _k8s/ application layer, addon by addon, with its own pitfalls sections

🚑Dépannage

Cette page couvre le lab lui-même : l'hôte, VirtualBox, keepalived, kubeadm et les nodes Debian. Les problèmes d'addons (Longhorn, Vault, Calico…) sont documentés avec les addons, dans k8s-playground.

Sauf mention contraire, les commandes se lancent depuis la racine du dépôt, avec export KUBECONFIG="$PWD/kubeconfig".

🖥️ 1. Hôte, dépôt et VirtualBox#

vagrant up meurt sur VERR_VMX_IN_VMX_ROOT_MODE#

VBoxManage: error: VT-x is being used by another hypervisor (VERR_VMX_IN_VMX_ROOT_MODE).

VirtualBox et KVM ne peuvent pas détenir VT-x en même temps, et la plupart des distributions Linux chargent KVM au démarrage.

lsmod | grep kvm                    # Intel : kvm_intel — AMD : kvm_amd
sudo modprobe -r kvm_intel kvm      # échoue si une VM KVM/libvirt tourne encore

💡 KVM revient à chaque démarrage. Si cet hôte ne fait jamais de KVM/libvirt, blackliste-le une fois :

echo -e "blacklist kvm_intel\nblacklist kvm" | sudo tee /etc/modprobe.d/disable-kvm.conf

VirtualBox refuse le réseau host-only 192.168.56.0/24#

VirtualBox 7 n'autorise que les plages host-only explicitement permises :

# /etc/vbox/networks.conf
* 192.168.56.0/21

Tout le lab vit dans ce /24 (nodes, VIP .5, pool LoadBalancer .200–.230), donc rien ne fonctionne avant que VirtualBox l'accepte.

vagrant up refuse un nombre pair de control planes#

Vagrant-KubeADM: CONTROL_PLANES=2 is EVEN — etcd requires an odd number to hold a
useful quorum (1, 3, 5). With 2 members, losing a single node freezes the API.

Un garde-fou, pas un bug : etcd tient son quorum à (n/2)+1, donc deux membres ne tolèrent aucune panne tout en coûtant deux fois un seul. Utilise 1, 3 ou 5. Le Vagrantfile refuse aussi une IP de node qui collisionne avec .1, .2, .100 ou la VIP, et refuse les doublons ; chaque erreur nomme la variable fautive.

_k8s/ est vide — ./_k8s/platform-up.sh: No such file or directory#

_k8s/ est un sous-module git. Un git clone simple l'enregistre mais ne le sort pas.

git submodule update --init --recursive     # remplit _k8s/
git -C _k8s log --oneline -1                # contrôle rapide

La prochaine fois, clone correctement : git clone --recurse-submodules <url>. git pull ne met pas non plus le sous-module à jour : répète la commande ci-dessus après chaque pull, ou git submodule update --remote _k8s pour sauter au dernier commit amont.

Les scripts _k8s/ ne trouvent ni lab.env ni le kubeconfig#

Symptômes : les addons s'installent sur le mauvais domaine (lab.example.io au lieu de ton LAB_DOMAIN), le mauvais CNI est choisi, ou kubectl échoue dans les scripts sur connection refused. La bannière affichée au démarrage indique lab.env: absent (defaults).

Le lab n'a pas été localisé. k8s-playground n'a pas de Vagrantfile : il prend comme lab le dossier qui contient _k8s/, à condition que ce dossier porte un Vagrantfile. C'est là que vivent lab.env, _out/ et kubeconfig. Le même parcours décide de la distribution (kubeadm/cluster-up.sh à côté du Vagrantfile = lab kubeadm), donc un lab non trouvé signifie aussi une distribution non détectée.

ls Vagrantfile lab.env kubeadm/cluster-up.sh   # marqueur, config, signature de distro
ls -d _k8s/lib                                 # _k8s/ est bien DANS le lab

Causes typiques : _k8s/ cloné seul ailleurs, un lab.env jamais créé depuis lab.env.example, ou des scripts appelés par un lien symbolique qui sort du lab. Le pointeur explicite gagne toujours sur la détection :

LAB_DIR=/chemin/vers/Vagrant-kubeadm ./_k8s/platform-up.sh

💡 LAB_ENV=/chemin/vers/lab.env fait pareil quand le fichier est ailleurs ou nommé autrement. LAB_DIR est celui à retenir : il pilote lab.env, _out/cluster.env et le KUBECONFIG par défaut d'un coup.

🌐 2. La VIP de l'API et keepalived#

cluster-up.sh échoue sur « l'apiserver ne répond pas sur la VIP »#

    - waiting for https://192.168.56.5:6443 ....................... FAILED (600s)
ERROR: the apiserver does not answer on the VIP 192.168.56.5 after 600s.

kubeadm init a déjà tourné : le script attend /readyz à travers la VIP, l'adresse que tous les autres nodes utiliseront pour joindre. Deux causes, par fréquence.

Cause 1 : keepalived ne porte pas la VIP.

vagrant ssh k8s-cp1 -c "ip -4 addr show | grep 192.168.56.5"
vagrant ssh k8s-cp1 -c "sudo systemctl status keepalived"
vagrant ssh k8s-cp1 -c "sudo journalctl -u keepalived -n 50 --no-pager"

L'interface est détectée, jamais codée en dur. Vérifie ce que provision.sh a trouvé :

vagrant ssh k8s-cp1 -c "cat /etc/kubeadm-lab/node.env"
vagrant ssh k8s-cp1 -c "sudo sed -n '/vrrp_instance/,\$p' /etc/keepalived/keepalived.conf"

Si HOSTONLY_IF est retombé sur eth1 alors que la VM utilise vraiment enp0s8, keepalived s'attache à une interface qui n'existe pas. Relance vagrant provision k8s-cp1 une fois que la VM a son adresse host-only.

Cause 2 : l'apiserver lui-même ne démarre pas.

vagrant ssh k8s-cp1 -c "sudo crictl ps -a | grep apiserver"
vagrant ssh k8s-cp1 -c "sudo journalctl -u kubelet -n 50 --no-pager"
vagrant ssh k8s-cp1 -c "sudo crictl logs \$(sudo crictl ps -a -q --name kube-apiserver | head -1)"

Un apiserver en CrashLoopBackOff, c'est presque toujours etcd en dessous (voir la section 5). À noter : le contrôle de santé de keepalived ne retire que 30 points de priorité, jamais la VIP, donc la VIP debout ne prouve rien sur l'apiserver.

La VIP est portée par DEUX nodes à la fois (split-brain VRRP)#

kubectl se comporte de façon erratique : une requête passe, la suivante expire. Le journal affiche Entering MASTER STATE sur deux nodes.

for n in k8s-cp1 k8s-cp2 k8s-cp3; do
  echo -n "$n: " ; vagrant ssh "$n" -c "ip -4 -o addr show | grep -c 192.168.56.5" -- -q
done
# sain : exactement un node répond 1, les autres 0

VRRP est ici en unicast (unicast_src_ip + unicast_peer), pas en multicast, parce que le multicast est la première chose à mal se comporter sur un switch host-only VirtualBox. Un control plane qui ne voit pas ses pairs se croit seul et se promeut.

# la liste des pairs doit contenir toutes les AUTRES IP de control plane
vagrant ssh k8s-cp1 -c "sudo sed -n '/unicast/,/}/p' /etc/keepalived/keepalived.conf"

# le router ID doit être IDENTIQUE sur tous les control planes
for n in k8s-cp1 k8s-cp2 k8s-cp3; do
  vagrant ssh "$n" -c "sudo sed -n 's/.*virtual_router_id //p' /etc/keepalived/keepalived.conf" -- -q
done

Trois causes, par ordre de probabilité :

1. Un bloc unicast_peer manquant — keepalived ne rejette pas une telle config, il retombe en silence sur le multicast, et les deux modes sont mutuellement sourds. vagrant provision <node> la réécrit (la config liste toujours les cinq IP de control plane que le plan d'adressage autorise, donc une config écrite pour un CP est déjà correcte pour trois).
2. VRRP_ROUTER_ID divergent — nodes provisionnés avec des lab.env différents. Tous les control planes d'un cluster doivent partager le même ID.
3. Un autre lab keepalived sur le même réseau host-only avec le même ID (défaut 51). Change VRRP_ROUTER_ID, puis vagrant provision.

ℹ️ Il n'y a volontairement aucun mot de passe VRRP : l'authentification VRRPv2 l'envoie en clair et n'apporte rien. La frontière de confiance est le réseau host-only ; le bouton d'isolation est VRRP_ROUTER_ID.

☸️ 3. Nodes et kubeadm#

Les nodes restent NotReady, CoreDNS reste Pending#

NAME      STATUS     ROLES           AGE   VERSION
k8s-cp1   NotReady   control-plane   2m    v1.36.3
k8s-w1    NotReady   worker          1m    v1.36.3

Normal entre cluster-up.sh et l'étape plateforme. kubeadm n'installe pas de CNI, et un node sans réseau de pods ne passe jamais Ready. CoreDNS suit : chaque node porte le taint node.kubernetes.io/not-ready, qu'il ne tolère pas.

kubectl describe node k8s-cp1 | sed -n '/Conditions:/,/Addresses:/p'
# Ready False — KubeletNotReady — cni plugin not initialized

Remède : ./_k8s/platform-up.sh. Avec CNI=none, rien n'installera jamais de réseau : c'est le sens du réglage, et cluster-up.sh affiche un message de fin différent dans ce cas.

Toujours NotReady après l'installation du CNI, ou CoreDNS toujours Pending après que les nodes sont Ready :

kubectl -n kube-system get pods -l k8s-app=cilium -o wide
kubectl -n kube-system logs ds/cilium --tail=50
kubectl -n kube-system logs deploy/cilium-operator --tail=50

WORKERS=0 avec UNTAINT_CP=false ne laisse également nulle part où planifier.

Tous les nodes affichent la même IP, 10.0.2.15#

# NAME      INTERNAL-IP
# k8s-cp1   10.0.2.15
# k8s-w1    10.0.2.15

Chaque VM a deux cartes : NIC1 = NAT VirtualBox (toujours 10.0.2.15, identique sur toutes les VM) et NIC2 = host-only (la vraie adresse du cluster). Sans kubeletExtraArgs: node-ip, le kubelet prend l'interface de la route par défaut, celle du NAT. kubectl get nodes paraît crédible, mais les logs, exec, les sondes et le trafic inter-nodes partent au mauvais endroit.

Le lab pose node-ip dans ses trois templates : tu ne rencontres donc ça que sur un node joint à la main avec la ligne kubeadm join imprimée, qui ne peut pas porter node-ip.

vagrant ssh k8s-w1 -c "cat /var/lib/kubelet/kubeadm-flags.env"
# attendu : KUBELET_KUBEADM_ARGS="… --node-ip=192.168.56.101 …"

Remède supporté : refaire la jonction par le dépôt (./kubeadm/cluster-reset.sh && ./kubeadm/cluster-up.sh). Pour réparer un seul node, ajoute --node-ip=<IP host-only> à /var/lib/kubelet/kubeadm-flags.env puis systemctl restart kubelet ; si INTERNAL-IP ne change pas, kubectl delete node k8s-w1 pour que le kubelet se réenregistre.

kubeadm join échoue sur un token ou une clé de certificats expirés#

Remède facile : relancer ./kubeadm/cluster-up.sh. Il est idempotent, et node-init.sh régénère les deux éléments à chaque passage avant de réécrire _out/join.env. Joindre un node des jours après l'init initial est un parcours supporté.

À la main, si tu conduis kubeadm toi-même (les deux se rejouent sans risque sur un cluster vivant) :

vagrant ssh k8s-cp1 -c "sudo kubeadm init phase upload-certs --upload-certs \\
     --config /vagrant/_out/kubeadm-init.yaml"                                  # nouvelle clé
vagrant ssh k8s-cp1 -c "sudo kubeadm token create --print-join-command"        # token + hash CA

⚠️ Lance upload-certs avec --config. Sans lui, kubeadm construit son client d'API depuis un LocalAPIEndpoint.AdvertiseAddress qu'il détecte sur la route par défaut (10.0.2.15 dans n'importe quelle VM Vagrant), et TLS échoue sur x509: certificate is valid for …, not 10.0.2.15. C'est l'endpoint qu'il faut corriger : n'ajoute jamais 10.0.2.15 aux certSANs, cette adresse n'identifie aucun node.

Le preflight kubeadm se plaint du swap, du nombre de CPU ou de la mémoire#

[ERROR Swap]: swap is enabled; production deployments should disable swap …
[ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
[ERROR Mem]: the system RAM (1024 MB) is less than the minimum 1700 MB

Le swap est déjà traité par provision.sh : swapoff -a, la ligne /etc/fstab commentée, et toute unité systemd de swap masquée (Debian 13 peut fournir du swap par une unité que /etc/fstab ne mentionne jamais ; c'est comme ça que le swap revient après un redémarrage). L'erreur qui apparaît quand même signifie que le provisioning n'est pas allé au bout :

vagrant ssh k8s-cp1 -c "free -m ; swapon --show ; systemctl list-unit-files --type=swap"
vagrant provision k8s-cp1

Les seuils CPU et mémoire sont ceux de kubeadm : 2 vCPU et ~1700 Mio sur un control plane. Les défauts du dépôt les passent, donc ça ne mord qu'après les avoir baissés dans lab.env. Les ressources changent au redémarrage de la VM : vagrant reload k8s-cp1.

ℹ️ NodeSwap est GA depuis 1.34, mais failSwapOn vaut toujours true par défaut : le kubelet refuse de démarrer avec du swap actif tant que tu ne le configures pas explicitement. Sur un lab, couper le swap est le chemin le plus court et le mieux testé.

Un avertissement de preflight sur RuntimeConfig ou le pilote cgroup#

Un avertissement, pas une erreur : kubeadm n'a pas pu lire le pilote cgroup depuis le runtime et est retombé sur le champ cgroupDriver de KubeletConfiguration. Seul containerd 2.x implémente la méthode CRI RuntimeConfig qu'il utilise ; Debian 13 livre 1.7.24, qui ne l'aura jamais (backport refusé en amont, containerd#11346).

vagrant ssh k8s-cp1 -c "containerd --version"
vagrant ssh k8s-cp1 -c "sudo grep SystemdCgroup /etc/containerd/config.toml"   # doit être true

Avec CONTAINERD_SOURCE=docker (le défaut), l'avertissement disparaît. Avec CONTAINERD_SOURCE=debian il est attendu : inoffensif en 1.36, fatal en 1.37 où le repli est retiré : cette valeur est une option pour lab hors-ligne et une impasse pour les montées de version.

⚠️ Ce qui compte vraiment, c'est SystemdCgroup = true. Debian 13 est en cgroup v2 avec systemd comme gestionnaire ; laisser containerd en cgroupfs met deux gestionnaires en concurrence sur la même hiérarchie et les nodes deviennent instables sous charge.

🔌 4. Réseau de pods et Services#

Après un cluster-reset.sh, le réseau de pods se comporte de façon inexplicable#

Les pods obtiennent des IP mais le trafic inter-nodes meurt ; le DNS échoue alors que ping 1.1.1.1 fonctionne ; l'agent Cilium se plaint de maps BPF préexistantes.

kubeadm reset laisse volontairement ce qu'il n'a pas posé : interfaces CNI, programmes eBPF épinglés, et règles iptables de kube-proxy, donc un kubeadm init ultérieur hérite d'un datapath fantôme. kubeadm/node-reset.sh est ce nettoyage et cluster-reset.sh le lance partout : il retire /etc/cni/net.d/*, les interfaces cilium_*/flannel.1/cni0/vxlan.calico/kube-ipvs0/lxc*/cali*, les programmes épinglés sous /sys/fs/bpf/tc/globals/cilium_*, les chaînes KUBE-/CILIUM_/cali- et IPVS, puis efface /var/lib/etcd, /var/lib/cni, /run/flannel et redémarre containerd.

Vérifie ce qui reste sur un node suspect :

vagrant ssh k8s-w1 -c "ip -o link show | grep -E 'cilium|lxc|flannel|cali|cni0'"
vagrant ssh k8s-w1 -c "sudo ls /sys/fs/bpf/tc/globals/ 2>/dev/null"
vagrant ssh k8s-w1 -c "sudo iptables-save | grep -cE 'KUBE-|CILIUM_|cali-'"

Tout ce qui n'est pas vide signifie que le nettoyage n'est pas allé au bout : le script affiche partial reset on <node> — carrying on plutôt que de s'arrêter. Relance-le là : vagrant ssh k8s-w1 -c "sudo bash /vagrant/kubeadm/node-reset.sh". Dans le doute, vagrant destroy -f && vagrant up est la table rase garantie.

ℹ️ cluster-reset.sh est aussi le bon outil pour changer POD_CIDR, SERVICE_CIDR, le CNI ou la VIP : les quatre sont figés au moment du kubeadm init.

Un Service LoadBalancer reste <pending>#

Cause 1 : le CNI n'est pas Cilium. Seul Cilium distribue des IP de Service ici (annonce L2/ARP). Calico a besoin de BGP et il n'y a pas de routeur pair sur un réseau host-only (MetalLB requis) ; flannel et none ne font rien.

sed -n 's/^CNI=//p' _out/cluster.env      # avec quoi le cluster a réellement été construit

⚠️ _out/cluster.env est la vérité (écrit au bootstrap) ; lab.env n'est qu'une intention et a peut-être été édité après.

Cause 2 : le pool L2 est absent, épuisé ou annoncé sur la mauvaise interface.

kubectl get ciliumloadbalancerippool
kubectl get ciliuml2announcementpolicy
kubectl -n kube-system logs deploy/cilium-operator --tail=50
sed -n 's/^HOSTONLY_IF=//p' _out/cluster.env

Le pool vaut 192.168.56.200–.230 par défaut, et l'interface d'annonce vient du HOSTONLY_IF détecté. Un pool qui chevauche la plage des nodes, ou une politique épinglée à une interface inexistante, donnent tous deux un <pending> permanent. Changer le pool tient en une relance : ./_k8s/cilium/cilium-up.sh.

🗄️ 5. etcd et performances du cluster#

etcd perd son leader, ou tout le cluster rampe#

etcdserver: request timed out
apply request took too long
waiting for ReadIndex response took too long, retrying
leader changed

kubectl met des secondes à répondre, les pods restent Pending, l'apiserver redémarre seul.

kubectl get --raw='/healthz/etcd'
kubectl -n kube-system logs -l component=etcd --tail=50
vagrant ssh k8s-cp1 -c "free -m ; uptime"

Causes, par ordre de fréquence sur ce lab :

1. Latence de fsync. etcd valide chaque écriture sur disque avant d'accuser réception. Sous VirtualBox, un disque de VM sur un plateau tournant (ou sur un SSD déjà saturé par l'hôte) pousse le fsync au-delà de la tolérance d'etcd et l'élection de leader se met à osciller. Garde les disques des VM sur SSD, et ne lance pas une topologie à 3 control planes à côté d'un build lourd.
2. CP_MEM trop bas. Un etcd empilé sur 2048 Mio a ~350 Mio de marge ; les premiers addons la mangent. 3072 est le vrai plancher, _k8s/observability/ demande 4096.
3. Dérive d'horloge. etcd y est très sensible. Le Vagrantfile abaisse le seuil de synchronisation des additions invité à 1000 ms, ce qui couvre un cycle suspend/resume, mais une VM laissée longtemps suspendue gagne à être passée en vagrant reload.

⚠️ Avec 3 control planes, etcd tolère une panne. N'en arrête pas deux en même temps (y compris pendant une montée de version, voir kubeadm/MISE-A-JOUR.md) : l'API gèle jusqu'au retour du quorum.

🔐 6. Les UI du lab en HTTPS#

Descends la chaîne dans l'ordre ; chaque étape suppose la précédente.

1. Le Gateway a-t-il une IP ?

kubectl -n envoy-gateway-system get gateway main-gateway -o jsonpath='{.status.addresses[0].value}'; echo

Vide ou <pending> → problème de LoadBalancer, voir la section 4. L'adresse attendue est la première IP du pool, 192.168.56.200 par défaut.

2. Le nom résout-il vers cette IP ?

LAB_DOMAIN n'a aucune raison de résoudre sur ta machine. platform-up.sh affiche la ligne à ajouter :

# /etc/hosts sur l'HÔTE
192.168.56.200  argo.kubeadm.lab.example.io grafana.kubeadm.lab.example.io

…ou un enregistrement A wildcard *.<LAB_DOMAIN> → 192.168.56.200 si tu possèdes une zone DNS (en DNS-only derrière Cloudflare : le proxy ne peut pas joindre une IP privée).

getent hosts argo.kubeadm.lab.example.io

⚠️ Ne teste pas l'IP du Gateway avec ping. Une IP de Service annoncée en L2 par Cilium répond à l'ARP et au TCP, mais pas à l'ICMP : aucune interface ne porte réellement l'adresse. Un ping qui échoue sur .200 est normal et ne prouve rien, alors que le ping d'un node fonctionne, ce qui rend le faux négatif convaincant. La vraie preuve de l'annonce, c'est l'entrée ARP qui se résout vers la MAC d'un node :

sudo ip neigh flush 192.168.56.200
curl -s -o /dev/null --max-time 5 http://192.168.56.200/    # 404 = Envoy répond
ip neigh show 192.168.56.200                                # lladdr = le node annonceur

3. Existe-t-il une HTTPRoute pour ce nom d'hôte ?

kubectl get httproute -A
kubectl -n <ns> describe httproute <nom> | sed -n '/Status:/,$p'   # Accepted / ResolvedRefs

ℹ️ Sur l'IP nue, http:// répond 404 (Envoy écoute, aucune route ne correspond) mais https:// ne répond rien du tout : le listener TLS est délimité par nom d'hôte, donc une requête sans SNI ne correspond à aucun listener. Teste avec le nom, en court-circuitant le DNS au besoin : curl -sk --resolve argo.kubeadm.lab.example.io:443:192.168.56.200 https://argo.kubeadm.lab.example.io/.

4. Le mode TLS est-il celui que tu crois ?

sed -n 's/^SELF_SIGNED=//p' lab.env
kubectl -n envoy-gateway-system get secret | grep wildcard

Un avertissement de navigateur est donc normal dans deux modes sur trois. Pour faire confiance à l'AC locale :

sudo cp _out/self-signed/ca.crt /usr/local/share/ca-certificates/vagrant-kubeadm-lab.crt
sudo update-ca-certificates

5. Toujours rien ? Regarde le proxy lui-même :

kubectl -n envoy-gateway-system get pods
kubectl -n envoy-gateway-system logs deploy/envoy-gateway --tail=50

🧰 7. Boîte à outils#

Depuis l'hôte#

vagrant status                       # quelles VM existent et tournent
vagrant ssh k8s-cp1                  # shell interactif
vagrant ssh k8s-cp1 -c "<commande>" -- -q -o LogLevel=ERROR   # one shot, silencieux (ce que font les scripts)
vagrant provision k8s-cp1            # rejoue provision.sh (idempotent)
vagrant reload k8s-cp1               # redémarre en appliquant les CPU/RAM de lab.env

export KUBECONFIG="$PWD/kubeconfig"
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl get events -A --sort-by=.lastTimestamp | tail -30
kubectl get --raw='/readyz?verbose'

cat _out/cluster.env                 # avec quoi le cluster a VRAIMENT été construit

⚠️ _out/join.env contient le token de jonction et la clé de certificats. _out/ est gitignoré, mais lisible par toutes les VM via le dossier synchronisé /vagrant. Ne colle jamais son contenu nulle part.

Dans une VM#

cat /etc/kubeadm-lab/node.env                 # rôle, IP du node, interface host-only détectée
ip -4 addr show                               # la VIP est-elle ici ?
sudo systemctl status kubelet containerd keepalived

sudo journalctl -u kubelet -f
sudo journalctl -u containerd -n 50 --no-pager
sudo journalctl -u keepalived -n 50 --no-pager

sudo crictl ps -a                             # conteneurs, morts inclus
sudo crictl logs <container-id>

sudo kubeadm certs check-expiration           # control planes seulement
sudo kubeadm config images list --kubernetes-version v1.36.3

💡 crictl parle au même socket que le kubelet grâce à /etc/crictl.yaml, écrit par provision.sh. Sans lui, crictl cherche dockershim et affiche des erreurs déroutantes.

Les options nucléaires, de la moins à la plus destructrice#

📚 Références#

• kubeadm — Troubleshooting
• kubeadm — Configuring a cgroup driver
• Cilium — Troubleshooting
• etcd — Tuning
• kubeadm/MISE-A-JOUR.md — montées de version et renouvellement des certificats
• k8s-playground — la couche applicative _k8s/, addon par addon, avec ses propres sections de pièges

🤖CLAUDE.md

Kubernetes built with kubeadm on Debian 13 VMs, on VirtualBox, driven by Vagrant. Unlike the Talos sibling of this lab, the nodes are ordinary Linux boxes: SSH, apt, systemd, journalctl all work, and every step is a kubeadm command you could type by hand. User docs: README.md · application layer: https://ops-nc.github.io/k8s-playground/ · symptoms: TROUBLESHOOTING.md · version bumps: kubeadm/UPGRADE.md.

🚫 There is NO cluster, and you must not try to build one#

No agent working in this repository runs vagrant, kubectl, helm or talosctl. There is no running lab attached to your session, kubeconfig does not exist, and vagrant up would spend fifteen minutes failing. Every claim you make must be backed by reading the code, not by running it. The one thing you may and should run is make validate — see below.

If a change can only be proven by a live cluster, say so explicitly and hand the verification back to the human, with the exact commands to run.

🚫 _k8s/ is a SUBMODULE — never edit it from here#

_k8s/ is not a directory of this repository. It is a pinned checkout of OPS-NC/k8s-playground, the application layer shared with the Talos sibling lab: one source, one place to maintain it. Its documentation is published separately at https://ops-nc.github.io/k8s-playground/.

• Read it freely to understand how the layer behaves — it is checked out on disk.
• Never write to it. Editing a file under _k8s/ dirties another repository's working tree and produces a commit that does not belong here. Addon changes are made in k8s-playground, then this repo bumps the pointer.
• Never link to _k8s/…*.md from a Markdown file of this repo. Those pages are not part of this documentation set and make validate-docs fails on the dead link. Point at https://ops-nc.github.io/k8s-playground/, or at the file on GitHub (https://github.com/OPS-NC/k8s-playground/blob/main/<dir>/README.md — the directories sit at the root of that repo, with no _k8s/ prefix).
• Paths on disk (./_k8s/platform-up.sh, _k8s/cilium/cilium-up.sh) stay correct and must not be rewritten: the submodule really does mount there.
• If _k8s/ is empty on the machine you work on, the submodule was never initialised: git submodule update --init --recursive. git pull alone does not update it.

🚀 Order of work#

lab.env  ──────────────► Vagrantfile ──► kubeadm/provision.sh        (in each VM, at `vagrant up`)
   │  (single source)         │
   │                          └─ 8 steps: /etc/hosts · system upgrade (SYSTEM_UPGRADE) ·
   │                             kernel prereqs · base packages · containerd ·
   │                             kubelet/kubeadm/kubectl (+ hold) · image pull ·
   │                             keepalived (control planes only)
   │
   ├──────────────────► kubeadm/cluster-up.sh                        (on the HOST)
   │                          ├─ renders kubeadm/templates/*.tpl into _out/
   │                          ├─ vagrant ssh cp1 → kubeadm/node-init.sh   (kubeadm init)
   │                          ├─ vagrant ssh others → kubeadm/node-join.sh (kubeadm join)
   │                          └─ writes ./kubeconfig and _out/cluster.env
   │
   └──────────────────► _k8s/platform-up.sh                           (on the HOST, SUBMODULE)
                              ├─ [1/4] CNI  → _k8s/cilium/cilium-up.sh (or calico/flannel/none)
                              ├─ [2/4] Envoy Gateway + main-gateway
                              ├─ [3/4] metrics-server
                              └─ [4/4] wildcard TLS → _k8s/self-signed/ or cert-manager
                                    then opt-in addons: ./_k8s/install.sh <addon>…

The application-layer entry point takes no distribution and no environment: it locates the lab (the directory that contains _k8s/ and carries the Vagrantfile) and reads the distribution off it — see the section below. The full sequence from the host:

./_k8s/platform-up.sh

Undo without destroying the VMs: kubeadm/cluster-reset.sh → kubeadm/node-reset.sh in every VM (workers first, so they deregister while the API still answers).

What lives where#

The files that carry state between layers#

The synced folder /vagrant is a mechanism, not a convenience: it is what removes every scp and every secret passed on a command line. Keep it that way.

🔑 The golden rule: lab.env is the single source, and its defaults are DUPLICATED#

Precedence, everywhere: real environment variable > lab.env > in-file fallback default. That is why WORKERS=5 vagrant up works, and why lab.env never has to be export-ed.

The fallback defaults exist so a freshly cloned repo works without a lab.env. They are deliberately copied into several files:

⚠️ Two defaults that diverge produce an incoherent lab — packages from one minor, generated configuration for another; a pod CIDR declared to kubeadm that the CNI does not announce; a wildcard Secret name the Gateway does not look for. Changing a default means changing it everywhere in the same commit, lab.env.example included.

⚠️ The last three rows straddle two repositories. Their consumers live in k8s-playground, which this repo only pins. A default changed here and not there (or the reverse) diverges silently — make validate-defaults only compares lab.env.example, the Vagrantfile and kubeadm/cluster-up.sh, and cannot see across the submodule. Changing one of those keys means a PR in both repos, and bumping the pointer here.

The k8s-playground *-up.sh scripts add one more layer, and the order matters: _out/cluster.env (facts about the running cluster) wins over lab.env (a mere intent, possibly edited after the bootstrap). cilium/cilium-up.sh implements this in lire_param. Both files are found in the lab directory, resolved automatically — see the section below.

✅ Validating a change WITHOUT a cluster (do this every time)#

make validate      # shell syntax + YAML parse + Vagrantfile + kubeadm templates + doc links
make docs          # regenerates docs/index.html (needs uv)

make docs also lists, at the end of the build, every link and cross-file anchor that does not resolve. Run it after renaming any heading.

⚠️ Design pitfalls — do NOT reintroduce these#

Shell#

• grep under set -e + pipefail kills the script. A grep with no match exits 1, and in a pipeline under pipefail that becomes the script's exit status — silently, long before the interesting part. The repo reads key/value files with sed -n 's/^KEY=//p' and a trailing || true (for the case where the file does not exist at all, where sed exits 2). Look at lire_lab_env / lire_cluster_env and copy them; never introduce a grep … | head -1 in that role.
• Backticks inside a double-quoted string are command substitution. Writing echo "use `kubeadm init`" runs kubeadm init. This repo's prose is full of backtick-quoted identifiers, so the risk is constant in echo/printf messages and in unquoted heredocs (<<EOF). Use '…', a quoted heredoc (<<'EOF'), or simply no backticks in shell output.
• ./script.sh; echo "EXIT=$?" reports echo's status, not the script's. Check ${PIPESTATUS[0]} or the exit line inside the log.
• lab.env is parsed, not sourced. Strict KEY=value, no spaces around =, no ;. The key name is validated against ^[A-Za-z_][A-Za-z0-9_]*$ before any eval: a hand-edited lab.env must not be able to execute arbitrary code. Keep that check if you touch the parser.

kubeadm#

• node-ip is mandatory on every node. Each VM has a NAT NIC at 10.0.2.15, identical on every VM. Without kubeletExtraArgs: node-ip, every node registers with that address and logs, exec, probes and cross-node traffic all go to the wrong place. This is the reason the lab joins nodes through JoinConfiguration files instead of the printed kubeadm join line: that line cannot carry node-ip, and kubeadm join has no equivalent flag. Never "simplify" the join back to the printed command.
• v1beta4: extraArgs and kubeletExtraArgs are LISTS, not dictionaries.

  # v1beta3 — invalid now
  extraArgs: {bind-address: "0.0.0.0"}
  # v1beta4 — correct
  extraArgs:
    - name: bind-address
      value: "0.0.0.0"
  

  The change exists so a flag can be repeated. Any pre-1.31 snippet copied from the internet is invalid, and the error message does not say so. make validate-kubeadm catches it.
• The host-only interface name is never hard-coded. provision.sh finds the interface that carries the node's IP and writes it to /etc/kubeadm-lab/node.env; cluster-up.sh copies it into _out/cluster.env as HOSTONLY_IF; Cilium's L2 announcement and keepalived both use it. Debian 13 usually gives enp0s8, some boxes still give eth1. Writing either literally anywhere is a bug.
• The pause image is never hard-coded either. provision.sh asks kubeadm config images list for it. A mismatch between containerd's pinned image and the one kubeadm expects is invisible online and fatal offline.
• controlPlaneEndpoint is the VIP even with one control plane. It is frozen in the certificates and in every kubeconfig at kubeadm init time; pointing it at cp1's real IP would make "1 CP → 3 CP" a full certificate regeneration instead of a join.
• certSANs pre-declares five control-plane IPs, including nodes that do not exist yet. A forgotten SAN can only be added by regenerating the certificates. Do not trim that list.
• --skip-phases=addon/kube-proxy is preferred to v1beta4's declarative proxy.disabled — identical result, but the flag is proven across versions and is what Cilium documents.
• KUBE_PROXY_REPLACEMENT=true requires CNI=cilium, and cluster-up.sh refuses any other combination. Without kube-proxy and without a replacement, no ClusterIP answers at all — not even CoreDNS reaching the API. Keep the refusal; do not downgrade it to a warning.
• Cilium needs k8sServiceHost/k8sServicePort = the VIP when kube-proxy is gone: nothing provisions the apiserver ClusterIP, so the agent cannot bootstrap through it. And Cilium's cluster-pool IPAM defaults to 10.0.0.0/8, unrelated to what kubeadm was told — that is why cilium-up.sh passes POD_CIDR explicitly.
• kubeadm reset leaves the CNI datapath behind — interfaces, pinned eBPF programs under /sys/fs/bpf, kube-proxy iptables rules. node-reset.sh is that cleanup; without it a later init inherits a ghost datapath. Do not slim it down.
• apt-mark hold on kubelet/kubeadm/kubectl is deliberate, and vagrant provision is not an upgrade path: it would jump every node's kubelet to a new minor at once, ahead of the control plane. See kubeadm/UPGRADE.md.
• Control planes must be odd, and the check exists in two places (Vagrantfile and cluster-up.sh) on purpose. CI asserts that the Vagrantfile really refuses CONTROL_PLANES=2 — a test that checks an error happens beats a comment claiming it does.
• containerd's config is regenerated from containerd config default on every provision, so the file format follows the installed binary. The pause key changed name and location between formats (sandbox_image under [plugins."io.containerd.grpc.v1.cri"] in v2, sandbox under [plugins.'io.containerd.cri.v1.images'.pinned_images] in v3) — both are patched. Never hand-edit that file and expect it to survive.
• keepalived, not kube-vip, and no authentication block. The VIP must exist before kubeadm init; kube-vip elects its leader through the API it is meant to front. VRRP is unicast (multicast misbehaves on a VirtualBox host-only switch), and the isolation knob is VRRP_ROUTER_ID, not a cleartext VRRPv2 password.

The _k8s/ submodule#

• The lab is found on its own — no LAB_DIR in the examples. k8s-playground takes the directory that contains _k8s/ as the lab, provided it carries a Vagrantfile; that is where lab.env, _out/ and kubeconfig live. In the submodule layout that is this repo, so documented commands are bare: ./_k8s/platform-up.sh. LAB_DIR (or LAB_ENV) stays documented as an escape hatch for an unusual layout, never as a required step — do not re-add it to the normal path.
• The distribution is detected, not passed. A kubeadm/cluster-up.sh next to the Vagrantfile identifies the kubeadm lab (talos/cluster-up.sh the Talos twin), from the clone alone, before any vagrant up. An explicit kubeadm argument (or --distro= / K8S_DISTRO) is still accepted and wins, but it is a possibility to mention, not the documented invocation. There is no DISTRO key in lab.env any more.
• Do not edit anything under _k8s/ from this repo, and do not link to its *.md files. See the dedicated section at the top of this file.
• make validate-shell / validate-yaml only cover files tracked by this repo. The submodule's scripts and manifests are validated in k8s-playground's own CI, not here. A green make validate says nothing about the application layer.

🔐 Secrets#

• lab.env is gitignored and may hold real secrets (CLOUDFLARE_API_TOKEN). Never commit it, never copy a value from it into a README, a commit message, a report or terminal output.
• _out/join.env holds the join token and the certificate key; _out/admin.conf and ./kubeconfig hold admin credentials; _out/self-signed/ca.key is a private CA key. _out/ is gitignored but readable by every VM through /vagrant.
• The repo is public: every versioned default must be neutral (kubeadm.lab.example.io, empty CLOUDFLARE_API_TOKEN, empty REGISTRY_MIRROR).
• Before committing: git status. No secret file may appear.

📝 Conventions#

• Bilingual docs, English first. Every page exists twice in the same directory: English carries the canonical name, French its mirror.

  English	French
  README.md	LISEZ-MOI.md
  TROUBLESHOOTING.md	DEPANNAGE.md
  UPGRADE.md	MISE-A-JOUR.md

  Both versions change in the same commit: an English page whose mirror did not follow is a documentation bug. This file is the exception — English only, on purpose, because it addresses coding agents (it is listed in WITHOUT_MIRROR in docs/build.py, so it carries no "not translated" badge).

• Every page starts with the i18n banner, which docs/build.py strips at build time (the HTML page has its own switcher). Keep it in the files, and put nothing else between the markers:

• EN and FR must share the exact same heading structure, in the same order: the site's language switcher keeps the current anchor when it toggles. Slugs derive from headings, so FR anchors differ from EN anchors by construction — which means renaming a heading breaks every link that targeted it, and make validate-docs is what tells you.

• docs/build.py discovers pages on its own: every *.md in the repo is picked up. Adding a page needs no code change; only its menu group (GROUPS) and its emoji (EMOJIS) are declared, and an unknown directory falls into "Other". Pages are grouped per directory through MIRRORS; a page with no mirror is shown in English inside the French menu with an EN badge — that badge is the symptom of a forgotten translation.

• Everything that is not a French documentation page is in English. Code comments, identifiers, script output, error messages, Makefile, CI workflows, .gitignore, Vagrantfile, lab.env.example, the kubeadm/templates/*.tpl and docs/build.py — all English. The repo used to keep its comments in French; that is no longer the case, so do not "restore" French in a script you touch.

• The only French left is the FR documentation mirrors (LISEZ-MOI.md, DEPANNAGE.md, kubeadm/MISE-A-JOUR.md) — their prose, not the output they quote. Three deliberate exceptions inside otherwise English code, all in docs/build.py:

  • the fr values of LABELS (they are the French UI);
  • the FR menu titles of GROUPS and of OTHER, same reason;
  • the French markers of the CALLOUTS table ("attention", "jamais", "astuce", "conseil", "remarque"…). These are not labels, they parse the French pages to pick a callout's colour. Translating them silently turns every French callout grey — the kind of breakage no test catches. The callout kinds (danger/tip/info) are English because they become CSS classes (.callout-tip).

• When a French page quotes script output, quote the English string the script now prints. A French page documenting an English-output tool is the expected result, not an oversight.

• ⚠️ .github/workflows/ci.yml greps the WORDING of a Vagrantfile error. The CONTROL_PLANES=2 guard-rail test matches 'is EVEN'. Reword that message and the test still passes while proving nothing — change both together.

• Commit messages in English, conventional (fix(...), feat(...), docs: ...). Branch from main, one feature per PR, squash merge.

• Every page of this repo follows the same skeleton (one emoji per ##, ⚠️/💡/ℹ️ callouts, a pitfalls section where it applies). Stick to plain CommonMark + GitHub tables so the generator renders it. The addon pages follow the same convention in k8s-playground, where they are written and published.

• No Markdown link may point into _k8s/. docs/build.py --strict resolves *.md links and anchors, the submodule's pages are not part of this documentation set, and make validate-docs fails on them. Use https://ops-nc.github.io/k8s-playground/ or a GitHub URL instead.

Adding a component = propagating it EVERYWHERE#

A variable, an option or an addon is only "done" once it appears at every level. One isolated mention is a documentation bug — the reader will never find it.

Then make docs, then make validate, before committing.

🧭 What is deliberately absent from this repo#

Knowing what is not here saves you from "adding" it back.

• No talosctl, no immutable OS, no API-driven machine config. The nodes are plain Debian; that is the entire point of this repo next to its Talos sibling.
• No kube-vip. keepalived carries the VIP because the VIP must pre-date kubeadm init. kube-vip stays a legitimate option once the cluster is up (--services mode) — worth mentioning, never the default path.
• No MetalLB. Cilium's L2/ARP announcement gives LoadBalancer Services their IP. MetalLB is only relevant on the CNI=calico branch, and that is documented with the calico/ addon in k8s-playground.
• No cluster bootstrap inside vagrant up. The Vagrantfile prepares VMs and stops there. Bootstrapping is a separate, re-runnable script — that separation is what makes growing the lab a re-run instead of a rebuild.
• No external etcd. Stacked etcd on the control planes: kubeadm's default, and the right call for a lab.
• No ingress-nginx. Gateway API through Envoy Gateway.
• No cert-manager by default. SELF_SIGNED=true builds a local CA with openssl, works offline, and burns no Let's Encrypt quota. Both TLS modes fill the same Secret, so no addon ever branches on the TLS mode — keep it that way.
• No CI that boots a VM or talks to a cluster. Everything CI does is a make validate-* target that also runs on a laptop. A check that passes in CI and fails locally is a broken check.
• No committed lab.env, _out/, kubeconfig or docs/index.html. All generated, all gitignored.

🤖CLAUDE.md

Kubernetes built with kubeadm on Debian 13 VMs, on VirtualBox, driven by Vagrant. Unlike the Talos sibling of this lab, the nodes are ordinary Linux boxes: SSH, apt, systemd, journalctl all work, and every step is a kubeadm command you could type by hand. User docs: README.md · application layer: https://ops-nc.github.io/k8s-playground/ · symptoms: TROUBLESHOOTING.md · version bumps: kubeadm/UPGRADE.md.

🚫 There is NO cluster, and you must not try to build one#

No agent working in this repository runs vagrant, kubectl, helm or talosctl. There is no running lab attached to your session, kubeconfig does not exist, and vagrant up would spend fifteen minutes failing. Every claim you make must be backed by reading the code, not by running it. The one thing you may and should run is make validate — see below.

If a change can only be proven by a live cluster, say so explicitly and hand the verification back to the human, with the exact commands to run.

🚫 _k8s/ is a SUBMODULE — never edit it from here#

_k8s/ is not a directory of this repository. It is a pinned checkout of OPS-NC/k8s-playground, the application layer shared with the Talos sibling lab: one source, one place to maintain it. Its documentation is published separately at https://ops-nc.github.io/k8s-playground/.

• Read it freely to understand how the layer behaves — it is checked out on disk.
• Never write to it. Editing a file under _k8s/ dirties another repository's working tree and produces a commit that does not belong here. Addon changes are made in k8s-playground, then this repo bumps the pointer.
• Never link to _k8s/…*.md from a Markdown file of this repo. Those pages are not part of this documentation set and make validate-docs fails on the dead link. Point at https://ops-nc.github.io/k8s-playground/, or at the file on GitHub (https://github.com/OPS-NC/k8s-playground/blob/main/<dir>/README.md — the directories sit at the root of that repo, with no _k8s/ prefix).
• Paths on disk (./_k8s/platform-up.sh, _k8s/cilium/cilium-up.sh) stay correct and must not be rewritten: the submodule really does mount there.
• If _k8s/ is empty on the machine you work on, the submodule was never initialised: git submodule update --init --recursive. git pull alone does not update it.

🚀 Order of work#

lab.env  ──────────────► Vagrantfile ──► kubeadm/provision.sh        (in each VM, at `vagrant up`)
   │  (single source)         │
   │                          └─ 8 steps: /etc/hosts · system upgrade (SYSTEM_UPGRADE) ·
   │                             kernel prereqs · base packages · containerd ·
   │                             kubelet/kubeadm/kubectl (+ hold) · image pull ·
   │                             keepalived (control planes only)
   │
   ├──────────────────► kubeadm/cluster-up.sh                        (on the HOST)
   │                          ├─ renders kubeadm/templates/*.tpl into _out/
   │                          ├─ vagrant ssh cp1 → kubeadm/node-init.sh   (kubeadm init)
   │                          ├─ vagrant ssh others → kubeadm/node-join.sh (kubeadm join)
   │                          └─ writes ./kubeconfig and _out/cluster.env
   │
   └──────────────────► _k8s/platform-up.sh                           (on the HOST, SUBMODULE)
                              ├─ [1/4] CNI  → _k8s/cilium/cilium-up.sh (or calico/flannel/none)
                              ├─ [2/4] Envoy Gateway + main-gateway
                              ├─ [3/4] metrics-server
                              └─ [4/4] wildcard TLS → _k8s/self-signed/ or cert-manager
                                    then opt-in addons: ./_k8s/install.sh <addon>…

The application-layer entry point takes no distribution and no environment: it locates the lab (the directory that contains _k8s/ and carries the Vagrantfile) and reads the distribution off it — see the section below. The full sequence from the host:

./_k8s/platform-up.sh

Undo without destroying the VMs: kubeadm/cluster-reset.sh → kubeadm/node-reset.sh in every VM (workers first, so they deregister while the API still answers).

What lives where#

The files that carry state between layers#

The synced folder /vagrant is a mechanism, not a convenience: it is what removes every scp and every secret passed on a command line. Keep it that way.

🔑 The golden rule: lab.env is the single source, and its defaults are DUPLICATED#

Precedence, everywhere: real environment variable > lab.env > in-file fallback default. That is why WORKERS=5 vagrant up works, and why lab.env never has to be export-ed.

The fallback defaults exist so a freshly cloned repo works without a lab.env. They are deliberately copied into several files:

⚠️ Two defaults that diverge produce an incoherent lab — packages from one minor, generated configuration for another; a pod CIDR declared to kubeadm that the CNI does not announce; a wildcard Secret name the Gateway does not look for. Changing a default means changing it everywhere in the same commit, lab.env.example included.

⚠️ The last three rows straddle two repositories. Their consumers live in k8s-playground, which this repo only pins. A default changed here and not there (or the reverse) diverges silently — make validate-defaults only compares lab.env.example, the Vagrantfile and kubeadm/cluster-up.sh, and cannot see across the submodule. Changing one of those keys means a PR in both repos, and bumping the pointer here.

The k8s-playground *-up.sh scripts add one more layer, and the order matters: _out/cluster.env (facts about the running cluster) wins over lab.env (a mere intent, possibly edited after the bootstrap). cilium/cilium-up.sh implements this in lire_param. Both files are found in the lab directory, resolved automatically — see the section below.

✅ Validating a change WITHOUT a cluster (do this every time)#

make validate      # shell syntax + YAML parse + Vagrantfile + kubeadm templates + doc links
make docs          # regenerates docs/index.html (needs uv)

make docs also lists, at the end of the build, every link and cross-file anchor that does not resolve. Run it after renaming any heading.

⚠️ Design pitfalls — do NOT reintroduce these#

Shell#

• grep under set -e + pipefail kills the script. A grep with no match exits 1, and in a pipeline under pipefail that becomes the script's exit status — silently, long before the interesting part. The repo reads key/value files with sed -n 's/^KEY=//p' and a trailing || true (for the case where the file does not exist at all, where sed exits 2). Look at lire_lab_env / lire_cluster_env and copy them; never introduce a grep … | head -1 in that role.
• Backticks inside a double-quoted string are command substitution. Writing echo "use `kubeadm init`" runs kubeadm init. This repo's prose is full of backtick-quoted identifiers, so the risk is constant in echo/printf messages and in unquoted heredocs (<<EOF). Use '…', a quoted heredoc (<<'EOF'), or simply no backticks in shell output.
• ./script.sh; echo "EXIT=$?" reports echo's status, not the script's. Check ${PIPESTATUS[0]} or the exit line inside the log.
• lab.env is parsed, not sourced. Strict KEY=value, no spaces around =, no ;. The key name is validated against ^[A-Za-z_][A-Za-z0-9_]*$ before any eval: a hand-edited lab.env must not be able to execute arbitrary code. Keep that check if you touch the parser.

kubeadm#

• node-ip is mandatory on every node. Each VM has a NAT NIC at 10.0.2.15, identical on every VM. Without kubeletExtraArgs: node-ip, every node registers with that address and logs, exec, probes and cross-node traffic all go to the wrong place. This is the reason the lab joins nodes through JoinConfiguration files instead of the printed kubeadm join line: that line cannot carry node-ip, and kubeadm join has no equivalent flag. Never "simplify" the join back to the printed command.
• v1beta4: extraArgs and kubeletExtraArgs are LISTS, not dictionaries.

  # v1beta3 — invalid now
  extraArgs: {bind-address: "0.0.0.0"}
  # v1beta4 — correct
  extraArgs:
    - name: bind-address
      value: "0.0.0.0"
  

  The change exists so a flag can be repeated. Any pre-1.31 snippet copied from the internet is invalid, and the error message does not say so. make validate-kubeadm catches it.
• The host-only interface name is never hard-coded. provision.sh finds the interface that carries the node's IP and writes it to /etc/kubeadm-lab/node.env; cluster-up.sh copies it into _out/cluster.env as HOSTONLY_IF; Cilium's L2 announcement and keepalived both use it. Debian 13 usually gives enp0s8, some boxes still give eth1. Writing either literally anywhere is a bug.
• The pause image is never hard-coded either. provision.sh asks kubeadm config images list for it. A mismatch between containerd's pinned image and the one kubeadm expects is invisible online and fatal offline.
• controlPlaneEndpoint is the VIP even with one control plane. It is frozen in the certificates and in every kubeconfig at kubeadm init time; pointing it at cp1's real IP would make "1 CP → 3 CP" a full certificate regeneration instead of a join.
• certSANs pre-declares five control-plane IPs, including nodes that do not exist yet. A forgotten SAN can only be added by regenerating the certificates. Do not trim that list.
• --skip-phases=addon/kube-proxy is preferred to v1beta4's declarative proxy.disabled — identical result, but the flag is proven across versions and is what Cilium documents.
• KUBE_PROXY_REPLACEMENT=true requires CNI=cilium, and cluster-up.sh refuses any other combination. Without kube-proxy and without a replacement, no ClusterIP answers at all — not even CoreDNS reaching the API. Keep the refusal; do not downgrade it to a warning.
• Cilium needs k8sServiceHost/k8sServicePort = the VIP when kube-proxy is gone: nothing provisions the apiserver ClusterIP, so the agent cannot bootstrap through it. And Cilium's cluster-pool IPAM defaults to 10.0.0.0/8, unrelated to what kubeadm was told — that is why cilium-up.sh passes POD_CIDR explicitly.
• kubeadm reset leaves the CNI datapath behind — interfaces, pinned eBPF programs under /sys/fs/bpf, kube-proxy iptables rules. node-reset.sh is that cleanup; without it a later init inherits a ghost datapath. Do not slim it down.
• apt-mark hold on kubelet/kubeadm/kubectl is deliberate, and vagrant provision is not an upgrade path: it would jump every node's kubelet to a new minor at once, ahead of the control plane. See kubeadm/UPGRADE.md.
• Control planes must be odd, and the check exists in two places (Vagrantfile and cluster-up.sh) on purpose. CI asserts that the Vagrantfile really refuses CONTROL_PLANES=2 — a test that checks an error happens beats a comment claiming it does.
• containerd's config is regenerated from containerd config default on every provision, so the file format follows the installed binary. The pause key changed name and location between formats (sandbox_image under [plugins."io.containerd.grpc.v1.cri"] in v2, sandbox under [plugins.'io.containerd.cri.v1.images'.pinned_images] in v3) — both are patched. Never hand-edit that file and expect it to survive.
• keepalived, not kube-vip, and no authentication block. The VIP must exist before kubeadm init; kube-vip elects its leader through the API it is meant to front. VRRP is unicast (multicast misbehaves on a VirtualBox host-only switch), and the isolation knob is VRRP_ROUTER_ID, not a cleartext VRRPv2 password.

The _k8s/ submodule#

• The lab is found on its own — no LAB_DIR in the examples. k8s-playground takes the directory that contains _k8s/ as the lab, provided it carries a Vagrantfile; that is where lab.env, _out/ and kubeconfig live. In the submodule layout that is this repo, so documented commands are bare: ./_k8s/platform-up.sh. LAB_DIR (or LAB_ENV) stays documented as an escape hatch for an unusual layout, never as a required step — do not re-add it to the normal path.
• The distribution is detected, not passed. A kubeadm/cluster-up.sh next to the Vagrantfile identifies the kubeadm lab (talos/cluster-up.sh the Talos twin), from the clone alone, before any vagrant up. An explicit kubeadm argument (or --distro= / K8S_DISTRO) is still accepted and wins, but it is a possibility to mention, not the documented invocation. There is no DISTRO key in lab.env any more.
• Do not edit anything under _k8s/ from this repo, and do not link to its *.md files. See the dedicated section at the top of this file.
• make validate-shell / validate-yaml only cover files tracked by this repo. The submodule's scripts and manifests are validated in k8s-playground's own CI, not here. A green make validate says nothing about the application layer.

🔐 Secrets#

• lab.env is gitignored and may hold real secrets (CLOUDFLARE_API_TOKEN). Never commit it, never copy a value from it into a README, a commit message, a report or terminal output.
• _out/join.env holds the join token and the certificate key; _out/admin.conf and ./kubeconfig hold admin credentials; _out/self-signed/ca.key is a private CA key. _out/ is gitignored but readable by every VM through /vagrant.
• The repo is public: every versioned default must be neutral (kubeadm.lab.example.io, empty CLOUDFLARE_API_TOKEN, empty REGISTRY_MIRROR).
• Before committing: git status. No secret file may appear.

📝 Conventions#

• Bilingual docs, English first. Every page exists twice in the same directory: English carries the canonical name, French its mirror.

  English	French
  README.md	LISEZ-MOI.md
  TROUBLESHOOTING.md	DEPANNAGE.md
  UPGRADE.md	MISE-A-JOUR.md

  Both versions change in the same commit: an English page whose mirror did not follow is a documentation bug. This file is the exception — English only, on purpose, because it addresses coding agents (it is listed in WITHOUT_MIRROR in docs/build.py, so it carries no "not translated" badge).

• Every page starts with the i18n banner, which docs/build.py strips at build time (the HTML page has its own switcher). Keep it in the files, and put nothing else between the markers:

• EN and FR must share the exact same heading structure, in the same order: the site's language switcher keeps the current anchor when it toggles. Slugs derive from headings, so FR anchors differ from EN anchors by construction — which means renaming a heading breaks every link that targeted it, and make validate-docs is what tells you.

• docs/build.py discovers pages on its own: every *.md in the repo is picked up. Adding a page needs no code change; only its menu group (GROUPS) and its emoji (EMOJIS) are declared, and an unknown directory falls into "Other". Pages are grouped per directory through MIRRORS; a page with no mirror is shown in English inside the French menu with an EN badge — that badge is the symptom of a forgotten translation.

• Everything that is not a French documentation page is in English. Code comments, identifiers, script output, error messages, Makefile, CI workflows, .gitignore, Vagrantfile, lab.env.example, the kubeadm/templates/*.tpl and docs/build.py — all English. The repo used to keep its comments in French; that is no longer the case, so do not "restore" French in a script you touch.

• The only French left is the FR documentation mirrors (LISEZ-MOI.md, DEPANNAGE.md, kubeadm/MISE-A-JOUR.md) — their prose, not the output they quote. Three deliberate exceptions inside otherwise English code, all in docs/build.py:

  • the fr values of LABELS (they are the French UI);
  • the FR menu titles of GROUPS and of OTHER, same reason;
  • the French markers of the CALLOUTS table ("attention", "jamais", "astuce", "conseil", "remarque"…). These are not labels, they parse the French pages to pick a callout's colour. Translating them silently turns every French callout grey — the kind of breakage no test catches. The callout kinds (danger/tip/info) are English because they become CSS classes (.callout-tip).

• When a French page quotes script output, quote the English string the script now prints. A French page documenting an English-output tool is the expected result, not an oversight.

• ⚠️ .github/workflows/ci.yml greps the WORDING of a Vagrantfile error. The CONTROL_PLANES=2 guard-rail test matches 'is EVEN'. Reword that message and the test still passes while proving nothing — change both together.

• Commit messages in English, conventional (fix(...), feat(...), docs: ...). Branch from main, one feature per PR, squash merge.

• Every page of this repo follows the same skeleton (one emoji per ##, ⚠️/💡/ℹ️ callouts, a pitfalls section where it applies). Stick to plain CommonMark + GitHub tables so the generator renders it. The addon pages follow the same convention in k8s-playground, where they are written and published.

• No Markdown link may point into _k8s/. docs/build.py --strict resolves *.md links and anchors, the submodule's pages are not part of this documentation set, and make validate-docs fails on them. Use https://ops-nc.github.io/k8s-playground/ or a GitHub URL instead.

Adding a component = propagating it EVERYWHERE#

A variable, an option or an addon is only "done" once it appears at every level. One isolated mention is a documentation bug — the reader will never find it.