🏠 🐧Vagrant-Talos

Talos has no SSH, no shell and no package manager: the OS is read-only and everything goes through the talosctl API from the host. Vagrant therefore only creates and boots the VMs; all the cluster configuration is talosctl, which also means an upgrade is an image swap with automatic rollback rather than a package upgrade.

git clone --recurse-submodules https://github.com/OPS-NC/Vagrant-Talos.git
cd Vagrant-Talos
vagrant up                      # creates the VMs, they boot into maintenance mode
./talos/cluster-up.sh           # config + etcd bootstrap + kubeconfig + health
export TALOSCONFIG="$PWD/_out/talosconfig" KUBECONFIG="$PWD/kubeconfig"
./_k8s/platform-up.sh           # CNI, Envoy Gateway, metrics-server, wildcard TLS

⚠️ --recurse-submodules is not optional. _k8s/ is a git submodule; a plain git clone leaves it empty and ./_k8s/install.sh returns No such file or directory. On a clone already made: git submodule update --init --recursive.

ℹ️ There is a twin lab, Vagrant-KubeADM: same IP plan, same application layer, opposite operating model: there you get an ordinary Debian box with SSH and apt and you drive kubeadm yourself. The application layer works out which of the two it is mounted in on its own, so the same commands work in both.

🧰 1. Prerequisites (on the host)#

The Talos ISO (metal-amd64.iso) is downloaded automatically on the first vagrant up, into iso/. No Vagrant box or plugin to install: the dummy communicator (no SSH) and the empty pace/empty box are handled by the Vagrantfile.

💡 Keep talosctl aligned with TALOS_VERSION. The binary version decides the generated configuration schema, and a mismatch with the ISO produces obscure errors. To pin it instead of taking the latest:

curl -Lo /tmp/talosctl https://github.com/siderolabs/talos/releases/download/v1.13.7/talosctl-linux-amd64
sudo install -m 0755 /tmp/talosctl /usr/local/bin/talosctl

Managing the submodule:

git submodule update --init --recursive     # fills _k8s/ on an existing clone
git submodule update --remote _k8s          # move it to the latest upstream commit

⚠️ git pull does not update the submodule. It moves this repo only, leaving _k8s/ on the commit pinned before, so you would run the documented commands against an older application layer.

⚠️ VirtualBox and KVM cannot share VT-x. With the KVM module loaded, vagrant up dies on VERR_VMX_IN_VMX_ROOT_MODE. Unload it first: TROUBLESHOOTING.md.

🗺️ 2. IP plan (host-only network 192.168.56.0/24)#

The IPs are deterministic without anything being written inside the guest: each VM has a fixed MAC and a DHCP reservation on the VirtualBox host-only network, created by the Vagrantfile. Every VM has 2 NICs: NIC1 = VirtualBox NAT (Internet) and NIC2 = host-only (cluster and API).

ℹ️ Interface naming: since Talos 1.5, NICs get predictable names (enp0s3, enp0s8…), so the host-only NIC is enp0s8 (VirtualBox NIC2 = PCI bus 0000:00:08.0). The patches never target by name: the VIP is set through busPath and the node IP through the 192.168.56.0/24 subnet, which survives any naming scheme.

⚠️ The subnet is only half configurable. NETWORK drives the Vagrantfile and cluster-up.sh, but 192.168.56.x is hard-coded in talos/patch-all.yaml (validSubnets), talos/patch-cp.yaml (vip.ip, advertisedSubnets) and talos/cni-flannel.yaml (--iface-can-reach). Changing NETWORK without editing those three files gives a silently broken cluster.

⚙️ 3. Pick the topology — lab.env#

lab.env is the single source read by the Vagrantfile and by talos/cluster-up.sh. Copy the versioned template (lab.env is gitignored):

cp lab.env.example lab.env

Read by cluster-up.sh but absent from the template (all have a default): VIP ($NETWORK.5), CLUSTER_NAME (talos-lab), INSTALL_DISK (/dev/sda), OUT (_out), FORCE.

What the default topology costs: 3 × 4 GB + 3 × 2 GB = 18 GB of RAM, 12 vCPU and ~6 × 20 GB of disk. A 16 GB host cannot run it: drop to CONTROL_PLANES=1 / WORKERS=1 (~6 GB), enough for Talos itself and platform-up.sh, but not for the data addons (Longhorn replicates ×3, observability/ wants 4 GB control planes).

⚠️ Edit the file rather than exporting the variable. CONTROL_PLANES=1 vagrant up affects vagrant only: cluster-up.sh re-reads lab.env and would wait for control planes .20/.30 that were never created. To override on the fly, pass the variable to both commands.

💡 Create lab.env anyway. Without it, both readers fall back to their internal defaults, aligned on v1.13.7 and CNI=cilium, but you lose the Image Factory installer image (iscsi extensions), and with it Longhorn. Keep CNI and KUBE_PROXY_REPLACEMENT consistent with what you actually want: cluster-up.sh decides what Talos lays down at bootstrap, platform-up.sh decides what Helm installs afterwards, and two disagreeing values give you two competing CNIs or a cluster with no Service routing at all.

☸️ The Kubernetes version does not follow the Talos version. Left empty (the template default), the cluster runs the version the local talosctl binary ships (v1.36.2 for talosctl v1.13.7), which is always one Talos supports. Set KUBERNETES_VERSION=1.36.3 to pin it (a leading v is tolerated); cluster-up.sh turns it into gen config --kubernetes-version, which pins the control-plane images and the kubelet image. Two traps: nothing validates the value (gen config only templates image tags, so a version that does not exist produces a config that validates perfectly and then leaves the static pods in ErrImagePull), and it is only read when the config is generated: on a running cluster the tool is talosctl upgrade-k8s (talos/UPGRADE.md).

🌐 LAB_DOMAIN defaults to a neutral value (talos.lab.example.io) because the repo is public. The application-layer manifests carry that domain and the *-up.sh scripts substitute LAB_DOMAIN on the fly, never rewriting a versioned file; see k8s-playground — LAB_DOMAIN.

The 1st control plane is always talos-cp1 (192.168.56.10), and the VirtualBox VM name is identical to the Talos hostname (§7).

🚀 4. Start the cluster#

vagrant up                      # the VMs boot from the ISO, in maintenance mode
./talos/cluster-up.sh           # everything else

cluster-up.sh chains: config generation → apply to the nodes (with deterministic hostnames) → etcd bootstrap → kubeconfig → health wait. It prints the exports you need and a final kubectl get nodes.

export TALOSCONFIG="$PWD/_out/talosconfig"
export KUBECONFIG="$PWD/kubeconfig"
kubectl get nodes -o wide

Two things never to do:

⚠️ Never re-run cluster-up.sh on an already-installed cluster. Its maintenance-mode wait polls the nodes with --insecure, which a node in secure mode never answers. The wait is bounded (WAIT_MAINTENANCE, 300 s) and then fails with an explicit message, but it wasted five minutes and applied nothing. To grow a running cluster, see §6.1.

⚠️ Never regenerate _out/ (nor FORCE=1) on a running cluster: gen config produces new secrets and new CAs, which breaks the existing cluster. Do it only after a vagrant destroy.

vagrant up
talosctl -n 192.168.56.10 get disks --insecure   # must list /dev/sda

set -a ; . ./lab.env ; set +a        # loads TALOS_VERSION, INSTALLER_IMAGE, CNI…

talosctl gen config talos-lab https://192.168.56.5:6443 \
  --install-disk /dev/sda \
  --install-image "$INSTALLER_IMAGE" \
  --additional-sans 192.168.56.5,192.168.56.10,192.168.56.20,192.168.56.30 \
  ${KUBERNETES_VERSION:+--kubernetes-version "${KUBERNETES_VERSION#v}"} \
  --config-patch               @talos/patch-all.yaml \
  --config-patch-control-plane @talos/patch-cp.yaml \
  --config-patch-control-plane "@talos/cni-${CNI:-cilium}.yaml" \
  --config-patch-control-plane @talos/patch-no-kube-proxy.yaml \
  --output-dir _out

export TALOSCONFIG="$PWD/_out/talosconfig"

talosctl apply-config --insecure -n 192.168.56.10  --file _out/controlplane.yaml
talosctl apply-config --insecure -n 192.168.56.101 --file _out/worker.yaml
talosctl apply-config --insecure -n 192.168.56.102 --file _out/worker.yaml

talosctl config endpoint 192.168.56.10        # HA: add .20 .30
talosctl config node     192.168.56.10
talosctl bootstrap -n 192.168.56.10           # ONCE ONLY, on a SINGLE control plane

talosctl kubeconfig -n 192.168.56.10 ./kubeconfig
export KUBECONFIG="$PWD/kubeconfig"

talosctl health --wait-timeout 10m -n 192.168.56.10 -e 192.168.56.10
talosctl -n 192.168.56.10 get members      # members seen by discovery
kubectl get nodes -o wide

📦 5. What comes next: the application layer#

A bare cluster does nothing useful, and with the default CNI=cilium it is not even Ready yet. Cilium, Envoy Gateway, cert-manager, metrics-server, Longhorn, Vault, CloudNativePG, Prometheus/Loki, Kyverno, Trivy, MinIO, Argo CD… all come from k8s-playground, mounted here as _k8s/ and shared with the kubeadm twin. Its documentation is published separately: https://ops-nc.github.io/k8s-playground/.

export TALOSCONFIG="$PWD/_out/talosconfig"   # where the Talos API is
export KUBECONFIG="$PWD/kubeconfig"          # where the cluster is

./_k8s/platform-up.sh                        # Cilium → Envoy Gateway → metrics-server → TLS
./_k8s/install.sh longhorn vault argocd      # opt-in addons
./_k8s/install.sh list                       # the full catalogue
./_k8s/install.sh all                        # platform + every addon, in dependency order
./_k8s/longhorn/longhorn-up.sh               # one addon on its own

Nothing to declare: the lab is the parent directory of _k8s/ that carries a Vagrantfile (so lab.env and _out/ are found there), and the distribution is read as Talos from the presence of talos/cluster-up.sh. That works right after the clone, before the first vagrant up. An explicit form still wins if you need it (./_k8s/install.sh talos platform, --distro=talos, K8S_DISTRO), and LAB_DIR remains the escape hatch for an unusual layout.

⚠️ TALOSCONFIG and KUBECONFIG are a different matter: they really are required. The addons that drive the Talos API (longhorn, local-path) need TALOSCONFIG, and everything touching the cluster needs KUBECONFIG. Nothing detects those for you.

⚠️ This layer expects CNI=cilium (the default): it relies on a LoadBalancer Service that actually gets an IP, which here only Cilium's L2/ARP announcement provides. With flannel, calico or none the Gateway stays at EXTERNAL-IP <pending> and no UI is reachable (§8).

After the bootstrap the nodes stay NotReady until the CNI is installed; platform-up.sh handles that in its first step.

5.1 DNS + TLS: the two manual prerequisites#

ℹ️ This whole subsection is for SELF_SIGNED=false only. With the default, platform-up.sh signs the wildcard itself with openssl under a local CA: no public DNS record and no Cloudflare token needed, and the domain never has to exist outside your machine. All you do is make the name resolve locally, with an /etc/hosts line pointing your subdomains at 192.168.56.200, and optionally import _out/self-signed/ca.crt to silence the browser warning. Read on only if you own a real domain and want a publicly trusted certificate.

a) A wildcard DNS record pointing at the Gateway IP. Every lab UI is served through Envoy's LoadBalancer Service, which takes the first IP of LB_POOL_START (192.168.56.200 by default), so one record covers every subdomain:

kubectl -n envoy-gateway-system get svc -o wide | grep LoadBalancer   # the IP actually assigned
dig +short argo.talos.lab.example.io                                  # must answer .200

⚠️ The Cloudflare proxy (orange cloud) cannot work here. It would have to reach your origin from the Internet, but 192.168.56.200 is a private, non-routable IP: in orange you get a 522. Stay on DNS-only: Envoy terminates TLS, not Cloudflare, hence the need for a publicly trusted certificate (point b).

The lab is therefore only reachable from the host, or through access to the host-only network (remote access). A public wildcard pointing at a private IP carries no exploitation risk, but it does publish the existence of the lab and its IP plan. With no DNS at all, short-circuit resolution: curl -sI --resolve argo.talos.lab.example.io:443:192.168.56.200 https://argo.talos.lab.example.io/.

b) A Cloudflare API token for the DNS-01 challenge. A wildcard cannot be validated over HTTP-01 (Let's Encrypt cannot reach a private IP), so cert-manager proves ownership by writing an _acme-challenge record. That takes a token scoped to Zone/DNS/Edit + Zone/Zone/Read on your zone only: an All zones token would let the lab rewrite the DNS of every domain you own. How to create it: k8s-playground — cert-manager/.

Then in lab.env (gitignored, and never in lab.env.example):

SELF_SIGNED=false                      # leave the default (true) and none of this is read
LAB_DOMAIN=talos.lab.example.io
LAB_DNS_ZONE=example.io                # the Cloudflare zone (derived if empty)
LAB_ACME_EMAIL=you@example.io
LAB_ACME_ISSUER=staging                # staging (default) | prod — see below
CLOUDFLARE_API_TOKEN=<your-token>

platform-up.sh creates the cloudflare-api-token Secret, substitutes the domain and waits for the certificate (kubectl -n envoy-gateway-system get certificate).

⚠️ prod costs a quota slot on every rebuild, which is why staging is the default. The wildcard lives only in etcd, so vagrant destroy burns it and the next platform-up.sh asks for a brand new one. Let's Encrypt production allows 5 certificates per week for the same *.<LAB_DOMAIN>: the 6th rebuild fails with 429 rateLimited and the lab stays without TLS until the 168 h window slides. Use prod on a stable lab, not while iterating, and back the wildcard up before a destroy:

kubectl -n envoy-gateway-system get secret wildcard-<your-domain-in-dashes>-tls \
  -o yaml > _out/wildcard-tls.backup.yaml     # contains the private key: _out/ is gitignored

♻️ 6. Lifecycle#

vagrant status                 # VM status
vagrant halt                   # power off
vagrant up                     # power back on
vagrant destroy -f             # delete everything (dedicated disks included)
rm -rf _out kubeconfig         # clear the local Talos state before starting over

Keeping the repo current takes two commands, since git pull leaves _k8s/ where it was:

git pull
git submodule update --init --recursive   # _k8s/ back onto the commit this repo pins
git submodule update --remote _k8s        # or: jump to the latest k8s-playground

⚠️ VirtualBox 7.x does not always clean up after a destroy, and the next up then fails on VERR_ALREADY_EXISTS. Purge the leftovers with ./talos/virtualbox-cleanup.sh; precautions in TROUBLESHOOTING.md.

6.1 Adding workers (live, without breaking the cluster)#

To grow an already running cluster, start the new VMs and apply the existing worker config to them (same secrets). Two rules: do not regenerate _out/ (new secrets would break the cluster) and do not re-run cluster-up.sh (it would wait for maintenance mode on already-installed nodes).

Going from 3 to 5 workers (talos-w4=.104, talos-w5=.105):

# 1. raise WORKERS in lab.env (here WORKERS=5), then start ONLY the new VMs
vagrant up talos-w4 talos-w5

# 2. apply the existing worker config, pinning the hostname (Nth worker = talos-w<N>)
export TALOSCONFIG="$PWD/_out/talosconfig"
WK_IP_START=101 ; WK_IP_STEP=1              # same values as lab.env
for n in 4 5; do
  ip="192.168.56.$(( WK_IP_START + (n - 1) * WK_IP_STEP ))"
  until talosctl -n "$ip" get disks --insecure >/dev/null 2>&1; do sleep 5; done
  talosctl apply-config --insecure -n "$ip" --file _out/worker.yaml \
    --config-patch "$(printf 'apiVersion: v1alpha1\nkind: HostnameConfig\nauto: "off"\nhostname: talos-w%s\n' "$n")"
done

The workers then join on their own: their config already points at the VIP. Adding control planes follows the same logic (controlplane.yaml, hostname talos-cp<N>); they join etcd through discovery, without re-running bootstrap.

💡 Removing a worker: kubectl drain talos-w5 --ignore-daemonsets --delete-emptydir-data, then vagrant destroy -f talos-w5, kubectl delete node talos-w5, and lower WORKERS.

🔍 7. Design notes#

• No SSH → a dummy communicator (in the Vagrantfile) reports "ready" immediately so vagrant up does not hang. It also means vagrant up returning is not a sign the nodes are ready: all the real waiting happens in cluster-up.sh.
• No Talos box → we start from the empty pace/empty box and boot the metal-amd64.iso ISO (SATA DVD drive, BIOS, boot disk then DVD).
• Deterministic IPs → fixed MAC per VM + host-only DHCP reservations (VBoxManage dhcpserver … --fixed-address) created by a before :up trigger, stale leases purged, so the node takes its reserved IP on the 1st DHCP.
• Deterministic hostnames → one HostnameConfig document per node (auto: "off") instead of the auto-generated talos-xxxxx. The VirtualBox VMs carry the same name.
• VIP / HA → talos/patch-cp.yaml sets a VIP shared between control planes (election through etcd), so the kube-apiserver endpoint stays stable when a CP goes down. It is also what Cilium is pointed at (k8sServiceHost) once kube-proxy is gone.
• No kube-proxy → talos/patch-no-kube-proxy.yaml sets cluster.proxy.disabled: true, so the bootstrap renders no kube-proxy manifest and Cilium serves the Services in eBPF (§8).
• Online discovery → talos/patch-all.yaml enables discovery.talos.dev and disables the kubernetes registry, deprecated and incompatible with Kubernetes ≥ 1.32.
• Default route through the NAT → deliberate (Internet access). What must be host-only is the node's identity (kubelet nodeIP, etcd, VIP), not the default route.

References: Talos Linux · siderolabs/talos · rgl/talos-vagrant · bjwschaap/vagrant-empty-box

🌐 8. CNI: Cilium, Calico or Flannel#

CNI (in lab.env) expresses an intent, read in two places: talos/cluster-up.sh applies the talos/cni-<CNI>.yaml patch, which fills cluster.network.cni in the control plane config, and ./_k8s/platform-up.sh installs the CNI through Helm in every case but flannel, which Talos lays down itself at bootstrap, without kubectl, from an internal manifest.

In practice: keep cilium. It is the only choice that makes the lab usable end to end, because it is the only one that gives Services an EXTERNAL-IP on a host-only network, and therefore the only one that gets you the HTTPS UIs. It is also the only one with NetworkPolicy and kube-proxy replacement and Hubble. calico is there to compare CNIs and work on NetworkPolicy; flannel for a bare cluster, if you just want to explore Talos. The Cilium install itself (pinned chart, L2 pool, --set devices=enp0s8) is documented and scripted in k8s-playground cilium/.

kube-proxy: replaced by Cilium in eBPF (the default)#

KUBE_PROXY_REPLACEMENT (default true) is read in the same two places as CNI, and this lab behaves exactly like the kubeadm sibling, where the equivalent is kubeadm init --skip-phases=addon/kube-proxy:

kubectl -n kube-system get ds kube-proxy      # NotFound with the default: expected
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status --verbose \
  | grep KubeProxyReplacement                 # must say True

platform-up.sh then installs Cilium with kubeProxyReplacement=true plus k8sServiceHost=<VIP> k8sServicePort=6443, mandatory since with no kube-proxy nothing provisions the apiserver ClusterIP and the agent could not bootstrap through kubernetes.default.

⚠️ KUBE_PROXY_REPLACEMENT=true requires CNI=cilium, and cluster-up.sh refuses any other combination, as does make validate-talos. Nothing else here replaces kube-proxy: without it and without a replacement, no ClusterIP answers at all, CoreDNS included.

⚠️ Both are decided at bootstrap and are not live toggles. Like CNI, the value is read only when the config is generated: changing it against an existing _out/ does nothing, and changing it on a running cluster is unsupported. vagrant destroy -f, then rebuild: two coexisting CNIs fight over the pod network.

ℹ️ Why the VIP and not KubePrism. Cilium's own Talos page suggests k8sServiceHost=localhost k8sServicePort=7445 (KubePrism, enabled by default in the generated config). The lab keeps the VIP 192.168.56.5:6443: it is the endpoint everything else already uses, it is in the apiserver certificate's SANs, and it lets both labs share a single code path. KubePrism stays available if you want to switch.

⚠️ Calico cannot announce LoadBalancer IPs. It can only do it over BGP, which assumes a peer router, non-existent on a VirtualBox host-only network. With CNI=calico you must install MetalLB (L2 mode); platform-up.sh also strips the Cilium-specific loadBalancerClass: io.cilium/l2-announcer from Envoy-Proxy.yml so another announcer can take over. Full procedure: k8s-playground calico/.

⚠️ Whatever the CNI, pin the host-only interface (enp0s8). Otherwise it picks the default-route NIC (the NAT, 10.0.2.15, identical on every VM), and the VXLAN tunnels are broken while Internet egress still works, which makes for a very confusing DNS failure.

🛠️ 9. Validating a change#

Everything validates without touching a cluster:

make validate       # script syntax + YAML + Vagrantfile + Talos config + doc links
make docs           # regenerates docs/index.html from every README (EN + FR)
make help           # lists the targets

make validate-talos generates the config in a temporary directory, then feeds it to talosctl validate --mode metal: no risk for _out/ nor for the cluster, unlike FORCE=1 ./talos/cluster-up.sh, which regenerates the secrets and breaks a running cluster. It also prints the versions it used, which is the cheap way to confirm your lab.env keys are really being read. make validate-docs builds the docs into a throwaway directory and fails if a *.md link or a cross-page anchor no longer resolves, and make validate-submodule checks the _k8s pointer itself: an https:// URL (an SSH one breaks the clone for everyone without a GitHub key) and a pinned commit that is really pushed.

On every pull request the ci workflow re-runs the shell, YAML and Vagrantfile checks by calling the very same make targets, so a check cannot pass in CI and fail on your machine. vagrant validate runs there with --ignore-provider, since a runner has no VirtualBox.

ℹ️ validate-shell and validate-yaml only cover files tracked by this repo. The _k8s/ submodule is a single pointer, so none of its scripts or manifests are checked here; they are validated in k8s-playground's own CI.

📄 10. License#

Apache License 2.0. See LICENSE. Use it, modify it, redistribute it, including commercially, as long as you keep the copyright notice and state your changes. No warranty: this is a lab, do not run it in production.

It covers what this repo contains: the Vagrantfile, the talos/ scripts, the config patches and the documentation. It does not extend to the third-party components those scripts download (Talos Linux, Cilium, Longhorn, Vault, Envoy Gateway…), nor to the _k8s/ submodule: k8s-playground carries its own LICENSE.

🏠 🐧Vagrant-Talos

Talos n'a ni SSH, ni shell, ni gestionnaire de paquets : l'OS est en lecture seule et tout passe par l'API talosctl depuis l'hôte. Vagrant ne fait donc que créer et démarrer les VM ; toute la configuration du cluster est du talosctl, ce qui veut aussi dire qu'une montée de version est un échange d'image avec rollback automatique, pas une mise à jour de paquets.

git clone --recurse-submodules https://github.com/OPS-NC/Vagrant-Talos.git
cd Vagrant-Talos
vagrant up                      # crée les VM, elles démarrent en mode maintenance
./talos/cluster-up.sh           # config + bootstrap etcd + kubeconfig + health
export TALOSCONFIG="$PWD/_out/talosconfig" KUBECONFIG="$PWD/kubeconfig"
./_k8s/platform-up.sh           # CNI, Envoy Gateway, metrics-server, TLS wildcard

⚠️ --recurse-submodules n'est pas optionnel. _k8s/ est un sous-module git ; un git clone simple le laisse vide et ./_k8s/install.sh répond No such file or directory. Sur un clone déjà fait : git submodule update --init --recursive.

ℹ️ Il existe un lab jumeau, Vagrant-KubeADM : même plan d'adressage, même couche applicative, modèle d'exploitation opposé : là-bas tu as une Debian ordinaire avec SSH et apt, et tu conduis kubeadm toi-même. La couche applicative reconnaît toute seule dans lequel des deux elle est montée, donc les mêmes commandes marchent des deux côtés.

🧰 1. Prérequis (sur l'hôte)#

L'ISO Talos (metal-amd64.iso) est téléchargée automatiquement au premier vagrant up, dans iso/. Aucune box ni plugin Vagrant à installer : le communicateur factice (pas de SSH) et la box vide pace/empty sont gérés par le Vagrantfile.

💡 Garde talosctl aligné sur TALOS_VERSION. La version du binaire décide du schéma de configuration généré, et un écart avec l'ISO produit des erreurs obscures. Pour l'épingler plutôt que de prendre la dernière :

curl -Lo /tmp/talosctl https://github.com/siderolabs/talos/releases/download/v1.13.7/talosctl-linux-amd64
sudo install -m 0755 /tmp/talosctl /usr/local/bin/talosctl

Gérer le sous-module :

git submodule update --init --recursive     # remplit _k8s/ sur un clone existant
git submodule update --remote _k8s          # le déplace sur le dernier commit amont

⚠️ git pull ne met pas le sous-module à jour. Il ne déplace que ce dépôt, _k8s/ reste sur le commit épinglé avant, et tu exécuterais les commandes documentées contre une couche applicative plus ancienne.

⚠️ VirtualBox et KVM ne peuvent pas partager VT-x. Module KVM chargé, vagrant up meurt sur VERR_VMX_IN_VMX_ROOT_MODE. Décharge-le d'abord : DEPANNAGE.md.

🗺️ 2. Plan d'adressage (réseau host-only 192.168.56.0/24)#

Les IP sont déterministes sans que rien ne soit écrit dans le guest : chaque VM a une MAC fixe et une réservation DHCP sur le réseau host-only VirtualBox, créée par le Vagrantfile. Chaque VM a 2 cartes : NIC1 = NAT VirtualBox (Internet) et NIC2 = host-only (cluster et API).

ℹ️ Nommage des interfaces : depuis Talos 1.5, les cartes reçoivent des noms prévisibles (enp0s3, enp0s8…), donc la carte host-only est enp0s8 (NIC2 VirtualBox = bus PCI 0000:00:08.0). Les patches ne ciblent jamais par nom : la VIP passe par busPath et l'IP du node par le sous-réseau 192.168.56.0/24, ce qui survit à n'importe quel schéma de nommage.

⚠️ Le sous-réseau n'est configurable qu'à moitié. NETWORK pilote le Vagrantfile et cluster-up.sh, mais 192.168.56.x est codé en dur dans talos/patch-all.yaml (validSubnets), talos/patch-cp.yaml (vip.ip, advertisedSubnets) et talos/cni-flannel.yaml (--iface-can-reach). Changer NETWORK sans éditer ces trois fichiers donne un cluster cassé en silence.

⚙️ 3. Choisir la topologie — lab.env#

lab.env est la source unique lue par le Vagrantfile et par talos/cluster-up.sh. Copie le modèle versionné (lab.env est gitignoré) :

cp lab.env.example lab.env

Lues par cluster-up.sh mais absentes du modèle (toutes ont un défaut) : VIP ($NETWORK.5), CLUSTER_NAME (talos-lab), INSTALL_DISK (/dev/sda), OUT (_out), FORCE.

Ce que coûte la topologie par défaut : 3 × 4 Go + 3 × 2 Go = 18 Go de RAM, 12 vCPU et ~6 × 20 Go de disque. Un hôte de 16 Go ne peut pas la faire tourner : descends à CONTROL_PLANES=1 / WORKERS=1 (~6 Go), suffisant pour Talos lui-même et platform-up.sh, mais pas pour les addons de données (Longhorn réplique ×3, observability/ veut des control planes de 4 Go).

⚠️ Édite le fichier plutôt que d'exporter la variable. CONTROL_PLANES=1 vagrant up n'affecte que vagrant : cluster-up.sh relit lab.env et attendrait des control planes .20/.30 jamais créés. Pour surcharger à la volée, passe la variable aux deux commandes.

💡 Crée lab.env quand même. Sans lui, les deux lecteurs retombent sur leurs défauts internes, alignés sur v1.13.7 et CNI=cilium, mais tu perds l'image d'installation Image Factory (extensions iscsi), et avec elle Longhorn. Garde CNI et KUBE_PROXY_REPLACEMENT cohérents avec ce que tu veux vraiment : cluster-up.sh décide ce que Talos pose au bootstrap, platform-up.sh décide ce que Helm installe ensuite, et deux valeurs qui divergent donnent deux CNI concurrents ou un cluster sans aucun routage de Services.

☸️ La version de Kubernetes ne suit pas celle de Talos. Laissée vide (le défaut du modèle), le cluster tourne sur la version que livre le binaire talosctl local (v1.36.2 pour talosctl v1.13.7), qui est toujours une version supportée par Talos. Mets KUBERNETES_VERSION=1.36.3 pour l'épingler (un v en tête est toléré) ; cluster-up.sh en fait un gen config --kubernetes-version, qui épingle les images du control plane et celle du kubelet. Deux pièges : rien ne valide la valeur (gen config ne fait que templater des tags d'image, donc une version qui n'existe pas produit une config qui valide parfaitement puis laisse les pods statiques en ErrImagePull), et elle n'est lue qu'à la génération de la config : sur un cluster vivant, l'outil est talosctl upgrade-k8s (talos/MISE-A-JOUR.md).

🌐 LAB_DOMAIN a un défaut neutre (talos.lab.example.io) parce que le dépôt est public. Les manifestes de la couche applicative portent ce domaine et les scripts *-up.sh y substituent LAB_DOMAIN à la volée, sans jamais réécrire un fichier versionné ; voir k8s-playground — LAB_DOMAIN.

Le 1er control plane est toujours talos-cp1 (192.168.56.10), et le nom de la VM VirtualBox est identique au hostname Talos (§7).

🚀 4. Démarrer le cluster#

vagrant up                      # les VM démarrent sur l'ISO, en mode maintenance
./talos/cluster-up.sh           # tout le reste

cluster-up.sh enchaîne : génération de la config → application aux nodes (avec des hostnames déterministes) → bootstrap etcd → kubeconfig → attente de santé. Il affiche les export dont tu as besoin et un kubectl get nodes final.

export TALOSCONFIG="$PWD/_out/talosconfig"
export KUBECONFIG="$PWD/kubeconfig"
kubectl get nodes -o wide

Deux choses à ne jamais faire :

⚠️ Ne relance jamais cluster-up.sh sur un cluster déjà installé. Son attente du mode maintenance interroge les nodes en --insecure, ce qu'un node en mode sécurisé ne répond jamais. L'attente est bornée (WAIT_MAINTENANCE, 300 s) puis échoue sur un message explicite, mais elle a perdu cinq minutes et n'a rien appliqué. Pour agrandir un cluster vivant, voir le §6.1.

⚠️ Ne régénère jamais _out/ (ni FORCE=1) sur un cluster vivant : gen config produit de nouveaux secrets et de nouvelles AC, ce qui casse le cluster existant. À faire seulement après un vagrant destroy.

vagrant up
talosctl -n 192.168.56.10 get disks --insecure   # doit lister /dev/sda

set -a ; . ./lab.env ; set +a        # charge TALOS_VERSION, INSTALLER_IMAGE, CNI…

talosctl gen config talos-lab https://192.168.56.5:6443 \
  --install-disk /dev/sda \
  --install-image "$INSTALLER_IMAGE" \
  --additional-sans 192.168.56.5,192.168.56.10,192.168.56.20,192.168.56.30 \
  ${KUBERNETES_VERSION:+--kubernetes-version "${KUBERNETES_VERSION#v}"} \
  --config-patch               @talos/patch-all.yaml \
  --config-patch-control-plane @talos/patch-cp.yaml \
  --config-patch-control-plane "@talos/cni-${CNI:-cilium}.yaml" \
  --config-patch-control-plane @talos/patch-no-kube-proxy.yaml \
  --output-dir _out

export TALOSCONFIG="$PWD/_out/talosconfig"

talosctl apply-config --insecure -n 192.168.56.10  --file _out/controlplane.yaml
talosctl apply-config --insecure -n 192.168.56.101 --file _out/worker.yaml
talosctl apply-config --insecure -n 192.168.56.102 --file _out/worker.yaml

talosctl config endpoint 192.168.56.10        # HA : ajoute .20 .30
talosctl config node     192.168.56.10
talosctl bootstrap -n 192.168.56.10           # UNE SEULE FOIS, sur UN SEUL control plane

talosctl kubeconfig -n 192.168.56.10 ./kubeconfig
export KUBECONFIG="$PWD/kubeconfig"

talosctl health --wait-timeout 10m -n 192.168.56.10 -e 192.168.56.10
talosctl -n 192.168.56.10 get members      # membres vus par la discovery
kubectl get nodes -o wide

📦 5. La suite : la couche applicative#

Un cluster nu ne sert à rien, et avec le CNI=cilium par défaut il n'est même pas encore Ready. Cilium, Envoy Gateway, cert-manager, metrics-server, Longhorn, Vault, CloudNativePG, Prometheus/Loki, Kyverno, Trivy, MinIO, Argo CD… viennent tous de k8s-playground, monté ici en _k8s/ et partagé avec le jumeau kubeadm. Sa documentation est publiée à part : https://ops-nc.github.io/k8s-playground/.

export TALOSCONFIG="$PWD/_out/talosconfig"   # où est l'API Talos
export KUBECONFIG="$PWD/kubeconfig"          # où est le cluster

./_k8s/platform-up.sh                        # Cilium → Envoy Gateway → metrics-server → TLS
./_k8s/install.sh longhorn vault argocd      # addons opt-in
./_k8s/install.sh list                       # le catalogue complet
./_k8s/install.sh all                        # plateforme + tous les addons, dans l'ordre
./_k8s/longhorn/longhorn-up.sh               # un addon seul

Rien à déclarer : le lab est le dossier parent de _k8s/ qui porte un Vagrantfile (donc lab.env et _out/ s'y trouvent), et la distribution est lue comme Talos à la présence de talos/cluster-up.sh. Ça marche dès le clone, avant le premier vagrant up. Une forme explicite gagne toujours si besoin (./_k8s/install.sh talos platform, --distro=talos, K8S_DISTRO), et LAB_DIR reste la porte de sortie pour une arborescence inhabituelle.

⚠️ TALOSCONFIG et KUBECONFIG sont un autre sujet : eux sont bel et bien nécessaires. Les addons qui pilotent l'API Talos (longhorn, local-path) ont besoin de TALOSCONFIG, et tout ce qui touche le cluster a besoin de KUBECONFIG. Rien ne les détecte pour toi.

⚠️ Cette couche attend CNI=cilium (le défaut) : elle repose sur un Service LoadBalancer qui obtient réellement une IP, ce que seule l'annonce L2/ARP de Cilium fournit ici. Avec flannel, calico ou none, le Gateway reste en EXTERNAL-IP <pending> et aucune UI n'est joignable (§8).

Après le bootstrap, les nodes restent NotReady jusqu'à l'installation du CNI ; platform-up.sh s'en occupe à sa première étape.

5.1 DNS + TLS : les deux prérequis manuels#

ℹ️ Toute cette sous-section ne concerne que SELF_SIGNED=false. Avec le défaut, platform-up.sh signe lui-même le wildcard avec openssl sous une AC locale : aucun enregistrement DNS public ni token Cloudflare nécessaire, et le domaine n'a jamais à exister en dehors de ta machine. Il suffit de faire résoudre le nom localement, avec une ligne /etc/hosts pointant tes sous-domaines vers 192.168.56.200, et éventuellement d'importer _out/self-signed/ca.crt pour faire taire l'avertissement du navigateur. Ne continue que si tu possèdes un vrai domaine et veux un certificat reconnu publiquement.

a) Un enregistrement DNS wildcard vers l'IP du Gateway. Toutes les UI du lab passent par le Service LoadBalancer d'Envoy, qui prend la première IP de LB_POOL_START (192.168.56.200 par défaut) : un seul enregistrement couvre donc tous les sous-domaines.

kubectl -n envoy-gateway-system get svc -o wide | grep LoadBalancer   # l'IP réellement attribuée
dig +short argo.talos.lab.example.io                                  # doit répondre .200

⚠️ Le proxy Cloudflare (nuage orange) ne peut pas fonctionner ici. Il devrait joindre ton origine depuis Internet, or 192.168.56.200 est une IP privée, non routable : en orange tu obtiens un 522. Reste en DNS-only : c'est Envoy qui termine TLS, pas Cloudflare, d'où le besoin d'un certificat reconnu publiquement (point b).

Le lab n'est donc joignable que depuis l'hôte, ou via un accès au réseau host-only (accès distant). Un wildcard public pointant une IP privée ne présente aucun risque d'exploitation, mais il publie l'existence du lab et son plan d'adressage. Sans DNS du tout, court-circuite la résolution : curl -sI --resolve argo.talos.lab.example.io:443:192.168.56.200 https://argo.talos.lab.example.io/.

b) Un token d'API Cloudflare pour le challenge DNS-01. Un wildcard ne peut pas être validé en HTTP-01 (Let's Encrypt ne peut pas joindre une IP privée), donc cert-manager prouve la propriété en écrivant un enregistrement _acme-challenge. Ça demande un token limité à Zone/DNS/Edit + Zone/Zone/Read sur ta zone seulement : un token All zones laisserait le lab réécrire le DNS de tous tes domaines. Comment le créer : k8s-playground — cert-manager/.

Puis dans lab.env (gitignoré, et jamais dans lab.env.example) :

SELF_SIGNED=false                      # laisse le défaut (true) et rien de tout ça n'est lu
LAB_DOMAIN=talos.lab.example.io
LAB_DNS_ZONE=example.io                # la zone Cloudflare (déduite si vide)
LAB_ACME_EMAIL=toi@example.io
LAB_ACME_ISSUER=staging                # staging (défaut) | prod — voir ci-dessous
CLOUDFLARE_API_TOKEN=<ton-token>

platform-up.sh crée le Secret cloudflare-api-token, substitue le domaine et attend le certificat (kubectl -n envoy-gateway-system get certificate).

⚠️ prod coûte un quota à chaque reconstruction, et c'est pourquoi staging est le défaut. Le wildcard ne vit que dans etcd, donc vagrant destroy le brûle et le platform-up.sh suivant en redemande un neuf. Let's Encrypt production autorise 5 certificats par semaine pour le même *.<LAB_DOMAIN> : la 6e reconstruction échoue sur 429 rateLimited et le lab reste sans TLS jusqu'à ce que la fenêtre de 168 h glisse. Utilise prod sur un lab stable, pas pendant que tu itères, et sauvegarde le wildcard avant un destroy :

kubectl -n envoy-gateway-system get secret wildcard-<ton-domaine-en-tirets>-tls \
  -o yaml > _out/wildcard-tls.backup.yaml     # contient la clé privée : _out/ est gitignoré

♻️ 6. Cycle de vie#

vagrant status                 # état des VM
vagrant halt                   # extinction
vagrant up                     # rallumage
vagrant destroy -f             # supprime tout (disques dédiés compris)
rm -rf _out kubeconfig         # nettoyer l'état Talos local avant de repartir

Garder le dépôt à jour prend deux commandes, git pull laissant _k8s/ où il était :

git pull
git submodule update --init --recursive   # _k8s/ revient sur le commit épinglé ici
git submodule update --remote _k8s        # ou : sauter au dernier k8s-playground

⚠️ VirtualBox 7.x ne nettoie pas toujours après un destroy, et le up suivant échoue alors sur VERR_ALREADY_EXISTS. Purge les résidus avec ./talos/virtualbox-cleanup.sh ; précautions dans DEPANNAGE.md.

6.1 Ajouter des workers à chaud (sans casser le cluster)#

Pour agrandir un cluster déjà en marche, démarre les nouvelles VM et applique-leur la config worker existante (mêmes secrets). Deux règles : ne régénère pas _out/ (de nouveaux secrets casseraient le cluster) et ne relance pas cluster-up.sh (il attendrait le mode maintenance sur des nodes déjà installés).

Passer de 3 à 5 workers (talos-w4=.104, talos-w5=.105) :

# 1. augmente WORKERS dans lab.env (ici WORKERS=5), puis démarre SEULEMENT les nouvelles VM
vagrant up talos-w4 talos-w5

# 2. applique la config worker existante en épinglant le hostname (Nième worker = talos-w<N>)
export TALOSCONFIG="$PWD/_out/talosconfig"
WK_IP_START=101 ; WK_IP_STEP=1              # mêmes valeurs que lab.env
for n in 4 5; do
  ip="192.168.56.$(( WK_IP_START + (n - 1) * WK_IP_STEP ))"
  until talosctl -n "$ip" get disks --insecure >/dev/null 2>&1; do sleep 5; done
  talosctl apply-config --insecure -n "$ip" --file _out/worker.yaml \
    --config-patch "$(printf 'apiVersion: v1alpha1\nkind: HostnameConfig\nauto: "off"\nhostname: talos-w%s\n' "$n")"
done

Les workers rejoignent ensuite tout seuls : leur config pointe déjà la VIP. Ajouter des control planes suit la même logique (controlplane.yaml, hostname talos-cp<N>) ; ils rejoignent etcd par la discovery, sans rejouer bootstrap.

💡 Retirer un worker : kubectl drain talos-w5 --ignore-daemonsets --delete-emptydir-data, puis vagrant destroy -f talos-w5, kubectl delete node talos-w5, et baisse WORKERS.

🔍 7. Notes de conception#

• Pas de SSH → un communicateur factice (dans le Vagrantfile) répond « prêt » immédiatement pour que vagrant up ne bloque pas. Corollaire : vagrant up qui rend la main n'est pas un signe que les nodes sont prêts : toute la vraie attente est dans cluster-up.sh.
• Pas de box Talos → on part de la box vide pace/empty et on démarre sur l'ISO metal-amd64.iso (lecteur DVD SATA, BIOS, disque de boot puis DVD).
• IP déterministes → MAC fixe par VM + réservations DHCP host-only (VBoxManage dhcpserver … --fixed-address) créées par un trigger before :up, baux périmés purgés, pour que le node prenne son IP réservée dès le 1er DHCP.
• Hostnames déterministes → un document HostnameConfig par node (auto: "off") au lieu du talos-xxxxx auto-généré. Les VM VirtualBox portent le même nom.
• VIP / HA → talos/patch-cp.yaml pose une VIP partagée entre control planes (élection par etcd), pour que l'endpoint du kube-apiserver reste stable quand un CP tombe. C'est aussi ce sur quoi Cilium est pointé (k8sServiceHost) une fois kube-proxy parti.
• Pas de kube-proxy → talos/patch-no-kube-proxy.yaml pose cluster.proxy.disabled: true, donc le bootstrap ne rend aucun manifeste kube-proxy et Cilium sert les Services en eBPF (§8).
• Discovery en ligne → talos/patch-all.yaml active discovery.talos.dev et désactive le registre kubernetes, déprécié et incompatible avec Kubernetes ≥ 1.32.
• Route par défaut par le NAT → volontaire (accès Internet). Ce qui doit être host-only, c'est l'identité du node (nodeIP du kubelet, etcd, VIP), pas la route par défaut.

Références : Talos Linux · siderolabs/talos · rgl/talos-vagrant · bjwschaap/vagrant-empty-box

🌐 8. CNI : Cilium, Calico ou Flannel#

CNI (dans lab.env) exprime une intention, lue à deux endroits : talos/cluster-up.sh applique le patch talos/cni-<CNI>.yaml, qui remplit cluster.network.cni dans la config du control plane, et ./_k8s/platform-up.sh installe le CNI via Helm dans tous les cas sauf flannel, que Talos pose lui-même au bootstrap, sans kubectl, depuis un manifeste interne.

En pratique : garde cilium. C'est le seul choix qui rend le lab utilisable de bout en bout, parce que c'est le seul qui donne une EXTERNAL-IP aux Services sur un réseau host-only, donc le seul qui te donne les UI HTTPS. C'est aussi le seul avec NetworkPolicy et remplacement de kube-proxy et Hubble. calico est là pour comparer les CNI et travailler sur NetworkPolicy ; flannel pour un cluster nu, si tu veux juste explorer Talos. L'installation de Cilium elle-même (chart épinglé, pool L2, --set devices=enp0s8) est documentée et scriptée dans k8s-playground cilium/.

kube-proxy : remplacé par Cilium en eBPF (le défaut)#

KUBE_PROXY_REPLACEMENT (défaut true) est lu aux deux mêmes endroits que CNI, et ce lab se comporte exactement comme le jumeau kubeadm, où l'équivalent est kubeadm init --skip-phases=addon/kube-proxy :

kubectl -n kube-system get ds kube-proxy      # NotFound avec le défaut : attendu
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status --verbose \
  | grep KubeProxyReplacement                 # doit dire True

platform-up.sh installe alors Cilium avec kubeProxyReplacement=true plus k8sServiceHost=<VIP> k8sServicePort=6443, obligatoire puisque sans kube-proxy plus rien ne provisionne la ClusterIP de l'apiserver et que l'agent ne pourrait pas s'amorcer par kubernetes.default.

⚠️ KUBE_PROXY_REPLACEMENT=true exige CNI=cilium, et cluster-up.sh refuse toute autre combinaison, comme make validate-talos. Rien d'autre ici ne remplace kube-proxy : sans lui et sans remplacement, aucune ClusterIP ne répond, CoreDNS compris.

⚠️ Les deux se décident au bootstrap et ne sont pas des interrupteurs à chaud. Comme CNI, la valeur n'est lue qu'à la génération de la config : la changer contre un _out/ existant ne fait rien, et la changer sur un cluster vivant n'est pas supporté. vagrant destroy -f, puis reconstruis : deux CNI coexistants se disputent le réseau de pods.

ℹ️ Pourquoi la VIP et pas KubePrism. La page Talos de Cilium suggère k8sServiceHost=localhost k8sServicePort=7445 (KubePrism, activé par défaut dans la config générée). Le lab garde la VIP 192.168.56.5:6443 : c'est l'endpoint que tout le reste utilise déjà, il est dans les SAN du certificat de l'apiserver, et ça laisse les deux labs partager un seul chemin de code. KubePrism reste disponible si tu veux basculer.

⚠️ Calico ne peut pas annoncer d'IP de LoadBalancer. Il ne sait le faire qu'en BGP, ce qui suppose un routeur pair, inexistant sur un réseau host-only VirtualBox. Avec CNI=calico il faut donc installer MetalLB (mode L2) ; platform-up.sh retire aussi le loadBalancerClass: io.cilium/l2-announcer propre à Cilium d'Envoy-Proxy.yml pour qu'un autre annonceur puisse prendre le relais. Procédure complète : k8s-playground calico/.

⚠️ Quel que soit le CNI, épingle l'interface host-only (enp0s8). Sinon il prend la carte de la route par défaut (le NAT, 10.0.2.15, identique sur toutes les VM), et les tunnels VXLAN sont cassés alors que la sortie Internet fonctionne encore, ce qui donne une panne de DNS très déroutante.

🛠️ 9. Valider une modification#

Tout se valide sans toucher au cluster :

make validate       # syntaxe des scripts + YAML + Vagrantfile + config Talos + liens de doc
make docs           # régénère docs/index.html depuis tous les README (EN + FR)
make help           # liste les cibles

make validate-talos génère la config dans un dossier temporaire, puis la passe à talosctl validate --mode metal : aucun risque pour _out/ ni pour le cluster, contrairement à FORCE=1 ./talos/cluster-up.sh, qui régénère les secrets et casse un cluster vivant. Elle affiche aussi les versions utilisées, moyen le moins cher de confirmer que tes clés lab.env sont vraiment lues. make validate-docs construit la doc dans un dossier jetable et échoue si un lien *.md ou une ancre inter-pages ne résout plus, et make validate-submodule vérifie le pointeur _k8s lui-même : une URL en https:// (une URL SSH casse le clone pour quiconque n'a pas de clé GitHub) et un commit épinglé réellement poussé.

À chaque pull request, le workflow ci rejoue les contrôles shell, YAML et Vagrantfile en appelant les mêmes cibles make, donc un contrôle ne peut pas passer en CI et échouer chez toi. vagrant validate y tourne avec --ignore-provider, un runner n'ayant pas VirtualBox.

ℹ️ validate-shell et validate-yaml ne couvrent que les fichiers suivis par ce dépôt. Le sous-module _k8s/ est un pointeur unique, donc aucun de ses scripts ni manifestes n'est vérifié ici ; ils le sont dans la CI de k8s-playground.

📄 10. Licence#

Apache License 2.0. Voir LICENSE. Utilise-le, modifie-le, redistribue-le, y compris commercialement, tant que tu conserves la notice de copyright et que tu signales tes modifications. Aucune garantie : c'est un lab, pas de production.

Elle couvre ce que ce dépôt contient : le Vagrantfile, les scripts talos/, les patches de config et la documentation. Elle ne s'étend pas aux composants tiers que ces scripts téléchargent (Talos Linux, Cilium, Longhorn, Vault, Envoy Gateway…), ni au sous-module _k8s/ : k8s-playground porte sa propre LICENSE.

⬆️Upgrade Talos (and Kubernetes)

Reference at test time: Talos v1.13.7, Kubernetes v1.36.2, CNI=cilium, 3 CP + 5 workers. Adapt the IPs to your topology (lab.env); the repo ships 3 CP + 3 workers.

⚠️ talosctl must be ≥ the target version: talosctl version --client.

🎯 1. How Talos upgrades itself#

Talos has no SSH and no package manager: an upgrade replaces the system image on disk, it patches nothing in place.

talosctl -n <node-ip> upgrade --image ghcr.io/siderolabs/installer:<vX.Y.Z>

The new image is written to the inactive partition and the node reboots onto it — A/B scheme, so a failed boot rolls back automatically (manual rollback: talosctl -n <ip> rollback). The upgrade preserves etcd and the machine config. The EPHEMERAL partition (/var) is kept except without --preserve on a single node; in HA you can wipe a node and let etcd rebuild from the quorum, but for a node that stores data (Longhorn → /var/lib/longhorn): always --preserve.

🔢 2. The version model in this lab#

Four version references coexist, and they do not move together:

⚠️ INSTALLER_IMAGE masks TALOS_VERSION. lab.env.example sets an Image Factory image whose tag carries its own version (factory.talos.dev/installer/<schematic>:v1.13.7). Bumping TALOS_VERSION alone then changes the ISO only and the disk stays on the old version — both lines must move together, along with the fallback defaults in the Vagrantfile and cluster-up.sh (also on v1.13.7), otherwise a lab brought up without lab.env restarts on the old version.

☸️ Kubernetes is the fourth axis, independent of the other three. KUBERNETES_VERSION becomes talosctl gen config --kubernetes-version; left empty it falls back to what the talosctl binary ships (v1.36.2 for talosctl v1.13.7). It is read only when the config is generated — on a running cluster it is upgrade-k8s that does the work (§4). Nothing validates the value: a version outside the skew Talos supports lands as an ErrImagePull on the static pods.

For an upgrade the ISO is irrelevant: you change the installer image of the already installed nodes (§3), then update lab.env for future rebuilds.

⚡ 3. Procedure (running cluster)#

Pre-flight — never start from an already degraded cluster:

export TALOSCONFIG=_out/talosconfig KUBECONFIG=./kubeconfig
talosctl -n 192.168.56.10 -e 192.168.56.10 health        # healthy cluster
talosctl -n 192.168.56.10 -e 192.168.56.10 etcd status   # 3 healthy members

Order: one node at a time, workers first, then the control planes.

NEW=v1.14.x                                    # target version
IMG=ghcr.io/siderolabs/installer:${NEW}        # ⚠️ see §5 if the nodes carry extensions

# a) Workers, one by one
for ip in 101 102 103 104 105; do
  talosctl -n 192.168.56.$ip -e 192.168.56.10 upgrade --image "$IMG" --preserve --wait
  kubectl wait --for=condition=Ready node/talos-w$((ip-100)) --timeout=5m
done

# b) Control planes, one by one, etcd checked BETWEEN each one
for ip in 10 20 30; do
  talosctl -n 192.168.56.$ip -e 192.168.56.20 upgrade --image "$IMG" --preserve --wait
  talosctl -n 192.168.56.10 -e 192.168.56.10 etcd status
done

⚠️ Never upgrade two CPs in parallel: the etcd quorum is 2/3, and losing two of them breaks the cluster. The .5 VIP switches over to another CP on its own during the reboot.

⚠️ -e/--endpoints must never point at the target node. For a CP, aim at another CP (otherwise you lose access when it reboots); a worker serves no kubeconfig at all — details in §6.

⚠️ On 2-3 GB VMs, let the disk/etcd load settle between two nodes: I/O starvation breaks the quorum.

☸️ 4. Upgrading Kubernetes#

Talos and Kubernetes upgrade independently.

talosctl -n 192.168.56.10 -e 192.168.56.10 upgrade-k8s --to 1.37.x

This orchestrates the static pods' apiserver / controller-manager / scheduler / kubelet, one component at a time. Check the supported Talos ↔ Kubernetes skew in the Talos release notes first.

💡 Then align KUBERNETES_VERSION in lab.env (and in the lab.env.example template if it is a repo-wide bump): upgrade-k8s only touches the live cluster, so without that line the next vagrant destroy + cluster-up.sh rebuilds on the old version. That variable is the fresh install path, upgrade-k8s is the running cluster path — same version, two mechanisms (§2).

🧩 5. Adding extensions = the same mechanism#

Adding iscsi-tools / util-linux-tools (required by Longhorn) is not a kubectl job: it is an upgrade to an Image Factory installer image that bakes them in. schematic.yaml lives in the _k8s/ submodule, so it assumes the submodule is checked out.

SCHEMATIC_ID=$(curl -sX POST --data-binary @_k8s/longhorn/schematic.yaml \
  https://factory.talos.dev/schematics -H "Content-Type: application/yaml" | jq -r .id)

talosctl -n 192.168.56.101 -e 192.168.56.10 upgrade \
  --image factory.talos.dev/installer/${SCHEMATIC_ID}:v1.13.7 --preserve --wait

talosctl -n 192.168.56.101 get extensions      # iscsi-tools + util-linux-tools present

⚠️ Never upgrade a "factory" node to the classic ghcr.io installer: that strips the extensions and breaks Longhorn. The factory image tag must carry the target version, with the same schematic ID.

💡 For a fresh cluster it is simpler to add --config-patch @_k8s/longhorn/patch-longhorn.yaml to the gen config — see k8s-playground — longhorn/.

📊 6. Real test: v1.13.5 → v1.13.7#

Run on 3 CP (3 GB / 3 vCPU) + 5 workers (2 GB / 2 vCPU), with Longhorn and Argo CD deployed, rolling one node at a time, with a probe hitting https://192.168.56.5:6443/livez every ~1 s.

API downtime: none. 1056 probes over ~17 min covering the 8 reboots (the 3 CPs included) → 100 % answered, 0 DOWN, longest outage 0 s. The VIP switching between control planes is transparent at one-second granularity. etcd stayed at 3/3, Kubernetes unchanged (v1.36.2), and the extensions survived (iscsi-tools + util-linux-tools still present, thanks to the factory image tagged :v1.13.7 with the same schematic ID as :v1.13.5).

Two pitfalls hit along the way:

1. Endpoint = the target node → failure. The drain done by talosctl upgrade fetches the kubeconfig through the endpoint, and a worker serves none (Unimplemented: kubeconfig is only available on control plane nodes), so the upgrade errors out before the reboot even starts. Point --endpoints at a control plane — and to upgrade a CP, at a CP other than the target.
2. Drain stuck on Longhorn. The instance-manager PodDisruptionBudget blocks eviction and the drain runs until --drain-timeout (5 min). On a lab, --drain=false (straight reboot; --preserve keeps /var/lib/longhorn and Longhorn rebuilds the replicas when the node returns). In production: tune Longhorn's node drain policy.

✅ 7. After the upgrade#

talosctl -n 192.168.56.10 version
kubectl get nodes -o wide

• Bump TALOS_VERSION and INSTALLER_IMAGE in lab.env (and in lab.env.example), so future vagrant up / cluster-up.sh runs start on the right version, ISO and installer.
• Bump the local talosctl binary to stay aligned.
• If you also moved Kubernetes (§4), set KUBERNETES_VERSION — otherwise the next rebuild silently goes back to the version the talosctl binary ships.

📚 References#

• Talos — Upgrading Talos Linux
• Talos — Upgrading Kubernetes
• Image Factory — extensions baked into the installer
• k8s-playground — longhorn/ — Longhorn's Talos prerequisites, in the _k8s/ submodule

⬆️Upgrade Talos (et Kubernetes)

Référence au moment du test : Talos v1.13.7, Kubernetes v1.36.2, CNI=cilium, 3 CP + 5 workers. Adapte les IP à ta topologie (lab.env) ; le dépôt livre 3 CP + 3 workers.

⚠️ talosctl doit être ≥ la version cible : talosctl version --client.

🎯 1. Comment Talos se met à jour#

Talos n'a ni SSH ni gestionnaire de paquets : une montée de version remplace l'image système sur le disque, elle ne corrige rien en place.

talosctl -n <ip-du-node> upgrade --image ghcr.io/siderolabs/installer:<vX.Y.Z>

La nouvelle image est écrite sur la partition inactive et le node redémarre dessus — schéma A/B, donc un démarrage raté revient en arrière automatiquement (rollback manuel : talosctl -n <ip> rollback). La montée préserve etcd et la config machine. La partition EPHEMERAL (/var) est conservée sauf sans --preserve sur un node unique ; en HA tu peux effacer un node et laisser etcd le reconstruire depuis le quorum, mais pour un node qui stocke des données (Longhorn → /var/lib/longhorn) : toujours --preserve.

🔢 2. Le modèle de versions de ce lab#

Quatre références de version coexistent, et elles ne bougent pas ensemble :

⚠️ INSTALLER_IMAGE masque TALOS_VERSION. lab.env.example pose une image Image Factory dont le tag porte sa propre version (factory.talos.dev/installer/<schematic>:v1.13.7). Incrémenter TALOS_VERSION seul ne change alors que l'ISO et le disque reste sur l'ancienne version — les deux lignes doivent bouger ensemble, ainsi que les défauts de repli du Vagrantfile et de cluster-up.sh (également sur v1.13.7), sinon un lab monté sans lab.env repart sur l'ancienne version.

☸️ Kubernetes est le quatrième axe, indépendant des trois autres. KUBERNETES_VERSION devient talosctl gen config --kubernetes-version ; laissée vide, elle retombe sur ce que livre le binaire talosctl (v1.36.2 pour talosctl v1.13.7). Elle n'est lue qu'à la génération de la config — sur un cluster vivant, c'est upgrade-k8s qui travaille (§4). Rien ne valide la valeur : une version hors de l'écart supporté par Talos se traduit par un ErrImagePull sur les pods statiques.

Pour une montée de version, l'ISO est hors sujet : tu changes l'image d'installation des nodes déjà installés (§3), puis tu mets lab.env à jour pour les reconstructions futures.

⚡ 3. Procédure (cluster vivant)#

Pré-vol — ne jamais partir d'un cluster déjà dégradé :

export TALOSCONFIG=_out/talosconfig KUBECONFIG=./kubeconfig
talosctl -n 192.168.56.10 -e 192.168.56.10 health        # cluster sain
talosctl -n 192.168.56.10 -e 192.168.56.10 etcd status   # 3 membres sains

Ordre : un node à la fois, les workers d'abord, puis les control planes.

NEW=v1.14.x                                    # version cible
IMG=ghcr.io/siderolabs/installer:${NEW}        # ⚠️ voir le §5 si les nodes portent des extensions

# a) Workers, un par un
for ip in 101 102 103 104 105; do
  talosctl -n 192.168.56.$ip -e 192.168.56.10 upgrade --image "$IMG" --preserve --wait
  kubectl wait --for=condition=Ready node/talos-w$((ip-100)) --timeout=5m
done

# b) Control planes, un par un, etcd vérifié ENTRE chacun
for ip in 10 20 30; do
  talosctl -n 192.168.56.$ip -e 192.168.56.20 upgrade --image "$IMG" --preserve --wait
  talosctl -n 192.168.56.10 -e 192.168.56.10 etcd status
done

⚠️ Ne monte jamais deux CP en parallèle : le quorum etcd est de 2/3, en perdre deux casse le cluster. La VIP .5 bascule toute seule vers un autre CP pendant le redémarrage.

⚠️ -e/--endpoints ne doit jamais pointer le node cible. Pour un CP, visez un autre CP (sinon tu perds l'accès quand il redémarre) ; un worker ne sert aucun kubeconfig — détails au §6.

⚠️ Sur des VM de 2-3 Go, laisse la charge disque/etcd retomber entre deux nodes : la famine d'I/O casse le quorum.

☸️ 4. Upgrade de Kubernetes#

Talos et Kubernetes montent de version indépendamment.

talosctl -n 192.168.56.10 -e 192.168.56.10 upgrade-k8s --to 1.37.x

Ça orchestre les pods statiques apiserver / controller-manager / scheduler / kubelet, un composant à la fois. Vérifie d'abord l'écart Talos ↔ Kubernetes supporté dans les notes de version de Talos.

💡 Aligne ensuite KUBERNETES_VERSION dans lab.env (et dans le modèle lab.env.example s'il s'agit d'une montée pour tout le dépôt) : upgrade-k8s ne touche que le cluster vivant, donc sans cette ligne le prochain vagrant destroy + cluster-up.sh reconstruit sur l'ancienne version. Cette variable est le chemin installation neuve, upgrade-k8s le chemin cluster vivant — même version, deux mécanismes (§2).

🧩 5. Ajouter des extensions = le même mécanisme#

Ajouter iscsi-tools / util-linux-tools (requis par Longhorn) n'est pas un travail de kubectl : c'est une montée vers une image d'installation Image Factory qui les intègre. schematic.yaml vit dans le sous-module _k8s/, donc les chemins ci-dessous supposent le sous-module sorti.

SCHEMATIC_ID=$(curl -sX POST --data-binary @_k8s/longhorn/schematic.yaml \
  https://factory.talos.dev/schematics -H "Content-Type: application/yaml" | jq -r .id)

talosctl -n 192.168.56.101 -e 192.168.56.10 upgrade \
  --image factory.talos.dev/installer/${SCHEMATIC_ID}:v1.13.7 --preserve --wait

talosctl -n 192.168.56.101 get extensions      # iscsi-tools + util-linux-tools présents

⚠️ Ne monte jamais un node « factory » vers l'installeur classique ghcr.io : ça retire les extensions et casse Longhorn. Le tag de l'image factory doit porter la version cible, avec le même schematic ID.

💡 Pour un cluster neuf, il est plus simple d'ajouter --config-patch @_k8s/longhorn/patch-longhorn.yaml au gen config — voir k8s-playground — longhorn/.

📊 6. Test réel : v1.13.5 → v1.13.7#

Exécuté sur 3 CP (3 Go / 3 vCPU) + 5 workers (2 Go / 2 vCPU), avec Longhorn et Argo CD déployés, un node à la fois, avec une sonde sur https://192.168.56.5:6443/livez toutes les ~1 s.

Interruption de l'API : aucune. 1056 sondes sur ~17 min couvrant les 8 redémarrages (les 3 CP compris) → 100 % de réponses, 0 DOWN, plus longue coupure 0 s. Le basculement de la VIP entre control planes est transparent à la seconde. etcd est resté à 3/3, Kubernetes inchangé (v1.36.2), et les extensions ont survécu (iscsi-tools + util-linux-tools toujours présents, grâce à l'image factory taguée :v1.13.7 avec le même schematic ID que :v1.13.5).

Deux pièges rencontrés en route :

1. Endpoint = le node cible → échec. La vidange faite par talosctl upgrade récupère le kubeconfig via l'endpoint, et un worker n'en sert aucun (Unimplemented: kubeconfig is only available on control plane nodes), donc la montée sort en erreur avant même le redémarrage. Pointe --endpoints sur un control plane — et pour monter un CP, sur un CP autre que la cible.
2. Vidange bloquée sur Longhorn. Le PodDisruptionBudget d'instance-manager bloque l'éviction et la vidange court jusqu'à --drain-timeout (5 min). Sur un lab, --drain=false (redémarrage direct ; --preserve garde /var/lib/longhorn et Longhorn reconstruit les répliques au retour du node). En production : règle la node drain policy de Longhorn.

✅ 7. Après la montée de version#

talosctl -n 192.168.56.10 version
kubectl get nodes -o wide

• Incrémente TALOS_VERSION et INSTALLER_IMAGE dans lab.env (et dans lab.env.example), pour que les futurs vagrant up / cluster-up.sh partent sur la bonne version, ISO et installeur.
• Incrémente le binaire talosctl local pour rester aligné.
• Si tu as aussi déplacé Kubernetes (§4), pose KUBERNETES_VERSION — sinon la prochaine reconstruction revient en silence à la version que livre le binaire talosctl.

📚 Références#

• Talos — Upgrading Talos Linux
• Talos — Upgrading Kubernetes
• Image Factory — extensions intégrées à l'installeur
• k8s-playground — longhorn/ — les prérequis Talos de Longhorn, dans le sous-module _k8s/

🚑Troubleshooting

This page covers the lab itself: the host, VirtualBox, addressing and the Talos nodes. Addon problems (Longhorn, Vault, Calico…) are documented with the addons, in k8s-playground.

Unless stated otherwise, every command runs from the repository root, with:

export TALOSCONFIG="$PWD/_out/talosconfig"
export KUBECONFIG="$PWD/kubeconfig"

🖥️ 1. Host, repository and VirtualBox#

VT-x conflict: unload KVM before starting VirtualBox#

VirtualBox and KVM cannot use VT-x at the same time, and most Linux distributions load KVM at boot:

VBoxManage: error: VT-x is being used by another hypervisor (VERR_VMX_IN_VMX_ROOT_MODE).

lsmod | grep kvm                    # Intel: kvm_intel — AMD: kvm_amd
sudo modprobe -r kvm_intel kvm      # fails if a KVM/libvirt VM is still running

💡 KVM is reloaded on every boot. If this host is never used for KVM/libvirt, blacklist it once:

echo -e "blacklist kvm_intel\nblacklist kvm" | sudo tee /etc/modprobe.d/disable-kvm.conf

VirtualBox refuses the 192.168.56.0/24 network#

Allow the range in /etc/vbox/networks.conf:

* 192.168.56.0/21

vagrant up fails after a destroy (VirtualBox leftovers)#

VirtualBox 7.x (linked clones) does not always clean up after a destroy:

The name of your virtual machine couldn't be set because VirtualBox
is reporting another VM with that name already exists.
VBoxManage: error: Could not rename the directory '.../temp_clone_...'
to '.../talos-cp1' ... (VERR_ALREADY_EXISTS)

Two layers of leftovers pile up: orphan directories ~/VirtualBox VMs/talos-*/ and dead entries in the media registry (talos-* disks still registered, plus accumulated inaccessible entries), which then make the up fail on "medium already registered".

DRY_RUN=1 ./talos/virtualbox-cleanup.sh   # shows what would be deleted
./talos/virtualbox-cleanup.sh             # actually purges

⚠️ Run it after vagrant destroy, never on a running cluster. The script targets the talos- prefix (PREFIX=) and the temp_clone_* VMs: if another Vagrant project is in the middle of an up on the same machine, its temporary clone would be deleted too.

💡 A destroy that reports success can still leave those directories behind, each holding a small snapshot. Run the cleanup after every destroy, not just after a visibly failed one. In dry-run the directories show up as "kept (contains files)": that is normal, the real run deletes the disk first and then finds them empty.

vagrant up fails on storagectl … --remove SAS#

The pace/empty box exposes its disk on a controller named SAS (replaced here with SATA/AHCI). If a future version of the box changes that name, list it with VBoxManage showvminfo <vm> | grep -i "Storage Controller Name" and adjust the Vagrantfile.

⚠️ The Vagrantfile uses the existence of the disk as a provisioning sentinel. If a destroy fails and leaves .vagrant/talos-disks/<vm>.vdi behind, the next up creates a VM with no disk attached and the install dies with an obscure error. Clean up with ./talos/virtualbox-cleanup.sh.

_k8s/ is empty — ./_k8s/install.sh: No such file or directory#

_k8s/ is a git submodule pointing at k8s-playground. A plain git clone records it but does not check it out.

git submodule update --init --recursive     # fills _k8s/
git -C _k8s log --oneline -1                # sanity check

Clone correctly next time with git clone --recurse-submodules <url>. git pull does not update the submodule either — repeat the command above after every pull, or git submodule update --remote _k8s to jump to the latest upstream commit.

The _k8s/ scripts find neither lab.env nor the kubeconfig#

Symptoms: addons install into the wrong domain (lab.example.io instead of your LAB_DOMAIN), the wrong CNI is chosen, or kubectl inside the scripts fails with connection refused. The banner the scripts print at start-up shows lab.env: absent (defaults). Nothing crashes — the run silently falls back on the built-in defaults and, with no kubeconfig, installs the addon against nothing. The Talos-specific consequence: no _out/talosconfig either, so any addon calling talosctl (Longhorn, local-path) fails on an unconfigured Talos API.

The lab was not located. k8s-playground takes the parent directory of _k8s/ as the lab, as long as that directory carries a Vagrantfile — mounted as a submodule, that parent is this clone.

ls Vagrantfile _k8s/     # from the lab root: both must exist
ls ../Vagrantfile        # from inside _k8s/: the same check, seen from the scripts

This legitimately breaks in two cases only: _k8s/ was cloned or moved outside the lab, or a copy of the scripts is being run from somewhere else. Run them from the clone that carries the Vagrantfile, or point at the lab explicitly:

LAB_DIR=/path/to/Vagrant-Talos ./_k8s/platform-up.sh

💡 LAB_DIR is an explicit override and takes priority over auto-detection; LAB_ENV=/path/to/lab.env does the same for that one file. Neither is needed in the normal case. TALOSCONFIG and KUBECONFIG are a different matter — keep exporting those, the addons driving the Talos API depend on them.

🗺️ 2. Addressing and DHCP#

A node does not get its .x IP#

Talos retries DHCP in a loop: wait ~30 s. Otherwise vagrant reload <node> (the trigger re-arms the host-only DHCP with the reservations). To see a VM's real IP, open its console (vb.gui = false → true in the Vagrantfile): Talos prints its IP on screen.

A node takes an unexpected IP (stale DHCP leases)#

Symptom: talosctl -n <reserved-ip> … --insecure returns no route to host while another IP answers. VirtualBox honours an already-acked DHCP lease before applying the MAC→IP reservations, so an old lease (typically in the ~.100 range, inherited from vboxnet0's default DHCP server) overrides the reservation.

The before :up trigger creates the reservations and purges those leases before the VMs boot, so every node gets its reserved IP on its 1st DHCP DISCOVER; the after :destroy trigger purges them too. To fix an already started cluster without destroying everything:

# 1. power off the nodes (maintenance mode => no data lost)
for v in talos-cp1 talos-cp2 talos-cp3; do VBoxManage controlvm "$v" poweroff; done

# 2. purge the host-only network lease file (adjust vboxnet0 if needed)
CFG="${VBOX_USER_HOME:-$HOME/.config/VirtualBox}"
rm -f "$CFG"/HostInterfaceNetworking-vboxnet0-Dhcpd.leases*
VBoxManage dhcpserver restart --network HostInterfaceNetworking-vboxnet0

# 3. power back on: the nodes redo a DHCP DISCOVER and get their reserved IP
vagrant up

Check: talosctl -n 192.168.56.10 version --insecure must answer NODE: 192.168.56.10.

The VIP 192.168.56.5 is unreachable#

The VIP only appears after etcd's bootstrap. Check that the host-only NIC really is 0000:00:08.0: talosctl -n 192.168.56.10 get links, then get addresses. If the interface differs, adjust busPath in talos/patch-cp.yaml.

🐧 3. Talos nodes#

talosctl … --insecure does not answer#

The node is not in maintenance mode yet, or has no host-only IP. Check talosctl -n <ip> get disks --insecure and §2.

⚠️ An already installed node (secure mode) never answers --insecure: expected, not a fault. This is exactly why cluster-up.sh must not be re-run on a live cluster — to grow one, see §6.1 of the README.

The Talos console shows KUBERNETES: n/a#

Normal before apply-config. The dashboard derives that version from the kubelet image tag in the KubeletSpec resource, which only exists once the machine config has been applied — in maintenance mode no kubelet is configured. Check outside the console with talosctl -n <ip> get kubeletspec.

The install disk is not /dev/sda#

Check with talosctl -n <ip> get disks --insecure and adjust INSTALL_DISK.

☸️ 4. Cluster and pods#

Pods can ping the Internet but have no DNS#

Symptom: ping 1.1.1.1 works from a pod, but nslookup/apk update fail (DNS: transient error).

flannel picks the public IP of its VXLAN tunnel on the default route interface = the NAT NIC (10.0.2.15, identical on every VM). All the VTEPs then point at an isolated NAT, so cross-node pod traffic is broken. DNS fails because CoreDNS often runs on a different node than the client pod; Internet egress, on the other hand, leaves through the local NAT and works — which is what makes this so confusing.

kubectl get nodes -o custom-columns='NODE:.metadata.name,FLANNEL-IP:.metadata.annotations.flannel\.alpha\.coreos\.com/public-ip'
# KO if FLANNEL-IP = 10.0.2.15 everywhere; OK if = 192.168.56.10/.20/.30

The fix lives in talos/cni-flannel.yaml (--iface-can-reach=192.168.56.1) and is picked up at bootstrap on a rebuild. On an already started cluster, Talos does not re-push the manifest update, so patch the DaemonSet:

kubectl -n kube-system patch ds kube-flannel --type=json \
  -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--iface-can-reach=192.168.56.1"}]'
kubectl -n kube-system rollout status ds/kube-flannel

ℹ️ Same root cause, same countermeasure for the other CNIs: Cilium pins devices=enp0s8, Calico pins nodeAddressAutodetectionV4.cidrs. The NAT NIC being identical on every VM is the recurring trap of this lab — see README.md.

The nodes stay NotReady after the bootstrap#

Expected with CNI=cilium, calico or none: Talos installs no CNI, and a node without a pod network never reports Ready. ./_k8s/platform-up.sh installs it in its first step and unblocks them. Only flannel is laid down by Talos itself, at bootstrap time.

If they are still NotReady after the CNI install, look at the CNI pods first (kubectl -n kube-system get pods for Cilium, kubectl -n calico-system get pods for Calico), then the addon's own page on https://ops-nc.github.io/k8s-playground/.

There is no kube-proxy DaemonSet#

Expected, and it is the lab default: KUBE_PROXY_REPLACEMENT=true makes cluster-up.sh add talos/patch-no-kube-proxy.yaml (cluster.proxy.disabled: true), so the bootstrap renders no kube-proxy manifest and Cilium serves the Services in eBPF.

kubectl -n kube-system get ds kube-proxy      # NotFound => expected
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status --verbose \
  | grep KubeProxyReplacement                 # must say True

No ClusterIP answers any more (CoreDNS included)#

The pathological case of the previous entry: kube-proxy is gone and nothing replaced it. It happens when KUBE_PROXY_REPLACEMENT and the CNI actually installed disagree — typically a lab.env edited after the bootstrap, or a Cilium installed by hand with kubeProxyReplacement=false on a cluster bootstrapped with true.

kubectl -n kube-system get ds kube-proxy                      # absent?
grep -A2 '^    proxy:' _out/controlplane.yaml                 # what the bootstrap really did
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status | grep KubeProxy

The machine config and cilium-dbg are the ground truth, not lab.env. Realign Cilium (./_k8s/cilium/cilium-up.sh with the right value), or rebuild the cluster — the bootstrap decision itself cannot be changed live. See README.md §8.

An HTTPS UI is unreachable#

Work down the chain: the Gateway must have an EXTERNAL-IP (kubectl -n envoy-gateway-system get svc), the name must resolve to it, and an HTTPRoute must match that hostname (kubectl get httproute -A).

⚠️ Do not test the Gateway IP with ping. A Service IP announced in L2 by Cilium answers ARP and TCP but not ICMP — no interface actually carries the address, so a failing ping on .200 proves nothing while ping on a node works. The real proof is the ARP entry resolving to a node's MAC:

sudo ip neigh flush 192.168.56.200
curl -s -o /dev/null --max-time 5 http://192.168.56.200/    # 404 = Envoy answers
ip neigh show 192.168.56.200                                # lladdr = the announcing node

ℹ️ On the bare IP, http:// answers 404 (Envoy is listening, no route matches) but https:// answers nothing at all: the TLS listener is scoped by hostname, so a request without SNI matches no listener. Test with the name instead, short-circuiting DNS if needed: curl -sk --resolve argo.talos.lab.example.io:443:192.168.56.200 https://argo.talos.lab.example.io/.

With the default SELF_SIGNED=true a browser warning is expected until you import _out/self-signed/ca.crt — as it is with LAB_ACME_ISSUER=staging, whose certificates are real but untrusted.

🚑Dépannage

Cette page couvre le lab lui-même : l'hôte, VirtualBox, l'adressage et les nodes Talos. Les problèmes d'addons (Longhorn, Vault, Calico…) sont documentés avec les addons, dans k8s-playground.

Sauf mention contraire, toutes les commandes se lancent depuis la racine du dépôt, avec :

export TALOSCONFIG="$PWD/_out/talosconfig"
export KUBECONFIG="$PWD/kubeconfig"

🖥️ 1. Hôte, dépôt et VirtualBox#

Conflit VT-x : décharger KVM avant de lancer VirtualBox#

VirtualBox et KVM ne peuvent pas utiliser VT-x en même temps, et la plupart des distributions Linux chargent KVM au démarrage :

VBoxManage: error: VT-x is being used by another hypervisor (VERR_VMX_IN_VMX_ROOT_MODE).

lsmod | grep kvm                    # Intel : kvm_intel — AMD : kvm_amd
sudo modprobe -r kvm_intel kvm      # échoue si une VM KVM/libvirt tourne encore

💡 KVM est rechargé à chaque démarrage. Si cet hôte ne sert jamais à KVM/libvirt, blackliste-le une fois :

echo -e "blacklist kvm_intel\nblacklist kvm" | sudo tee /etc/modprobe.d/disable-kvm.conf

VirtualBox refuse le réseau 192.168.56.0/24#

Autorise la plage dans /etc/vbox/networks.conf :

* 192.168.56.0/21

vagrant up échoue après un destroy (résidus VirtualBox)#

VirtualBox 7.x (clones liés) ne nettoie pas toujours après un destroy :

The name of your virtual machine couldn't be set because VirtualBox
is reporting another VM with that name already exists.
VBoxManage: error: Could not rename the directory '.../temp_clone_...'
to '.../talos-cp1' ... (VERR_ALREADY_EXISTS)

Deux couches de résidus s'accumulent : des dossiers orphelins ~/VirtualBox VMs/talos-*/ et des entrées mortes dans le registre de médias (disques talos-* toujours enregistrés, plus des entrées inaccessible accumulées), qui font ensuite échouer le up sur « medium already registered ».

DRY_RUN=1 ./talos/virtualbox-cleanup.sh   # montre ce qui serait supprimé
./talos/virtualbox-cleanup.sh             # purge réellement

⚠️ Lance-le après vagrant destroy, jamais sur un cluster vivant. Le script cible le préfixe talos- (PREFIX=) et les VM temp_clone_* : si un autre projet Vagrant est en pleine exécution d'un up sur la même machine, son clone temporaire serait supprimé aussi.

💡 Un destroy qui annonce un succès peut quand même laisser ces dossiers derrière, chacun avec un petit snapshot. Lance le nettoyage après chaque destroy, pas seulement après un échec visible. En dry-run les dossiers apparaissent en « kept (contains files) » : c'est normal, la vraie passe supprime d'abord le disque puis les trouve vides.

vagrant up échoue sur storagectl … --remove SAS#

La box pace/empty expose son disque sur un contrôleur nommé SAS (remplacé ici par SATA/AHCI). Si une future version de la box change ce nom, liste-le avec VBoxManage showvminfo <vm> | grep -i "Storage Controller Name" et ajuste le Vagrantfile.

⚠️ Le Vagrantfile utilise l'existence du disque comme sentinelle de provisioning. Si un destroy échoue et laisse .vagrant/talos-disks/<vm>.vdi, le up suivant crée une VM sans disque attaché et l'installation meurt sur une erreur obscure. Nettoie avec ./talos/virtualbox-cleanup.sh.

_k8s/ est vide — ./_k8s/install.sh: No such file or directory#

_k8s/ est un sous-module git pointant k8s-playground. Un git clone simple l'enregistre mais ne le sort pas.

git submodule update --init --recursive     # remplit _k8s/
git -C _k8s log --oneline -1                # contrôle rapide

La prochaine fois, clone correctement : git clone --recurse-submodules <url>. git pull ne met pas non plus le sous-module à jour — répète la commande ci-dessus après chaque pull, ou git submodule update --remote _k8s pour sauter au dernier commit amont.

Les scripts _k8s/ ne trouvent ni lab.env ni le kubeconfig#

Symptômes : les addons s'installent sur le mauvais domaine (lab.example.io au lieu de ton LAB_DOMAIN), le mauvais CNI est choisi, ou kubectl échoue dans les scripts sur connection refused. La bannière affichée au démarrage indique lab.env: absent (defaults). Rien ne plante — l'exécution retombe en silence sur les défauts internes et, sans kubeconfig, installe l'addon contre rien. Conséquence propre à Talos : pas de _out/talosconfig non plus, donc tout addon qui appelle talosctl (Longhorn, local-path) échoue sur une API Talos non configurée.

Le lab n'a pas été localisé. k8s-playground prend comme lab le dossier parent de _k8s/, à condition qu'il porte un Vagrantfile — monté en sous-module, ce parent est ce clone.

ls Vagrantfile _k8s/     # depuis la racine du lab : les deux doivent exister
ls ../Vagrantfile        # depuis _k8s/ : le même test, vu par les scripts

Ça ne casse légitimement que dans deux cas : _k8s/ a été cloné ou déplacé hors du lab, ou une copie des scripts est lancée depuis ailleurs. Lance-les depuis le clone qui porte le Vagrantfile, ou désigne le lab explicitement :

LAB_DIR=/chemin/vers/Vagrant-Talos ./_k8s/platform-up.sh

💡 LAB_DIR est une surcharge explicite et prime sur l'auto-détection ; LAB_ENV=/chemin/vers/lab.env fait pareil pour ce seul fichier. Aucun des deux n'est nécessaire dans le cas normal. TALOSCONFIG et KUBECONFIG sont un autre sujet — continue de les exporter, les addons qui pilotent l'API Talos en dépendent.

🗺️ 2. Adressage et DHCP#

Un node n'obtient pas son IP .x#

Talos réessaie le DHCP en boucle : attends ~30 s. Sinon vagrant reload <node> (le trigger réarme le DHCP host-only avec les réservations). Pour voir l'IP réelle d'une VM, ouvre sa console (vb.gui = false → true dans le Vagrantfile) : Talos affiche son IP à l'écran.

Un node prend une IP inattendue (baux DHCP périmés)#

Symptôme : talosctl -n <ip-réservée> … --insecure répond no route to host alors qu'une autre IP répond. VirtualBox honore un bail DHCP déjà acked avant d'appliquer les réservations MAC→IP : un vieux bail (typiquement dans la plage ~.100, héritée du serveur DHCP par défaut de vboxnet0) prend le pas sur la réservation.

Le trigger before :up crée les réservations et purge ces baux avant le démarrage des VM, pour que chaque node obtienne son IP réservée dès son 1er DHCP DISCOVER ; le trigger after :destroy les purge aussi. Pour réparer un cluster déjà démarré sans tout détruire :

# 1. éteindre les nodes (mode maintenance => aucune donnée perdue)
for v in talos-cp1 talos-cp2 talos-cp3; do VBoxManage controlvm "$v" poweroff; done

# 2. purger le fichier de baux du réseau host-only (adapte vboxnet0 si besoin)
CFG="${VBOX_USER_HOME:-$HOME/.config/VirtualBox}"
rm -f "$CFG"/HostInterfaceNetworking-vboxnet0-Dhcpd.leases*
VBoxManage dhcpserver restart --network HostInterfaceNetworking-vboxnet0

# 3. rallumer : les nodes refont un DHCP DISCOVER et prennent leur IP réservée
vagrant up

Contrôle : talosctl -n 192.168.56.10 version --insecure doit répondre NODE: 192.168.56.10.

La VIP 192.168.56.5 est injoignable#

La VIP n'apparaît qu'après le bootstrap d'etcd. Vérifie que la carte host-only est bien 0000:00:08.0 : talosctl -n 192.168.56.10 get links, puis get addresses. Si l'interface diffère, ajuste busPath dans talos/patch-cp.yaml.

🐧 3. Nodes Talos#

talosctl … --insecure ne répond pas#

Le node n'est pas encore en mode maintenance, ou n'a pas d'IP host-only. Vérifie talosctl -n <ip> get disks --insecure et le §2.

⚠️ Un node déjà installé (mode sécurisé) ne répond jamais à --insecure : c'est attendu, pas une panne. C'est exactement pour ça que cluster-up.sh ne doit pas être rejoué sur un cluster vivant — pour en agrandir un, voir le §6.1 du LISEZ-MOI.

La console Talos affiche KUBERNETES: n/a#

Normal avant apply-config. Le tableau de bord déduit cette version du tag de l'image kubelet dans la ressource KubeletSpec, qui n'existe qu'une fois la config machine appliquée — en mode maintenance, aucun kubelet n'est configuré. Vérifie hors console avec talosctl -n <ip> get kubeletspec.

Le disque d'installation n'est pas /dev/sda#

Vérifie avec talosctl -n <ip> get disks --insecure et ajuste INSTALL_DISK.

☸️ 4. Cluster et pods#

Les pods pinguent Internet mais n'ont pas de DNS#

Symptôme : ping 1.1.1.1 fonctionne depuis un pod, mais nslookup/apk update échouent (DNS: transient error).

flannel choisit l'IP publique de son tunnel VXLAN sur l'interface de la route par défaut = la carte NAT (10.0.2.15, identique sur toutes les VM). Tous les VTEP pointent alors vers un NAT isolé, donc le trafic de pods inter-nodes est cassé. Le DNS échoue parce que CoreDNS tourne souvent sur un autre node que le pod client ; la sortie Internet, elle, part par le NAT local et fonctionne — ce qui rend le diagnostic déroutant.

kubectl get nodes -o custom-columns='NODE:.metadata.name,FLANNEL-IP:.metadata.annotations.flannel\.alpha\.coreos\.com/public-ip'
# KO si FLANNEL-IP = 10.0.2.15 partout ; OK si = 192.168.56.10/.20/.30

Le correctif vit dans talos/cni-flannel.yaml (--iface-can-reach=192.168.56.1) et est pris en compte au bootstrap lors d'une reconstruction. Sur un cluster déjà démarré, Talos ne repousse pas la mise à jour du manifeste : patche le DaemonSet.

kubectl -n kube-system patch ds kube-flannel --type=json \
  -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--iface-can-reach=192.168.56.1"}]'
kubectl -n kube-system rollout status ds/kube-flannel

ℹ️ Même cause racine, même contre-mesure pour les autres CNI : Cilium épingle devices=enp0s8, Calico épingle nodeAddressAutodetectionV4.cidrs. La carte NAT identique sur toutes les VM est LE piège récurrent de ce lab — cf. LISEZ-MOI.md.

Les nodes restent NotReady après le bootstrap#

Attendu avec CNI=cilium, calico ou none : Talos n'installe aucun CNI, et un node sans réseau de pods ne passe jamais Ready. ./_k8s/platform-up.sh l'installe à sa première étape et les débloque. Seul flannel est posé par Talos lui-même, au bootstrap.

S'ils sont toujours NotReady après l'installation du CNI, regarde d'abord les pods du CNI (kubectl -n kube-system get pods pour Cilium, kubectl -n calico-system get pods pour Calico), puis la page de l'addon sur https://ops-nc.github.io/k8s-playground/.

Il n'y a pas de DaemonSet kube-proxy#

Attendu, et c'est le défaut du lab : KUBE_PROXY_REPLACEMENT=true fait ajouter par cluster-up.sh le patch talos/patch-no-kube-proxy.yaml (cluster.proxy.disabled: true), donc le bootstrap ne rend aucun manifeste kube-proxy et Cilium sert les Services en eBPF.

kubectl -n kube-system get ds kube-proxy      # NotFound => attendu
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status --verbose \
  | grep KubeProxyReplacement                 # doit dire True

Plus aucune ClusterIP ne répond (CoreDNS compris)#

Le cas pathologique de l'entrée précédente : kube-proxy est parti et rien ne l'a remplacé. Ça arrive quand KUBE_PROXY_REPLACEMENT et le CNI réellement installé divergent — typiquement un lab.env édité après le bootstrap, ou un Cilium installé à la main avec kubeProxyReplacement=false sur un cluster bootstrapé avec true.

kubectl -n kube-system get ds kube-proxy                      # absent ?
grep -A2 '^    proxy:' _out/controlplane.yaml                 # ce que le bootstrap a vraiment fait
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status | grep KubeProxy

La config machine et cilium-dbg sont la vérité terrain, pas lab.env. Réaligne Cilium (./_k8s/cilium/cilium-up.sh avec la bonne valeur), ou reconstruis le cluster — la décision du bootstrap elle-même ne se change pas à chaud. Voir LISEZ-MOI.md §8.

Une UI HTTPS est injoignable#

Descends la chaîne : le Gateway doit avoir une EXTERNAL-IP (kubectl -n envoy-gateway-system get svc), le nom doit résoudre vers elle, et une HTTPRoute doit correspondre à ce nom d'hôte (kubectl get httproute -A).

⚠️ Ne teste pas l'IP du Gateway avec ping. Une IP de Service annoncée en L2 par Cilium répond à l'ARP et au TCP, mais pas à l'ICMP — aucune interface ne porte réellement l'adresse, donc un ping qui échoue sur .200 ne prouve rien, alors que le ping d'un node fonctionne. La vraie preuve, c'est l'entrée ARP qui se résout vers la MAC d'un node :

sudo ip neigh flush 192.168.56.200
curl -s -o /dev/null --max-time 5 http://192.168.56.200/    # 404 = Envoy répond
ip neigh show 192.168.56.200                                # lladdr = le node annonceur

ℹ️ Sur l'IP nue, http:// répond 404 (Envoy écoute, aucune route ne correspond) mais https:// ne répond rien du tout : le listener TLS est délimité par nom d'hôte, donc une requête sans SNI ne correspond à aucun listener. Teste avec le nom, en court-circuitant le DNS au besoin : curl -sk --resolve argo.talos.lab.example.io:443:192.168.56.200 https://argo.talos.lab.example.io/.

Avec le défaut SELF_SIGNED=true, un avertissement du navigateur est attendu jusqu'à l'import de _out/self-signed/ca.crt — comme avec LAB_ACME_ISSUER=staging, dont les certificats sont réels mais non reconnus.

🤖CLAUDE.md

Talos Linux on VirtualBox lab, driven by Vagrant. Talos has neither SSH nor a shell: everything is driven with talosctl from the host. User docs: README.md · application layer: https://ops-nc.github.io/k8s-playground/ · symptoms: TROUBLESHOOTING.md · version bumps: talos/UPGRADE.md.

🚫 _k8s/ is a SUBMODULE — never edit it from here#

_k8s/ is not a directory of this repository. It is a pinned checkout of OPS-NC/k8s-playground, the application layer shared with the kubeadm sibling lab: one source, one place to maintain it. Its documentation is published separately at https://ops-nc.github.io/k8s-playground/.

• Read it freely to understand how the layer behaves — it is checked out on disk.
• Never write to it. Editing a file under _k8s/ dirties another repository's working tree and produces a commit that does not belong here. Addon changes are made in k8s-playground, then this repo bumps the pointer.
• Never link to _k8s/…*.md from a Markdown file of this repo. Those pages are not part of this documentation set and make validate-docs fails on the dead link. Point at https://ops-nc.github.io/k8s-playground/, or at the file on GitHub (https://github.com/OPS-NC/k8s-playground/blob/main/<dir>/README.md — the directories sit at the root of that repo, with no _k8s/ prefix).
• Paths on disk (./_k8s/install.sh, _k8s/longhorn/schematic.yaml) stay correct and must not be rewritten: the submodule really does mount there.
• If _k8s/ is empty on the machine you work on, the submodule was never initialised: git submodule update --init --recursive. git pull alone does not update it.
• Nothing Talos-specific was lost when the layer moved out. longhorn/schematic.yaml, longhorn/patch-longhorn.yaml and the talosctl/TALOSCONFIG support in lib/common.sh, longhorn/longhorn-up.sh and local-path-storage/local-path-up.sh all live in k8s-playground now, behind the talos profile (lib/profiles/talos.sh).

🚀 Order of work#

1. vagrant up creates/starts the VMs (Talos boots off the ISO in maintenance mode).
2. ./talos/cluster-up.sh generates the config, applies it, bootstraps etcd, fetches the kubeconfig and waits for health. This is the real path (the <details> in §4 of the README is the manual "to understand what happens" version).
3. ./_k8s/platform-up.sh lays down the base platform, then the addons, opt-in (./_k8s/install.sh <addon>…, or a single _k8s/<addon>/<addon>-up.sh).

The application-layer entry point needs no argument and no LAB_DIR: it finds the lab by itself (the parent directory of _k8s/ carries a Vagrantfile) and detects the distribution from talos/cluster-up.sh — see the pitfall section below. The full sequence from the host:

export TALOSCONFIG="$PWD/_out/talosconfig"
export KUBECONFIG="$PWD/kubeconfig"
./_k8s/platform-up.sh

The reference lab runs CNI=cilium — the repo default, in lab.env.example, in talos/cluster-up.sh and in k8s-playground's platform-up.sh. Talos installs no CNI at bootstrap and platform-up.sh installs Cilium right after; that is what the application layer assumes everywhere (LoadBalancer Services depend on Cilium's L2 announcement). CNI=none produces the exact same machine config but installs nothing at all — it means "I lay down my own CNI", and platform-up.sh then stops on no node Ready.

🚧 Working rules (non-negotiable)#

• "Install X" is still a repo change first. The deliverable is the reproducible path — manifests, *-up.sh, README — never a hand-rolled kubectl apply that leaves no trace in git. Deploying to the lab afterwards is fine and expected: run the *-up.sh you just wrote, which is also how you find out whether it actually works. What is NOT fine is a cluster carrying state no script can rebuild.
• Ask before anything destructive. vagrant destroy, talosctl reset/upgrade, regenerating _out/, deleting a PVC or a namespace holding data: these are one-way on a lab that takes ~15 min to rebuild. Reading is always free (kubectl get, talosctl read, helm show values, helm template) — use it to back up your claims rather than guessing.
• One feature = one merged PR. Branch from main, conventional commit, PR, squash merge (1 commit on main). No big catch-all commit mixing several topics: split by feature, even if that means several PRs back to back.

✅ Validating a change WITHOUT touching a cluster (do this every time)#

make validate      # bash -n on every script + YAML parse + vagrant validate + config gen
make docs          # regenerates docs/index.html from every README (needs uv)

make validate-yaml alone parses every git-tracked *.yaml/*.yml (PyYAML pulled in by uv, so nothing to install). The ci workflow re-runs validate-shell, validate-yaml and validate-vagrant on every PR through the same make targets — never duplicate a check's definition in the workflow. A runner has no VirtualBox, hence make validate-vagrant VAGRANT_VALIDATE_FLAGS=--ignore-provider there.

⚠️ validate-shell and validate-yaml only cover files tracked by this repo. The _k8s/ submodule is tracked as a single pointer, not file by file, so none of its scripts or manifests are checked here — they are validated in k8s-playground's own CI. A green make validate says nothing about the application layer. make validate-submodule checks the pointer, not its content: that .gitmodules uses an https:// URL (an SSH one breaks the clone for everyone without a GitHub key, on a public repo) and that the pinned commit is publicly fetchable (a never-pushed commit makes git clone --recurse-submodules fail for everyone but you). Both failures are invisible from your own working copy.

make validate-talos generates the config in an mktemp -d, then feeds it to talosctl validate --mode metal: neither _out/ nor the cluster is touched. To test a patch against an existing config without applying it: talosctl machineconfig patch <file> --patch <inline|@file> -o /tmp/x.yaml, then validate.

make docs regenerates the bilingual page and lists, at the end of the build, every *.md link and cross-file anchor that does not resolve. make validate-docs (included in make validate) builds into a throwaway directory and fails on the first unresolved link — that is the guard to run after renaming a heading or adding a page.

⚠️ Pitfalls (already hit — do not repeat)#

• Do NOT re-run cluster-up.sh against an already-installed cluster: wait_maintenance polls get disks --insecure, which a node in secure mode never answers. Both waits are bounded since #53 (WAIT_MAINTENANCE, 300 s; WAIT_SECURE, 600 s — both overridable) and fail with a message naming the two likely causes, so this no longer hangs forever — it just wastes the timeout. To grow a running cluster: README §6.1.
• Do NOT regenerate _out/ (nor FORCE=1) on a running cluster: new secrets/CA ⇒ broken cluster. Only regenerate after vagrant destroy.
• Addressing: topology and addressing live in lab.env (single source read by both the Vagrantfile AND talos/cluster-up.sh). Versioned template lab.env.example; lab.env is gitignored. CP = .10/.20/.30, workers = .101+. A real environment variable still wins (WORKERS=6 vagrant up).
• NETWORK is only half configurable: 192.168.56.x is hardcoded in talos/patch-all.yaml (validSubnets), talos/patch-cp.yaml (vip.ip, advertisedSubnets) and talos/cni-flannel.yaml (--iface-can-reach). Changing NETWORK without editing those three files gives you a silently broken cluster.
• Three places carry the Talos version: Vagrantfile (fallback default), talos/cluster-up.sh (fallback default) and lab.env. Both defaults are now aligned on v1.13.7 — keep them that way on every bump, and remember that INSTALLER_IMAGE (factory image, tag included) overrides TALOS_VERSION for what actually lands on disk.
• The Kubernetes version is a FOURTH version axis (KUBERNETES_VERSION in lab.env, empty by default = whatever the talosctl binary ships). cluster-up.sh maps it to talosctl gen config --kubernetes-version, and it is read only at generation time — on a running cluster the tool is talosctl upgrade-k8s. Two traps: (1) talosctl validates the value not at all (it just templates image tags — even 9.99.99 and abc generate a config that passes talosctl validate), so a bad version only shows up as ErrImagePull on the static pods; (2) passing the flag with an empty value is NOT the same as omitting it — empty leaves every image: field commented out (no pin at all), which is why both cluster-up.sh and validate-talos build the flag conditionally. Do not "simplify" that into an unconditional --kubernetes-version "$KUBERNETES_VERSION".
• Never lower CP_MEM below 3072: 2 GB control planes starve etcd as soon as _k8s/ addons stack up. The template now ships 4096 (and so does the Vagrantfile fallback), which observability/ requires. Cost of the default topology: 18 GB of host RAM.
• Renaming VMs: destroy (vagrant destroy) BEFORE changing s[:name] in the Vagrantfile, otherwise the old VMs become orphans in VirtualBox.
• vagrant up fails after a destroy (VERR_ALREADY_EXISTS on the temp_clone_… rename): VirtualBox 7.x leaves orphaned ~/VirtualBox VMs/talos-*/ directories plus dead entries in the media registry. Cleanup: ./talos/virtualbox-cleanup.sh (idempotent, DRY_RUN=1 to preview). NEVER on a running cluster — and note that it also deletes temp_clone_* VMs, including those of another Vagrant project mid-up.
• Disk sentinel: the Vagrantfile considers a VM provisioned if .vagrant/talos-disks/<vm>.vdi exists. A destroy that fails and leaves the .vdi behind makes the next up create a VM with no disk attached, with an obscure install error.
• kube-proxy: KUBE_PROXY_REPLACEMENT=true is the default, and it is read in the same two places as CNI. cluster-up.sh adds talos/patch-no-kube-proxy.yaml (cluster.proxy.disabled: true) to the generated config, so the bootstrap renders no kube-proxy manifest, and k8s-playground's cilium-up.sh installs Cilium with kubeProxyReplacement=true + k8sServiceHost=<VIP>:6443 (mandatory: nothing provisions the apiserver ClusterIP any more). This aligns the lab with the kubeadm sibling, where the equivalent is kubeadm init --skip-phases=addon/kube-proxy. Three traps: (1) it requires CNI=cilium — cluster-up.sh AND make validate-talos refuse any other pair, because without kube-proxy and without a replacement no ClusterIP answers at all, CoreDNS included; (2) like CNI it is read only at generation time, and it is not a live toggle — destroy, then rebuild; (3) it is a control-plane patch: cluster.proxy is not part of a worker machine config. On Talos there is no _out/cluster.env to turn it into a detected fact, so lab.env and the real cluster can silently disagree — the ground truth is kubectl -n kube-system get ds kube-proxy.
• CNI: CNI=cilium|calico|flannel|none (default cilium) expresses an intent, read in two places — cluster-up.sh applies talos/cni-<CNI>.yaml, then ./_k8s/platform-up.sh installs the CNI unless Talos already did. Note the two readers now live in two repositories: changing the default here means changing it in k8s-playground too. Only flannel is laid down by Talos at bootstrap time (cluster.network.cni); cilium and calico go through cni.name: none then Helm. Any manual gen config MUST include --config-patch-control-plane @talos/cni-<CNI>.yaml and --install-image "$INSTALLER_IMAGE" — without it the classic installer is laid down, without the iscsi extensions, and Longhorn fails later on iscsiadm: not found.
• TLS: SELF_SIGNED=true is the default, and it skips cert-manager entirely. k8s-playground's platform-up.sh step [4/4] branches on it: true runs _k8s/self-signed/selfsigned-up.sh (local CA + openssl wildcard into _out/self-signed/, then the TLS Secret) and strips the cert-manager.io/cluster-issuer annotation from main-gateway; false installs cert-manager as before. Both modes fill the SAME Secret (wildcard-<LAB_DOMAIN with dashes>-tls), so no addon ever branches on the TLS mode — keep it that way. LAB_DNS_ZONE, LAB_ACME_EMAIL, LAB_ACME_ISSUER and CLOUDFLARE_API_TOKEN are dead variables when SELF_SIGNED=true. Switching modes on a live cluster leaves the other mode's object behind (a Certificate, or a hand-made Secret) — see the self-signed/ page of k8s-playground, §⚠️.
• ACME: staging is the default, and prod has a weekly quota (SELF_SIGNED=false only). LAB_ACME_ISSUER (staging|prod, default staging) drives the cert-manager.io/cluster-issuer annotation — the versioned Envoy-Proxy.yml carries letsencrypt-staging, and k8s-playground's platform-up.sh rewrites it. Do NOT switch the repo default back to prod: the wildcard lives only in etcd, so every vagrant destroy burns one of the 5 certificates/week per identifier set Let's Encrypt production allows. Already hit on 2026-07-26: 5/5 consumed, 429 rateLimited, no TLS for 18 h — while the destroyed cert was valid for another 3 months. Before a destroy on a prod lab: kubectl -n envoy-gateway-system get secret <wildcard>-tls -o yaml > _out/wildcard-tls.backup.yaml (private key inside — _out/ is gitignored).
• Calico/tigera-operator: two bootstrap traps, both fixed in _k8s/calico/ (in k8s-playground — fix them there, never here) — do not undo them. (1) The chart renders four CRs (Installation, APIServer, Goldmane, Whisker) but ships no crds/ directory — the operator creates the CRDs at runtime (-manage-crds=true), so any CR left enabled kills helm install on a fresh cluster with no matches for kind … ensure CRDs are installed first. All four stay enabled=false; the ones we want live in installation.yaml / apiserver.yaml, applied after the CRD wait. (2) The operator needs hostNetwork + a hostPath, which Talos's default baseline PodSecurity rejects, and helm --create-namespace sets no PSS label ⇒ _k8s/calico/namespace.yaml must be applied before the chart. Failure mode is nasty: get pods shows zero pod (not a failing one), the Deployment just never rolls out — the reason is only in kubectl -n tigera-operator describe rs. After such a failure, relabelling is not enough: the ReplicaSet backoff outlives the 300 s timeout, so rollout restart then re-run.
• Only Cilium gives an IP to LoadBalancer Services in this lab (L2/ARP announcement). Calico can only do it over BGP (no peer router on a host-only network) ⇒ MetalLB required, and loadBalancerClass: io.cilium/l2-announcer in Envoy-Proxy.yml has to go — which is what platform-up.sh does when the CNI is not Cilium. Changing CNI = vagrant destroy, not a live switch.
• Flannel/VXLAN: without --iface-can-reach=192.168.56.1 — which lives in talos/cni-flannel.yaml, not in patch-cp.yaml — flannel picks the NAT interface (10.0.2.15, identical on every VM) ⇒ broken cross-node traffic and DNS. Same for Cilium: pin the enp0s8 host-only interface.
• Vault + integrated Raft: vault-1/vault-2 start NOT initialized. They only join through retry_join once vault-0 is unsealed, so unsealing them immediately after helm install fails with 400 — Vault is not initialized. Wait for initialized=true per pod before unsealing (_k8s/vault-cluster/vault-up.sh does this). Symptom of the race: vault-0 unsealed, the other two sealed, script dead at exit 2.
• jq: // treats false exactly like null. .sealed // true therefore returns true for an unsealed Vault, which made an idempotent re-run try to unseal an open Vault and abort on 400 — already unsealed. On any boolean field, use .field | tostring and compare to "true"/"false" instead.
• lab.env is PARSED, never SOURCED — and the parser must stay identical to kubeadm's. Three rules, each covering a bug this repo actually shipped: (1) while IFS='=' read -r key val || [ -n "$key" ] — without the || [ -n "$key" ], a last line with no trailing newline is silently dropped; (2) the key name is validated against ^[A-Za-z_][A-Za-z0-9_]*$ before any eval; (3) eval ": \${$key:=\$val}" and never :=\"$val\" — quoting the value inside the evaluated string makes LAB_DOMAIN=$(cmd) run cmd. The same applies to the Makefile: . ./lab.env not only executes the file, it inverts the documented precedence (real env var > lab.env > default), so CNI=flannel make validate-talos used to validate the cilium patch and announce it as such — a target that validates something other than what you asked is worse than no target. validate-talos now reads keys with the same sed extraction as lire_lab_env in _k8s/lib/common.sh. If you touch any of these three readers, touch them all.
• ./script.sh; echo "EXIT=$?" reports the exit code of echo, not of the script, so a background wrapper built that way reports success no matter what failed. Check the EXIT= line inside the log, or use ${PIPESTATUS[0]} — a shell that swallows failures is worse than no check at all.
• chaoskube is dry-run by default, and _k8s/chaos-kube/ deletes a pod every hour. Without --no-dry-run the chart only logs would kill … — check dryRun=false in the pod logs, never the manifest. Going back to dry-run requires REMOVING the no-dry-run key: the chart renders --<key> for any falsy value, so --set …no-dry-run=null keeps the flag (hence the mktemp+sed in chaoskube-up.sh). Exclusion list: kube-system, longhorn-system, vault, cnpg-demo — vault is in there because a killed Vault pod comes back SEALED (no auto-unseal), and cnpg-demo is the demo Postgres cluster namespace, not the operator's (cnpg-system, still a target). Excluding a namespace that does not exist yet is harmless.
• Hostname: per-node, therefore outside the shared patches. Set at apply-config time through a HostnameConfig document (auto: "off" + hostname). Vagrant VM name == Talos hostname.
• Dashboard KUBERNETES: n/a: normal in maintenance mode (the KubeletSpec resource only exists after apply-config). Nothing to fix.
• _k8s/longhorn/patch-longhorn.yaml is NOT applied by cluster-up.sh (which only passes patch-all, patch-cp and cni-*): the rshared mount of /var/lib/longhorn is applied by _k8s/longhorn/longhorn-up.sh, to the workers, right before the chart. A freshly bootstrapped cluster therefore has no extraMounts — see the longhorn/ page of k8s-playground. That script, its schematic.yaml and its patch-longhorn.yaml moved into the submodule with the rest of the layer; the talos/UPGRADE.md commands still reference them at _k8s/longhorn/…, which only resolves once the submodule is checked out.
• The default gateway through NAT 10.0.2.2 is intentional (Internet access). What must be host-only is the node's identity (kubelet nodeIP / etcd / VIP), not the default route.
• Bilingual docs: docs/build.py pairs pages per directory through MIRRORS (README.md ↔ LISEZ-MOI.md, UPGRADE.md ↔ MISE-A-JOUR.md). A page with no mirror does not fail the build: it shows up in English inside the French menu, with an EN badge. That badge is the symptom of a forgotten mirror — except for the pages listed in WITHOUT_MIRROR (this file), which are English-only on purpose and carry no badge.
• FR anchors ≠ EN anchors: slugs derive from headings, so translating a heading breaks every link that targeted it. *.md links are rewritten into internal routes at build time; make docs lists whatever no longer resolves. Two contractual anchors now live in k8s-playground (README.md#-lab_domain--the-ui-domain and #-remote-access-tailscale--cloudflare); this repo links them as absolute GitHub URLs, so renaming those headings there breaks the links here — and make validate-docs cannot see it, because it does not follow external URLs.
• No Markdown link may point into _k8s/. docs/build.py --strict resolves *.md links and anchors, the submodule's pages are not part of this documentation set, and make validate-docs fails on them. Use https://ops-nc.github.io/k8s-playground/ or a GitHub URL instead. Disk paths in prose or in code blocks (./_k8s/install.sh) are fine.
• The `` banner at the top of every page is there for GitHub readers; docs/build.py strips it (it has its own switcher). Do not remove it from the files, and do not put anything else between the markers.

The _k8s/ submodule#

• The lab is located automatically — never export LAB_DIR in a doc example. k8s-playground walks up from _k8s/ and takes the parent directory that carries a Vagrantfile as the lab root, so lab.env and _out/ resolve on their own from anywhere. LAB_DIR (like LAB_ENV) survives only as an explicit override for odd setups — mention it as such, never as a step. Doc examples that run the application layer must NOT show export LAB_DIR="$PWD". Do not "helpfully" re-add it.
• TALOSCONFIG is a different matter — it IS required. The addons that drive the Talos API (longhorn/longhorn-up.sh, local-path-storage/local-path-up.sh) need export TALOSCONFIG="$PWD/_out/talosconfig", and everything touching the cluster needs export KUBECONFIG="$PWD/kubeconfig". Both stay in the examples. Do not confuse this rule with the LAB_DIR one above and strip them together.
• The distribution is auto-detected, not an argument. k8s-playground reads this lab as talos from the presence of talos/cluster-up.sh (the sibling lab: kubeadm/cluster-up.sh), so detection works straight after clone, before any vagrant up; secondary signal, _out/talosconfig → talos. The bare form is the documented invocation: ./_k8s/platform-up.sh, ./_k8s/install.sh longhorn vault argocd, ./_k8s/install.sh list, ./_k8s/longhorn/longhorn-up.sh. An explicit talos argument still wins over everything and --distro= / K8S_DISTRO still work, but they are overrides, not the normal path. There is no DISTRO= key in lab.env any more — do not reintroduce it in any doc.
• Do not edit anything under _k8s/ from this repo, and do not link to its *.md files. See the dedicated section at the top of this file.
• docs/build.py excludes _k8s/ from page discovery and carries an external "☸️ Plateforme" link to https://ops-nc.github.io/k8s-playground/ in the sidebar instead. Do not re-add _k8s menu groups: those pages are built and published by the other repo.

🔐 Secrets#

• lab.env is gitignored and holds real secrets (Cloudflare token, Vault token, unseal keys). Never commit it, never copy its values into a README, a commit, a report or terminal output.
• _out/*.yaml holds the cluster CA and keys; kubeconfig holds the admin credentials.
• _k8s/databasement/ is gitignored on both sides (here, and in k8s-playground's own .gitignore): its values.yaml carries an application key in the clear. Since _k8s/ became a submodule, this repo no longer tracks its contents file by file — the rule that matters now is the one in k8s-playground.
• The repo is public: every versioned default must stay neutral (talos.lab.example.io, empty CLOUDFLARE_API_TOKEN).
• Before committing: git status — no secret file may show up, and _k8s must appear as a submodule pointer at most, never as modified content.

📝 Conventions#

• Bilingual docs, English first: README.md and talos/UPGRADE.md are in English; their French mirror lives in the same directory — LISEZ-MOI.md, talos/MISE-A-JOUR.md. Both versions change in the same commit: an English page whose mirror did not follow is a documentation bug. This file is the exception: it is English-only (it addresses coding agents, and there is no French mirror to keep in sync).
• Commit messages in English, conventional (fix(...), feat(...), docs: ...). Branch from main, then PR (squash).
• Everything that is not a French documentation page is in English. Code comments, identifiers, script output, error messages, Makefile, CI workflows, .gitignore, Vagrantfile, lab.env.example and docs/build.py — all English. The repo used to keep its comments in French; that is no longer the case, so do not "restore" French in a script you touch.
• The only French left is the FR documentation mirrors (LISEZ-MOI.md, DEPANNAGE.md, talos/MISE-A-JOUR.md) — their prose, not the output they quote. Three deliberate exceptions inside otherwise English code:
  • the fr values of LABELS in docs/build.py (they are the French UI);
  • the FR menu titles of GROUPS and of OTHER, same reason;
  • the French markers of the CALLOUTS table ("attention", "jamais", "astuce", "conseil", "remarque"…). These are not labels, they parse the French pages to pick a callout's colour. Translating them silently turns every French callout grey — the kind of breakage no test catches. Note the callout kinds (danger/tip/info) are English because they become CSS classes (.callout-tip).
• When a French page quotes script output, quote the English string the script now prints. A French page documenting an English-output tool is the expected result, not an oversight.

⚠️ Adding a component = propagating it EVERYWHERE#

An addon, a variable or an option is only "done" once it is documented at every level. A single isolated mention is a documentation bug: the reader will never find the component. Run this checklist on every addition:

Then make docs to regenerate the page, and make validate before committing.

• "Test" topology: edit lab.env (gitignored, therefore never committed). The repo default stays in lab.env.example (3 CP / 3 workers) — do not change it "just to test".
• READMEs follow a shared structure (one emoji per ## heading, ⚠️/💡/ℹ️ callouts) and are published as HTML by docs/build.py. Stick to standard markdown (CommonMark + GitHub tables) so the generator renders them correctly.

🤖CLAUDE.md

Talos Linux on VirtualBox lab, driven by Vagrant. Talos has neither SSH nor a shell: everything is driven with talosctl from the host. User docs: README.md · application layer: https://ops-nc.github.io/k8s-playground/ · symptoms: TROUBLESHOOTING.md · version bumps: talos/UPGRADE.md.

🚫 _k8s/ is a SUBMODULE — never edit it from here#

_k8s/ is not a directory of this repository. It is a pinned checkout of OPS-NC/k8s-playground, the application layer shared with the kubeadm sibling lab: one source, one place to maintain it. Its documentation is published separately at https://ops-nc.github.io/k8s-playground/.

• Read it freely to understand how the layer behaves — it is checked out on disk.
• Never write to it. Editing a file under _k8s/ dirties another repository's working tree and produces a commit that does not belong here. Addon changes are made in k8s-playground, then this repo bumps the pointer.
• Never link to _k8s/…*.md from a Markdown file of this repo. Those pages are not part of this documentation set and make validate-docs fails on the dead link. Point at https://ops-nc.github.io/k8s-playground/, or at the file on GitHub (https://github.com/OPS-NC/k8s-playground/blob/main/<dir>/README.md — the directories sit at the root of that repo, with no _k8s/ prefix).
• Paths on disk (./_k8s/install.sh, _k8s/longhorn/schematic.yaml) stay correct and must not be rewritten: the submodule really does mount there.
• If _k8s/ is empty on the machine you work on, the submodule was never initialised: git submodule update --init --recursive. git pull alone does not update it.
• Nothing Talos-specific was lost when the layer moved out. longhorn/schematic.yaml, longhorn/patch-longhorn.yaml and the talosctl/TALOSCONFIG support in lib/common.sh, longhorn/longhorn-up.sh and local-path-storage/local-path-up.sh all live in k8s-playground now, behind the talos profile (lib/profiles/talos.sh).

🚀 Order of work#

1. vagrant up creates/starts the VMs (Talos boots off the ISO in maintenance mode).
2. ./talos/cluster-up.sh generates the config, applies it, bootstraps etcd, fetches the kubeconfig and waits for health. This is the real path (the <details> in §4 of the README is the manual "to understand what happens" version).
3. ./_k8s/platform-up.sh lays down the base platform, then the addons, opt-in (./_k8s/install.sh <addon>…, or a single _k8s/<addon>/<addon>-up.sh).

The application-layer entry point needs no argument and no LAB_DIR: it finds the lab by itself (the parent directory of _k8s/ carries a Vagrantfile) and detects the distribution from talos/cluster-up.sh — see the pitfall section below. The full sequence from the host:

export TALOSCONFIG="$PWD/_out/talosconfig"
export KUBECONFIG="$PWD/kubeconfig"
./_k8s/platform-up.sh

The reference lab runs CNI=cilium — the repo default, in lab.env.example, in talos/cluster-up.sh and in k8s-playground's platform-up.sh. Talos installs no CNI at bootstrap and platform-up.sh installs Cilium right after; that is what the application layer assumes everywhere (LoadBalancer Services depend on Cilium's L2 announcement). CNI=none produces the exact same machine config but installs nothing at all — it means "I lay down my own CNI", and platform-up.sh then stops on no node Ready.

🚧 Working rules (non-negotiable)#

• "Install X" is still a repo change first. The deliverable is the reproducible path — manifests, *-up.sh, README — never a hand-rolled kubectl apply that leaves no trace in git. Deploying to the lab afterwards is fine and expected: run the *-up.sh you just wrote, which is also how you find out whether it actually works. What is NOT fine is a cluster carrying state no script can rebuild.
• Ask before anything destructive. vagrant destroy, talosctl reset/upgrade, regenerating _out/, deleting a PVC or a namespace holding data: these are one-way on a lab that takes ~15 min to rebuild. Reading is always free (kubectl get, talosctl read, helm show values, helm template) — use it to back up your claims rather than guessing.
• One feature = one merged PR. Branch from main, conventional commit, PR, squash merge (1 commit on main). No big catch-all commit mixing several topics: split by feature, even if that means several PRs back to back.

✅ Validating a change WITHOUT touching a cluster (do this every time)#

make validate      # bash -n on every script + YAML parse + vagrant validate + config gen
make docs          # regenerates docs/index.html from every README (needs uv)

make validate-yaml alone parses every git-tracked *.yaml/*.yml (PyYAML pulled in by uv, so nothing to install). The ci workflow re-runs validate-shell, validate-yaml and validate-vagrant on every PR through the same make targets — never duplicate a check's definition in the workflow. A runner has no VirtualBox, hence make validate-vagrant VAGRANT_VALIDATE_FLAGS=--ignore-provider there.

⚠️ validate-shell and validate-yaml only cover files tracked by this repo. The _k8s/ submodule is tracked as a single pointer, not file by file, so none of its scripts or manifests are checked here — they are validated in k8s-playground's own CI. A green make validate says nothing about the application layer. make validate-submodule checks the pointer, not its content: that .gitmodules uses an https:// URL (an SSH one breaks the clone for everyone without a GitHub key, on a public repo) and that the pinned commit is publicly fetchable (a never-pushed commit makes git clone --recurse-submodules fail for everyone but you). Both failures are invisible from your own working copy.

make validate-talos generates the config in an mktemp -d, then feeds it to talosctl validate --mode metal: neither _out/ nor the cluster is touched. To test a patch against an existing config without applying it: talosctl machineconfig patch <file> --patch <inline|@file> -o /tmp/x.yaml, then validate.

make docs regenerates the bilingual page and lists, at the end of the build, every *.md link and cross-file anchor that does not resolve. make validate-docs (included in make validate) builds into a throwaway directory and fails on the first unresolved link — that is the guard to run after renaming a heading or adding a page.

⚠️ Pitfalls (already hit — do not repeat)#

• Do NOT re-run cluster-up.sh against an already-installed cluster: wait_maintenance polls get disks --insecure, which a node in secure mode never answers. Both waits are bounded since #53 (WAIT_MAINTENANCE, 300 s; WAIT_SECURE, 600 s — both overridable) and fail with a message naming the two likely causes, so this no longer hangs forever — it just wastes the timeout. To grow a running cluster: README §6.1.
• Do NOT regenerate _out/ (nor FORCE=1) on a running cluster: new secrets/CA ⇒ broken cluster. Only regenerate after vagrant destroy.
• Addressing: topology and addressing live in lab.env (single source read by both the Vagrantfile AND talos/cluster-up.sh). Versioned template lab.env.example; lab.env is gitignored. CP = .10/.20/.30, workers = .101+. A real environment variable still wins (WORKERS=6 vagrant up).
• NETWORK is only half configurable: 192.168.56.x is hardcoded in talos/patch-all.yaml (validSubnets), talos/patch-cp.yaml (vip.ip, advertisedSubnets) and talos/cni-flannel.yaml (--iface-can-reach). Changing NETWORK without editing those three files gives you a silently broken cluster.
• Three places carry the Talos version: Vagrantfile (fallback default), talos/cluster-up.sh (fallback default) and lab.env. Both defaults are now aligned on v1.13.7 — keep them that way on every bump, and remember that INSTALLER_IMAGE (factory image, tag included) overrides TALOS_VERSION for what actually lands on disk.
• The Kubernetes version is a FOURTH version axis (KUBERNETES_VERSION in lab.env, empty by default = whatever the talosctl binary ships). cluster-up.sh maps it to talosctl gen config --kubernetes-version, and it is read only at generation time — on a running cluster the tool is talosctl upgrade-k8s. Two traps: (1) talosctl validates the value not at all (it just templates image tags — even 9.99.99 and abc generate a config that passes talosctl validate), so a bad version only shows up as ErrImagePull on the static pods; (2) passing the flag with an empty value is NOT the same as omitting it — empty leaves every image: field commented out (no pin at all), which is why both cluster-up.sh and validate-talos build the flag conditionally. Do not "simplify" that into an unconditional --kubernetes-version "$KUBERNETES_VERSION".
• Never lower CP_MEM below 3072: 2 GB control planes starve etcd as soon as _k8s/ addons stack up. The template now ships 4096 (and so does the Vagrantfile fallback), which observability/ requires. Cost of the default topology: 18 GB of host RAM.
• Renaming VMs: destroy (vagrant destroy) BEFORE changing s[:name] in the Vagrantfile, otherwise the old VMs become orphans in VirtualBox.
• vagrant up fails after a destroy (VERR_ALREADY_EXISTS on the temp_clone_… rename): VirtualBox 7.x leaves orphaned ~/VirtualBox VMs/talos-*/ directories plus dead entries in the media registry. Cleanup: ./talos/virtualbox-cleanup.sh (idempotent, DRY_RUN=1 to preview). NEVER on a running cluster — and note that it also deletes temp_clone_* VMs, including those of another Vagrant project mid-up.
• Disk sentinel: the Vagrantfile considers a VM provisioned if .vagrant/talos-disks/<vm>.vdi exists. A destroy that fails and leaves the .vdi behind makes the next up create a VM with no disk attached, with an obscure install error.
• kube-proxy: KUBE_PROXY_REPLACEMENT=true is the default, and it is read in the same two places as CNI. cluster-up.sh adds talos/patch-no-kube-proxy.yaml (cluster.proxy.disabled: true) to the generated config, so the bootstrap renders no kube-proxy manifest, and k8s-playground's cilium-up.sh installs Cilium with kubeProxyReplacement=true + k8sServiceHost=<VIP>:6443 (mandatory: nothing provisions the apiserver ClusterIP any more). This aligns the lab with the kubeadm sibling, where the equivalent is kubeadm init --skip-phases=addon/kube-proxy. Three traps: (1) it requires CNI=cilium — cluster-up.sh AND make validate-talos refuse any other pair, because without kube-proxy and without a replacement no ClusterIP answers at all, CoreDNS included; (2) like CNI it is read only at generation time, and it is not a live toggle — destroy, then rebuild; (3) it is a control-plane patch: cluster.proxy is not part of a worker machine config. On Talos there is no _out/cluster.env to turn it into a detected fact, so lab.env and the real cluster can silently disagree — the ground truth is kubectl -n kube-system get ds kube-proxy.
• CNI: CNI=cilium|calico|flannel|none (default cilium) expresses an intent, read in two places — cluster-up.sh applies talos/cni-<CNI>.yaml, then ./_k8s/platform-up.sh installs the CNI unless Talos already did. Note the two readers now live in two repositories: changing the default here means changing it in k8s-playground too. Only flannel is laid down by Talos at bootstrap time (cluster.network.cni); cilium and calico go through cni.name: none then Helm. Any manual gen config MUST include --config-patch-control-plane @talos/cni-<CNI>.yaml and --install-image "$INSTALLER_IMAGE" — without it the classic installer is laid down, without the iscsi extensions, and Longhorn fails later on iscsiadm: not found.
• TLS: SELF_SIGNED=true is the default, and it skips cert-manager entirely. k8s-playground's platform-up.sh step [4/4] branches on it: true runs _k8s/self-signed/selfsigned-up.sh (local CA + openssl wildcard into _out/self-signed/, then the TLS Secret) and strips the cert-manager.io/cluster-issuer annotation from main-gateway; false installs cert-manager as before. Both modes fill the SAME Secret (wildcard-<LAB_DOMAIN with dashes>-tls), so no addon ever branches on the TLS mode — keep it that way. LAB_DNS_ZONE, LAB_ACME_EMAIL, LAB_ACME_ISSUER and CLOUDFLARE_API_TOKEN are dead variables when SELF_SIGNED=true. Switching modes on a live cluster leaves the other mode's object behind (a Certificate, or a hand-made Secret) — see the self-signed/ page of k8s-playground, §⚠️.
• ACME: staging is the default, and prod has a weekly quota (SELF_SIGNED=false only). LAB_ACME_ISSUER (staging|prod, default staging) drives the cert-manager.io/cluster-issuer annotation — the versioned Envoy-Proxy.yml carries letsencrypt-staging, and k8s-playground's platform-up.sh rewrites it. Do NOT switch the repo default back to prod: the wildcard lives only in etcd, so every vagrant destroy burns one of the 5 certificates/week per identifier set Let's Encrypt production allows. Already hit on 2026-07-26: 5/5 consumed, 429 rateLimited, no TLS for 18 h — while the destroyed cert was valid for another 3 months. Before a destroy on a prod lab: kubectl -n envoy-gateway-system get secret <wildcard>-tls -o yaml > _out/wildcard-tls.backup.yaml (private key inside — _out/ is gitignored).
• Calico/tigera-operator: two bootstrap traps, both fixed in _k8s/calico/ (in k8s-playground — fix them there, never here) — do not undo them. (1) The chart renders four CRs (Installation, APIServer, Goldmane, Whisker) but ships no crds/ directory — the operator creates the CRDs at runtime (-manage-crds=true), so any CR left enabled kills helm install on a fresh cluster with no matches for kind … ensure CRDs are installed first. All four stay enabled=false; the ones we want live in installation.yaml / apiserver.yaml, applied after the CRD wait. (2) The operator needs hostNetwork + a hostPath, which Talos's default baseline PodSecurity rejects, and helm --create-namespace sets no PSS label ⇒ _k8s/calico/namespace.yaml must be applied before the chart. Failure mode is nasty: get pods shows zero pod (not a failing one), the Deployment just never rolls out — the reason is only in kubectl -n tigera-operator describe rs. After such a failure, relabelling is not enough: the ReplicaSet backoff outlives the 300 s timeout, so rollout restart then re-run.
• Only Cilium gives an IP to LoadBalancer Services in this lab (L2/ARP announcement). Calico can only do it over BGP (no peer router on a host-only network) ⇒ MetalLB required, and loadBalancerClass: io.cilium/l2-announcer in Envoy-Proxy.yml has to go — which is what platform-up.sh does when the CNI is not Cilium. Changing CNI = vagrant destroy, not a live switch.
• Flannel/VXLAN: without --iface-can-reach=192.168.56.1 — which lives in talos/cni-flannel.yaml, not in patch-cp.yaml — flannel picks the NAT interface (10.0.2.15, identical on every VM) ⇒ broken cross-node traffic and DNS. Same for Cilium: pin the enp0s8 host-only interface.
• Vault + integrated Raft: vault-1/vault-2 start NOT initialized. They only join through retry_join once vault-0 is unsealed, so unsealing them immediately after helm install fails with 400 — Vault is not initialized. Wait for initialized=true per pod before unsealing (_k8s/vault-cluster/vault-up.sh does this). Symptom of the race: vault-0 unsealed, the other two sealed, script dead at exit 2.
• jq: // treats false exactly like null. .sealed // true therefore returns true for an unsealed Vault, which made an idempotent re-run try to unseal an open Vault and abort on 400 — already unsealed. On any boolean field, use .field | tostring and compare to "true"/"false" instead.
• lab.env is PARSED, never SOURCED — and the parser must stay identical to kubeadm's. Three rules, each covering a bug this repo actually shipped: (1) while IFS='=' read -r key val || [ -n "$key" ] — without the || [ -n "$key" ], a last line with no trailing newline is silently dropped; (2) the key name is validated against ^[A-Za-z_][A-Za-z0-9_]*$ before any eval; (3) eval ": \${$key:=\$val}" and never :=\"$val\" — quoting the value inside the evaluated string makes LAB_DOMAIN=$(cmd) run cmd. The same applies to the Makefile: . ./lab.env not only executes the file, it inverts the documented precedence (real env var > lab.env > default), so CNI=flannel make validate-talos used to validate the cilium patch and announce it as such — a target that validates something other than what you asked is worse than no target. validate-talos now reads keys with the same sed extraction as lire_lab_env in _k8s/lib/common.sh. If you touch any of these three readers, touch them all.
• ./script.sh; echo "EXIT=$?" reports the exit code of echo, not of the script, so a background wrapper built that way reports success no matter what failed. Check the EXIT= line inside the log, or use ${PIPESTATUS[0]} — a shell that swallows failures is worse than no check at all.
• chaoskube is dry-run by default, and _k8s/chaos-kube/ deletes a pod every hour. Without --no-dry-run the chart only logs would kill … — check dryRun=false in the pod logs, never the manifest. Going back to dry-run requires REMOVING the no-dry-run key: the chart renders --<key> for any falsy value, so --set …no-dry-run=null keeps the flag (hence the mktemp+sed in chaoskube-up.sh). Exclusion list: kube-system, longhorn-system, vault, cnpg-demo — vault is in there because a killed Vault pod comes back SEALED (no auto-unseal), and cnpg-demo is the demo Postgres cluster namespace, not the operator's (cnpg-system, still a target). Excluding a namespace that does not exist yet is harmless.
• Hostname: per-node, therefore outside the shared patches. Set at apply-config time through a HostnameConfig document (auto: "off" + hostname). Vagrant VM name == Talos hostname.
• Dashboard KUBERNETES: n/a: normal in maintenance mode (the KubeletSpec resource only exists after apply-config). Nothing to fix.
• _k8s/longhorn/patch-longhorn.yaml is NOT applied by cluster-up.sh (which only passes patch-all, patch-cp and cni-*): the rshared mount of /var/lib/longhorn is applied by _k8s/longhorn/longhorn-up.sh, to the workers, right before the chart. A freshly bootstrapped cluster therefore has no extraMounts — see the longhorn/ page of k8s-playground. That script, its schematic.yaml and its patch-longhorn.yaml moved into the submodule with the rest of the layer; the talos/UPGRADE.md commands still reference them at _k8s/longhorn/…, which only resolves once the submodule is checked out.
• The default gateway through NAT 10.0.2.2 is intentional (Internet access). What must be host-only is the node's identity (kubelet nodeIP / etcd / VIP), not the default route.
• Bilingual docs: docs/build.py pairs pages per directory through MIRRORS (README.md ↔ LISEZ-MOI.md, UPGRADE.md ↔ MISE-A-JOUR.md). A page with no mirror does not fail the build: it shows up in English inside the French menu, with an EN badge. That badge is the symptom of a forgotten mirror — except for the pages listed in WITHOUT_MIRROR (this file), which are English-only on purpose and carry no badge.
• FR anchors ≠ EN anchors: slugs derive from headings, so translating a heading breaks every link that targeted it. *.md links are rewritten into internal routes at build time; make docs lists whatever no longer resolves. Two contractual anchors now live in k8s-playground (README.md#-lab_domain--the-ui-domain and #-remote-access-tailscale--cloudflare); this repo links them as absolute GitHub URLs, so renaming those headings there breaks the links here — and make validate-docs cannot see it, because it does not follow external URLs.
• No Markdown link may point into _k8s/. docs/build.py --strict resolves *.md links and anchors, the submodule's pages are not part of this documentation set, and make validate-docs fails on them. Use https://ops-nc.github.io/k8s-playground/ or a GitHub URL instead. Disk paths in prose or in code blocks (./_k8s/install.sh) are fine.
• The `` banner at the top of every page is there for GitHub readers; docs/build.py strips it (it has its own switcher). Do not remove it from the files, and do not put anything else between the markers.

The _k8s/ submodule#

• The lab is located automatically — never export LAB_DIR in a doc example. k8s-playground walks up from _k8s/ and takes the parent directory that carries a Vagrantfile as the lab root, so lab.env and _out/ resolve on their own from anywhere. LAB_DIR (like LAB_ENV) survives only as an explicit override for odd setups — mention it as such, never as a step. Doc examples that run the application layer must NOT show export LAB_DIR="$PWD". Do not "helpfully" re-add it.
• TALOSCONFIG is a different matter — it IS required. The addons that drive the Talos API (longhorn/longhorn-up.sh, local-path-storage/local-path-up.sh) need export TALOSCONFIG="$PWD/_out/talosconfig", and everything touching the cluster needs export KUBECONFIG="$PWD/kubeconfig". Both stay in the examples. Do not confuse this rule with the LAB_DIR one above and strip them together.
• The distribution is auto-detected, not an argument. k8s-playground reads this lab as talos from the presence of talos/cluster-up.sh (the sibling lab: kubeadm/cluster-up.sh), so detection works straight after clone, before any vagrant up; secondary signal, _out/talosconfig → talos. The bare form is the documented invocation: ./_k8s/platform-up.sh, ./_k8s/install.sh longhorn vault argocd, ./_k8s/install.sh list, ./_k8s/longhorn/longhorn-up.sh. An explicit talos argument still wins over everything and --distro= / K8S_DISTRO still work, but they are overrides, not the normal path. There is no DISTRO= key in lab.env any more — do not reintroduce it in any doc.
• Do not edit anything under _k8s/ from this repo, and do not link to its *.md files. See the dedicated section at the top of this file.
• docs/build.py excludes _k8s/ from page discovery and carries an external "☸️ Plateforme" link to https://ops-nc.github.io/k8s-playground/ in the sidebar instead. Do not re-add _k8s menu groups: those pages are built and published by the other repo.

🔐 Secrets#

• lab.env is gitignored and holds real secrets (Cloudflare token, Vault token, unseal keys). Never commit it, never copy its values into a README, a commit, a report or terminal output.
• _out/*.yaml holds the cluster CA and keys; kubeconfig holds the admin credentials.
• _k8s/databasement/ is gitignored on both sides (here, and in k8s-playground's own .gitignore): its values.yaml carries an application key in the clear. Since _k8s/ became a submodule, this repo no longer tracks its contents file by file — the rule that matters now is the one in k8s-playground.
• The repo is public: every versioned default must stay neutral (talos.lab.example.io, empty CLOUDFLARE_API_TOKEN).
• Before committing: git status — no secret file may show up, and _k8s must appear as a submodule pointer at most, never as modified content.

📝 Conventions#

• Bilingual docs, English first: README.md and talos/UPGRADE.md are in English; their French mirror lives in the same directory — LISEZ-MOI.md, talos/MISE-A-JOUR.md. Both versions change in the same commit: an English page whose mirror did not follow is a documentation bug. This file is the exception: it is English-only (it addresses coding agents, and there is no French mirror to keep in sync).
• Commit messages in English, conventional (fix(...), feat(...), docs: ...). Branch from main, then PR (squash).
• Everything that is not a French documentation page is in English. Code comments, identifiers, script output, error messages, Makefile, CI workflows, .gitignore, Vagrantfile, lab.env.example and docs/build.py — all English. The repo used to keep its comments in French; that is no longer the case, so do not "restore" French in a script you touch.
• The only French left is the FR documentation mirrors (LISEZ-MOI.md, DEPANNAGE.md, talos/MISE-A-JOUR.md) — their prose, not the output they quote. Three deliberate exceptions inside otherwise English code:
  • the fr values of LABELS in docs/build.py (they are the French UI);
  • the FR menu titles of GROUPS and of OTHER, same reason;
  • the French markers of the CALLOUTS table ("attention", "jamais", "astuce", "conseil", "remarque"…). These are not labels, they parse the French pages to pick a callout's colour. Translating them silently turns every French callout grey — the kind of breakage no test catches. Note the callout kinds (danger/tip/info) are English because they become CSS classes (.callout-tip).
• When a French page quotes script output, quote the English string the script now prints. A French page documenting an English-output tool is the expected result, not an oversight.

⚠️ Adding a component = propagating it EVERYWHERE#

An addon, a variable or an option is only "done" once it is documented at every level. A single isolated mention is a documentation bug: the reader will never find the component. Run this checklist on every addition:

Then make docs to regenerate the page, and make validate before committing.

• "Test" topology: edit lab.env (gitignored, therefore never committed). The repo default stays in lab.env.example (3 CP / 3 workers) — do not change it "just to test".
• READMEs follow a shared structure (one emoji per ## heading, ⚠️/💡/ℹ️ callouts) and are published as HTML by docs/build.py. Stick to standard markdown (CommonMark + GitHub tables) so the generator renders them correctly.