EmergencyE4

KubeCoder lost cluster access

When: Every environment's kubectl is refused at once, on both contexts, and so is the controller's: after a cluster rebuild, or after the kubecoder ServiceAccounts or their token Secrets in kube-system were deleted.

The procedure is KubeCoder's Re-minting KubeCoder's cluster identities. This card gets you there with the right things on the desk.

Every environment's ~/.kube rests on two ServiceAccounts per cluster, kubecoder-ro and kubecoder-rw in kube-system, and their non-expiring token Secrets: minted by hand, reconciled by nothing. Three kubeconfigs are built from them (config merges both clusters, config-dev-write, config-prd-write), held in two OpenBao leaves (eso/prd/kubecoder/prd/catalog and eso/prd/kubecoder/dev/catalog), extracted by ESO, and mounted into each env pod once, at its start.

Identify #

You see	Go to
The prd cluster was rebuilt empty	Workloads on a rebuilt cluster first; this procedure is its "Afterwards"
srvk8sdev was replaced	Rebuild srvk8sdev first, then the dev half of the procedure. The base `config` merges both halves, so every environment is affected, not only `--context dev`
The ServiceAccounts, bindings or token Secrets were deleted on a running cluster	The procedure from step 1, for that cluster only
Only `kubectl --context dev` fails; prd is fine	srvk8sdev is off by default (`onboot: 0`). Not this card
`argocd` in environments is refused while kubectl works	Argo's `kubecoder` account token: argocd.md

Before you start #

You cannot use KubeCoder to fix KubeCoderevery kubectl runs as sudo microk8s kubectl over SSH on a node, ansible@srvk8s1 for prd and ansible@srvk8sdev for dev, from wrkdev. The SSH option pile is in Ansible live-infra-access.md.
Both clusters upneither half is optional. srvk8sdev is VM 919 on pve and off by default: start it first.
A KubeCoder checkoutthe read role is docs/operations/kubecoder-ro-read.yaml, streamed into the node's kubectl apply -f -.
The apiserver addressesthe one value no cluster read yields. They are in the doc's table; never lift them from the node's own admin config.
OpenBao adminbao logged in with the openbao-admin AppRole (openbao.md § Admin access) to kv patch both catalog leaves. patch, never put: each leaf holds the stage's other catalog keys too.
A way to restart every running environmentKubeCoder's own restart (the Telegram bot's restart control, or the MCP restart tool). Never kubectl delete pod: nothing recreates an environment's pod, so that leaves the fleet stopped with the right files.

Then #

Work the doc top to bottom: ServiceAccounts, bindings (one recipe per cluster, they are not mirror images: prd's kubecoder-rw is cluster-admin, dev's is edit), token Secrets of the non-expiring kind, the three files (a rebuild changes the CA, so it is never a token-only swap), both catalog leaves, then the env pods. Finish with its "Verifying the finished state".

Not rehearsed

The doc says so itself: re-minting identities on a working cluster to prove a document is not worth the risk. Read the whole page before the first command.

End of procedure