QRHHomelab Quick Reference Handbook
Ansible runbookmirror

Read-only copy of pvginkel/Ansible docs/runbooks/cold-boot.md as of 80f07c4 (2026-10-17). The source is authoritative: change it there.

Estate cold boot #

When to use this runbook: the whole estate restarted at once. A power cut takes all three PVE hosts down together, as on 2026-09-25 (report and follow-up: AnsibleSpecs handovers/dhcp-outage-2026-09-25/). It gives the order things come back in, a check per layer, and the break-glass moves for when a layer doesn't come back. When OpenBao is the layer that's down and srviac has to run without it, the escape hatch is iac-cold-boot.md. A cluster rebuilt empty, with no objects left to restart, is cluster-bootstrap.md.

What happens on its own #

Every VM has onboot: 1 and no startup order, so all of them start within a minute of their PVE host. The layers then wait on each other through retries. On 2026-09-25 the PVE hosts were up at 15:03 UTC and pods were starting at 15:11. Everything recovered unaided except two things:

  • Keycloak. Its pinned image digest had been garbage-collected from the registry. dhcpapp-app crash-looped on OIDC discovery and held the dhcp pod not-Ready, so there was no DHCP for 2h45m. DHCPApp has been out of the dhcp pod since (DnsmasqDeploy ca30bbe). Deploy repos pin images from registry:5000 to a per-build tag, never a digest, so garbage collection cannot delete a pinned image while its tag stands. registry-cleanup deletes only when the registry app's registryCleanup.dryRun (HomelabAppsDeploy registry/) is false; in dry-run it logs what it would delete.
  • srvk8s4. It takes its address from the in-cluster DHCP and stayed off the LAN until DHCP was back. It still does; srviac, which had the same problem, is static now (static-address.md). Expect srvk8s4 NotReady until DHCP answers.

So the job is mostly to watch, top to bottom, and step in at the first layer that doesn't come up.

Order and checks #

The kubectl lines work from KubeCoder (cexec iac kubectl --kubeconfig ~/.kube/config-prd-write …) once srvk8s4 is back. Before that, run them on a control-plane node: ssh ansible@srvk8s1 sudo microk8s kubectl …. With DNS down, use SSH by IP (break-glass below).

  1. PVE: pve 10.1.0.20, pve1 .21, pve2 .22. pvecm status is quorate, and qm list on each host shows its guests running.
  2. Ceph: srvceph1–3 at 10.1.0.24–26, VIP ceph.home 10.1.0.38. The prd Ceph nodes reject the ansible key, so go through the guest agent: ssh root@pve1 "qm guest exec 113 --timeout 60 -- microceph.ceph -s" should show HEALTH_OK and every OSD up.
  3. OpenBao: srvvault1–3 at 10.1.0.40–42, VIP secrets.home 10.1.0.39. The seal is static auto-unseal, so the nodes unseal themselves. curl -sk https://10.1.0.39:8200/v1/sys/health should show "sealed":false.
  4. Kubernetes: srvk8s1–3 at 10.1.0.27–29; srvk8s4 is 10.1.3.5 from a DHCP reservation, so it joins only after step 6. The apiserver VIP is 10.1.0.37:16443. kubectl get nodes should show srvk8s1–3 Ready. Check that the Ceph CSI nodeplugins are Running (-n ceph-csi-rbd-prd, -n ceph-csi-cephfs-prd): without them no PV mounts. Check that Kyverno's admission controller is Ready, one replica per node on srvk8s1–3 (kubectl -n kyverno-prd get pods -l app.kubernetes.io/component=admission-controller): until a replica answers, every new pod its policy covers is refused (kyverno.md), dnsmasq-prd's and registry-prd's included. Its images come from Kyverno's own registries, not registry:5000.
  5. Registry: registry-prd, LB 10.2.1.9, ClusterIP 172.17.0.3. The nodes pin that ClusterIP in /etc/hosts, so image pulls don't wait on dnsmasq. Its storage is the static PV registry-pv. curl -s http://10.2.1.9:5000/v2/_catalog answers.
  6. dnsmasq, then DNS and DHCP: dnsmasq-prd: dns-0/dns-1 on 10.2.1.2/.3, dhcp on 10.2.1.10. The UDM relays DHCP from Intranet, IoT and Guest to 10.2.1.10; it serves no DHCP of its own.
    • The pods are Ready, and every EndpointSlice endpoint shows ready: true: kubectl -n dnsmasq-prd get endpointslices -o yaml | grep -B3 'ready:'.
    • MetalLB announces all three: kubectl get servicel2statuses.metallb.io -A | grep -E 'dns-0|dns-1|dhcp'. No status means no ready endpoint, so the address is dark on the LAN.
    • dig +short router.home @10.2.1.2 returns 10.1.0.1.
    • Leases are being handed out: kubectl -n dnsmasq-prd logs deploy/dhcp -c dhcp-dnsmasq --since=10m | grep -c DHCPACK is above zero.
  7. Postgres, then Keycloak: postgres-pas-prd, then keycloak-prd, which reaches its database through postgres-pooler-rw. curl -s -o /dev/null -w '%{http_code}\n' https://auth.ginbov.nl/realms/homelab/.well-known/openid-configuration returns 200. Until it does, apps that discover OIDC at startup crash-loop by design, and every SSO login fails: use the admin logins under break-glass. Keycloak is one replica and needs, besides the database: the CNPG cluster postgres (three instances, one per node on srvk8s1–3, each on that node's local ZFS pool) and its pooler; its image from the registry, unless the node still has it cached; the CephFS PV keycloak-themes-prd-pv; and nginx for auth.ginbov.nl. If it is stuck, check them in that order: kubectl -n postgres-pas-prd get cluster,pods, then kubectl -n keycloak-prd describe pod -l app=keycloak for the pull or mount error.
  8. Apps: kubectl get pods -A | grep -vE 'Running|Completed' is short and shrinking, and kubectl -n argocd-prd get applications shows everything Synced/Healthy.

Break-glass #

The front door is down while the cluster is up. nginx-prd/nginx (LB 10.2.1.7) fronts every webathome.org and .home host, Jenkins included, and Jenkins builds on GitHub's webhook alone. So while nginx is down, pushing a fix ships nothing: the webhook has nowhere to land and is not retried. Restore out of band first, then push the durable fix, then start the build by hand for every push whose webhook was lost. The 2026-08-19 case: a KubeCoder environment published a hostname over nginx's server_names_hash_bucket_size, and nginx-configurator's reload is a rollout restart, so the running master was gone with it (the bucket size is 128 in HomelabAppsDeploy's nginx/ since). The cheapest restore there was deleting the offending front Service with config-prd-write — not the environment: reconcile_front_services only runs when the env recomposes, so the deletion sticks and the env pod keeps running.

DHCP down because the dhcp pod is not Ready. Publish the not-ready endpoint, and MetalLB announces 10.2.1.10 again:

kubectl -n dnsmasq-prd patch svc dhcp -p '{"spec":{"publishNotReadyAddresses":true}}'

The chart pins the field to false, so Argo shows the patch as drift. With selfHeal off it stays until the next DnsmasqDeploy sync, which reverts it. Don't push DnsmasqDeploy while DHCP depends on the patch. Once the pod is Ready, sync or patch it back.

New pods refused by Kyverno's webhook. Rollouts and Jobs create nothing, and their FailedCreate events name mpol.validate.kyverno.svc-fail. Scale Kyverno to zero and delete its webhook registrations as kyverno.md says; Kyverno registers them again once it is scaled back up.

A pinned image missing from the registry (Keycloak's digest on 2026-09-25): the pod sits in ImagePullBackOff with NotFound. Repin the deploy repo to the image's newest per-build tag, which registry-cleanup always keeps, never to a digest. Where the image's build writes the pin, a rebuild does the same.

Admin logins without Keycloak. Argo CD and Jenkins each keep a local admin beside SSO, and the operator keeps both passwords in RoboForm. Try each once at a quiet moment, so the first use is not mid-incident.

  • Argo CD: https://argocd.home, user admin, in the username/password form beneath the SSO button. If RoboForm's copy fails, the chart's own copy is in argocd-initial-admin-secret, readable with config-prd-write, which is certificate-based and does not go through Keycloak. A rebuilt Argo mints a new one. The command and the reset fallback are in argocd.md. kubectl with config-prd-write does everything Argo does, as a last resort.
  • Jenkins: https://jenkins.webathome.org/login. The OIDC plugin's escape hatch is the username/password form on that page, which posts to securityRealm/escapeHatch; log in as admin. Jenkins keeps the account in its own configuration on its volume, not in a deploy repo.
  • The others: Grafana's local admin is the leaf eso/prd/grafana/prd/admin, and its login form is not disabled. pgAdmin allows internal logins beside Keycloak; its default account is the one in HomelabAppsDeploy's pgadmin/chart/values.yaml. Headlamp never used Keycloak: it takes a service-account token, printed by the command in PlatformAddOnsDeploy's headlamp/config/prd/values.yaml. Guacamole lists OIDC first but keeps its database accounts; whether one of them still logs in is unconfirmed.

One push that syncs two stages of the same deploy repo no longer clashes on the hook name. Since homelab-shared 0.3.1 each app's Job is tf-presync-<app>-<stage>. A deploy repo still pinning an older version names its Jobs tf-presync-<rev>-presync-<ts>. There, delete the other stage's finished Job in argocd-hooks, and Argo's retry goes through.

The operator desktop without DHCP. Set a static address on "Ethernet 2". The LAN is a /16. The 2026-09-25 attempt used 255.0.0.0, which treats all of 10/8 as on-link, so the desktop ARPs for addresses it should route through 10.1.0.1. In an elevated prompt:

netsh interface ipv4 set address "Ethernet 2" static 10.1.0.253 255.255.0.0 10.1.0.1
netsh interface ipv4 set dnsservers "Ethernet 2" static 8.8.8.8 primary

Undo afterwards with netsh interface ipv4 set address "Ethernet 2" dhcp and netsh interface ipv4 set dnsservers "Ethernet 2" dhcp. .home names don't resolve on public DNS, so reach hosts by IP.

SSH by IP. Host certificates carry names, not addresses, so give ssh the name to check the certificate against. From ansible/:

ssh -o UserKnownHostsFile=files/known_hosts.d/homelab -o GlobalKnownHostsFile=/dev/null \
  -o HostKeyAlias=srvk8s1.home -i ~/.ssh/id_ed25519_ansible ansible@10.1.0.27

If the pod or the usual workstation is gone too, where the ansible private key is depends on whether SecretRotator's ssh-key kind is in its nightly run's kinds_enabled. Until it is, it is in RoboForm under "Homelab SSH key > Private key for the ansible user"; save it as ~/.ssh/id_ed25519_ansible (chmod 600). Once it is, RoboForm holds none: the key is in OpenBao, kv/iac/ansible-ssh-key#private, and in srviac's own copy, /var/lib/iac/ansible-ssh-key/id_ed25519_ansible (root's). Without either, log in as pvginkel with your own key, the same way, and sudo with its password. PVE hosts take root with id_ed25519_pve.

The UDM is not estate-managed and has no homelab host certificate, so this recipe does not verify it. The operator reaches it with the credentials in RoboForm, and lends them to an agent only when needed.

Address list: router 10.1.0.1; pve/pve1/pve2 .20–.22; srvceph1–3 .24–.26; srvk8s1–3 .27–.29; kubernetes-api .37; ceph .38; secrets .39; srvvault1–3 .40–.42; srviac .45; srvk8s4 10.1.3.5 (a reservation, so only while DHCP is up). The static-hosts list in DnsmasqDeploy chart/templates/stage-manifests.yaml is authoritative.