QRHHomelab Quick Reference Handbook
QRH

Homelab
Quick Reference Handbook

Read it before you need it. Follow it when you do.

The homelab's standard operating procedures: what gets done by hand, on a trigger, and is too easy to get wrong from memory. It is published on GitHub Pages, so it stays up when the homelab doesn't. Each card is a checklist: tick the steps as you go, and the page remembers them for a day.

Memory items

  1. The homelab is down and you are reading this: GitHub is up. Start at the Emergency tab.
  2. Roboform holds what no machine does: the ansible-vault passphrase, the OpenBao Shamir recovery keys (3 of 5) and the age key that decrypts the backups.
  3. Every runbook's source is public: git clone https://github.com/pvginkel/Ansible.
  4. Keep the offline copy (bottom of every page) on a laptop, for the day GitHub is down too.

Emergency

  • E1Estate down, or a power cutAnsible
    All three PVE hosts restarted together, or everything is unreachable at once. The first sign is often healthchecks.io on Telegram: Prometheus's heartbeat stopped.
  • E2No DHCP or DNS in the houseAnsible
    Telegram shows DHCPNotAnswering or LoadBalancerNotAnnounced for dnsmasq-prd; devices can't get a lease; .home names don't resolve; or healthchecks.io says Prometheus's heartbeat stopped.
  • E3Front-door nginx downAnsible
    Every .home, webathome.org and ginbov.nl host is unreachable at once while the cluster is up; or LoadBalancerNotAnnounced names nginx-prd/nginx.
  • E4KubeCoder lost cluster accessKubeCoder
    Every environment's kubectl is refused at once, on both contexts, and so is the controller's: after a cluster rebuild, or after the kubecoder ServiceAccounts or their token Secrets in kube-system were deleted.
  • E5Kyverno refuses new podsAnsible
    New pods on prd are not created: rollouts stall, Jobs start nothing, and the FailedCreate events name the webhook mpol.validate.kyverno.svc-fail.
  • E6OpenBao is down, or a secret is needed without itAnsible
    An srvvault node or the whole OpenBao cluster is lost, or a secret has to be read while there is no cluster to read it from.

Abnormal

Normal

Periodic

  • P1Annual rotation of the bootstrap-tier secretsAnsible
    Once a year, when SecretRotator announces a bootstrap marker: Telegram warns 28, 21 and 14 days before it falls due and every night from 13 days, and secret-rotator ui lists its box. At once on suspicion that a workstation or Roboform is compromised.
  • P2Bump the microk8s channelAnsible
    A new microk8s minor on the stable channel. One minor per round: the playbook refuses a bigger step, so two minors behind is two rounds of this card, with a soak between.
  • P3Install a pushed plugin versionAIWorkflow
    After pushing a version bump of the dev plugin (AIWorkflow) or the kubecoder plugin (KubeCoderConfig) by hand, or when the nightly card pass's notification names a release it pushed. The pass never installs one itself: installing is the operator's call.
  • P4Re-capture the claude stand-in's goldensKubeCoder
    After a Claude Code release: once the Claude Code binary in KubeCoder's local-home image has moved past the one the goldens record.
  • P5Re-run prompt-audit on the dev pluginAIWorkflow
    After a model change: the model behind any dev-plugin role changes. Run it before reading the first slices run on the new model.
  • P6Restore drills, yearlyAnsible
    Once a year, and after a change to a backup path. A round restores every backup somewhere harmless and proves the result is usable.
  • P7Upgrade Argo CDAnsible
    A new argo-cd Helm chart (argoproj/argo-helm) or Argo CD release. On 2026-10-06: chart 10.3.3, Argo CD v3.5.1 on argocd-prd.

Ansible runbooks

Mirrored from pvginkel/Ansible docs/runbooks as of 80f07c4 (2026-10-17).