IaC agent cold-boot runbook #

When to use this runbook: the OpenBao cluster is unreachable, or the homelab is recovering from a whole-cluster loss, and the IaC agent still needs to run (to bring OpenBao itself back, or to bring up the rest of the fleet that OpenBao secrets gate).

The IaC agent's iac-impl resolves !bao references in /etc/iac/secrets.yaml against OpenBao at container startup. When OpenBao is down, every !bao ref hard-fails — by design — and iac won't run. This runbook is the escape hatch: temporarily substitute literals for every !bao ref so iac runs without an OpenBao round trip, do the recovery work, then flip back.

Design context: /work/AnsibleSpecs/slices/completed/iac-secrets-resolver.md, "Runtime secrets — IaC agent resolver" in /work/AnsibleSpecs/decisions.md, and §Secrets resolver in /work/AnsibleSpecs/phases/completed/openbao.md.

Conventions #

• All steps run on srviac (or any host with a working iac install and Docker), as root or via sudo. wrkdev works as a fallback if srviac itself is down — install.sh runs on either host. srviac's own copy of the Ansible key (see "What gets substituted") is on srviac's disk, out of reach while srviac is down.
• "Roboform" means the operator's password manager of record. Every literal you'll need is already in there, one entry per secret, except SecretRotator's and, once SecretRotator's ssh-key kind is enabled, the Ansible key's (see "What gets substituted").
• The OpenBao admin path through the Jenkins agent VM is unavailable during cold boot — that's the whole reason you're here. Don't invent a half-restore that depends on it.

What gets substituted #

Today, /etc/iac/secrets.yaml has two kinds of entries:

Kind	Example	Cold-boot action
Literal	value: ghp_xxxxx	Leave as-is.
!bao ref	value: !bao kv/iac/ha-token#token	Replace with the Roboform-held literal.

The irreducible-literal set (OPENBAO_URL, OPENBAO_ROLE_ID, OPENBAO_SECRET_ID, GIT_API_TOKEN) is already literal; cold boot doesn't touch it. Everything else with !bao becomes a literal for the duration of the recovery.

SecretRotator's three entries (SECRET_ROTATOR_ROLE_ID, SECRET_ROTATOR_SECRET_ID, SECRET_ROTATOR_K8S_TOKEN) are the exception: they have no Roboform entry, since the rotator rewrites its own secret_id in kv/iac/rotator-approle and its own ServiceAccount token in kv/iac/rotator-k8s-token. Comment them out instead. Only secret-rotator reads them, and it cannot run while OpenBao is down; step 5's restore brings them back.

The Ansible key, the files: entry /root/.ssh/id_ed25519_ansible (!bao kv/iac/ansible-ssh-key#private), depends on whether SecretRotator's ssh-key kind is in its nightly run's kinds_enabled. Until it is, its literal is in Roboform like the others. Once it is, the kind replaces the key every 14 days and Roboform holds none: the literal is srviac's own copy of the key, /var/lib/iac/ansible-ssh-key/id_ed25519_ansible (root's, 0600). The rotation switches that copy with the key, and nothing writes it from the leaf, so it holds the key the hosts take. Step 3 splices it in. Without it, a person's way in to the hosts is the pvginkel account.

Procedure #

1 — Snapshot the current file #

sudo cp /etc/iac/secrets.yaml /etc/iac/secrets.yaml.pre-cold-boot

The snapshot is the artifact you'll restore from. Don't skip it.

2 — Inventory the !bao refs you need to substitute #

sudo grep -nE '^\s*[^#].*!bao ' /etc/iac/secrets.yaml

Every match is a ref that needs a Roboform value, or, for the Ansible key once ssh-key is enabled, srviac's copy. The match line shows the mount/path#key triple; Roboform entries are named after the consumer (the Proxmox password, for one, is the PVE root account entry) rather than the KV path, so use the surrounding YAML context (the name: key on env: entries, the path: on files: entries) to match up.

3 — Substitute literals in place #

sudo nano /etc/iac/secrets.yaml (or your editor of choice). For each !bao ref:

• env entry: replace value: !bao kv/... with value: "<literal>".
• files entry: replace content: !bao kv/... with a folded block. Mode lines stay as-is.
• SecretRotator's three SECRET_ROTATOR_* entries: comment them out (see "What gets substituted").
• The Ansible key's entry, once ssh-key is enabled: leave it as it is, for the splice below.

Example before:

- name: TF_VAR_proxmox_password
  value: !bao kv/iac/proxmox#password

- path: /root/.ssh/id_ed25519_ansible
  content: !bao kv/iac/ansible-ssh-key#private
  mode: "0600"

Example after:

- name: TF_VAR_proxmox_password
  value: "<copied from Roboform>"

- path: /root/.ssh/id_ed25519_ansible
  content: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    <copied from Roboform>
    -----END OPENSSH PRIVATE KEY-----
  mode: "0600"

Save the file. Once ssh-key is enabled, splice srviac's copy in as the Ansible key's literal, without showing it:

sudo python3 - <<'EOF'
from pathlib import Path
f = Path("/etc/iac/secrets.yaml")
key = Path("/var/lib/iac/ansible-ssh-key/id_ed25519_ansible").read_text()
lines = f.read_text().splitlines(keepends=True)
(i,) = [n for n, line in enumerate(lines) if line.strip() == "content: !bao kv/iac/ansible-ssh-key#private"]
pad = lines[i][: len(lines[i]) - len(lines[i].lstrip())]
lines[i] = pad + "content: |\n" + "".join(f"{pad}  {line}\n" for line in key.splitlines())
f.write_text("".join(lines))
EOF

A ValueError means the file holds that !bao line not exactly once: edit the entry by hand.

Confirm permissions stayed at 0600 root:root:

sudo stat -c '%a %U:%G' /etc/iac/secrets.yaml

4 — Run iac against the cold-boot config #

sudo iac -c 'echo cold-boot smoke test passed'

If iac-impl complains about a still-!bao-shaped value, you missed a ref — go back to step 3. If it complains about anything else, fix it.

From here, drive recovery via iac -c '...' as normal. Examples:

• Recover OpenBao — follow openbao.md recovery sections (single- node loss or whole-cluster loss). Both run via iac and need ansible-vault content; the vault passphrase is its own irreducible literal and is independent of this runbook.

• Bring up other consumers — Helm-deployed workloads that read from OpenBao via ESO will fail until ESO has a working SecretStore. The fix is to bring OpenBao back, not to expand this runbook.

5 — Flip refs back after OpenBao is restored #

Once bao status reports healthy on the leader and the iac-agent AppRole is usable, restore the snapshot:

sudo mv /etc/iac/secrets.yaml.pre-cold-boot /etc/iac/secrets.yaml
sudo stat -c '%a %U:%G' /etc/iac/secrets.yaml   # expect: 600 root:root

Smoke-test:

sudo iac -c 'env | grep -E "^(TF_VAR_proxmox_password|JENKINS_AGENT_SECRET)" | wc -l'

A value of 2 (or however many !bao-resolved env vars you have) means the resolver is back in service.

After a restore from a snapshot (openbao.md §3), once SecretRotator's ssh-key kind is enabled: iac now starts with the key the restored iac/ansible-ssh-key holds. Where the key was rotated after the snapshot, the hosts no longer take that one. Re-running the ssh-key plan cannot repair it, since the rotator logs in with the key the leaf holds. srviac's copy holds the key the hosts take: write it back to the leaf and its catalog copy, by openbao.md §3 step 8, before any iac run that logs in as ansible.

6 — Rotate any literal that left Roboform #

A literal that sat on disk during cold boot has had its blast radius expanded for the duration. If the cold boot lasted long enough to matter, rotate the affected secret(s) — generate a new value, write it under the same KV path, and let the next iac invocation pick it up. This is a judgment call; for a 30-minute recovery, skip; for a multi-day outage, do it.

What can go wrong #

• You misspell a literal, and the consumer logs an auth error that's easy to confuse for OpenBao downtime. The fix is to verify the literal in Roboform and re-edit. Don't chase OpenBao.
• You forget step 5, and the rest of the fleet runs with stale literals after OpenBao is back. Drift cycle won't catch this — the resolver only reads secrets.yaml, it doesn't audit whether refs used to be !bao. Set a reminder before you start step 3.
• You commit /etc/iac/secrets.yaml by mistake. The file is not in a git repo, but cold-boot procedures sometimes invite cp into the wrong directory. If literals leak into git history, rotate.

Pre-flight checklist #

Before the next time you might need this runbook:

• Every !bao ref in /etc/iac/secrets.yaml has a Roboform entry with a clear name. Drift between the two is what makes cold boot slow. Once SecretRotator's ssh-key kind is enabled, the Ansible key's has none: srviac's copy, /var/lib/iac/ansible-ssh-key/id_ed25519_ansible, exists instead.
• The operator's age private key (for OpenBao backup decrypt) is in Roboform — separate from this runbook, but the same trip to Roboform.
• The ansible-vault passphrase is in Roboform under the "operator workstation" entry — needed for the static seal key during OpenBao recovery.