IaC agent VM (srviac) #

The dedicated VM that runs Terraform and Ansible against the homelab in production. After Phase 1 (iac-agent), routine TF + Ansible flows through srviac; the operator workstation is reserved for break-glass and for mutating srviac itself.

See /work/AnsibleSpecs/phases/completed/iac-agent.md for the design rationale.

What lives where #

Where	What
srviac host	Docker, the iac shim, a daily docker image prune -f cron, a systemd unit running the Jenkins inbound-agent container, /etc/iac/secrets.yaml (operator-curated, 0600), /var/lib/iac/ansible-ssh-key/ (root, 0700, mounted writable into every iac container: srviac's own copy of the Ansible key, id_ed25519_ansible, for a run while OpenBao is down — the go-live seeds it and SecretRotator rewrites it at each rotation, secret-rotator-go-live.md), and the dhcp-probe timer (role dhcp_probe), which checks every 3 minutes that the production DHCP service answers and leaves the result for Prometheus.
registry:5000/iac image	Terraform, Ansible, kubectl, helm, python, poetry, terraform-backend-git (the estate's patched build, copied from DockerImages' terraform-backend-git image), SecretRotator's secret-rotator, plus iac-impl — the in-container entrypoint that parses secrets.yaml, clones the repos secrets.yaml names (Ansible alone by default), starts the terraform-backend-git daemon on 127.0.0.1:6061, then exec's whatever you asked for. The Python venv is baked in at image build from this repo's pyproject.toml/poetry.lock; iac-impl installs nothing at runtime and instead warns when the cloned poetry.lock differs from the baked one. SecretRotator is installed from its prd branch as a uv tool, with a venv of its own. Built from support/iac-image/Dockerfile by this repo's IaC/IaC Docker Image job.
pvginkel/Ansible (this repo)	Roles, playbooks, inventory, the Terraform configs (terraform/{prd,scratch}/, each with a backend.tf http block), the Jenkins pipeline scripts (Jenkinsfile.*) every job checks out, the iac image's build context (support/iac-image/), and the srviac host glue (support/iac-agent/ — bin/iac.j2, install.sh, the systemd unit, the secrets.example.yaml template).
pvginkel/TerraformState	tfstate served through the terraform-backend-git http backend, sops+age-encrypted at rest. Private. Holds the same sensitivity as any secret-bearing repo (VM host private keys, API tokens, proxmox creds). Not srviac's alone: the Argo CD Terraform PreSync hook (/work/ArgoCDTools) writes into the same repo under argocd/<repo>/[<path>/]<stage>/terraform.tfstate (<path>/ for an app in a directory of a deploy monorepo), starting its own terraform-backend-git in the hook pod, and its destroy mode, which IaC/Destroy Stage runs, removes a retired stage's argocd/<repo>/[<path>/]<stage>/ with a commit of its own. Both sides decrypt with the one age keypair at kv/iac/tf-backend: the prd eso AppRole is granted read on that single leaf rather than a copy being made, so the two cannot drift onto different keys.
Jenkins controller (jenkins.webathome.org)	Seven jobs on the iac-controller-labelled agent: IaC/Build-Main, IaC/Apply, IaC/Scheduled Update, IaC/Scheduled Drift, IaC/Scheduled Calico Rollout, IaC/Scheduled Certs, IaC/Scheduled Secret Rotation. IaC/IaC Docker Image and AaC/Ansible also build from this repo, but on Kubernetes pod agents — they use none of the host glue.

Operator workflow #

Routine: push to main, then apply #

Pushing and applying are two acts. A merge to main on pvginkel/Ansible triggers IaC/Build-Main, which is read-only. It first runs the gates kc project lint runs — terraform fmt -check, yamllint, and ansible-lint in strict mode, whose syntax-check covers every playbook — and terraform validate on terraform/prd and terraform/scratch. Then it plan-checks terraform/prd and fails fast if the plan deletes or replaces any prd VM. Terraform refuses such a plan itself (prevent_destroy in the managed-vm module); check-protected-vms.sh fails it for a config without that line. Last, when the push touched docs/runbooks/, it starts the Publish workflow of pvginkel/SOPs, whose site mirrors the runbooks, with the GitHub token in kv/jenkins/sops-publish. Nothing converges. A red build means the commit fails a lint or validate gate, would not apply cleanly, or could not start that workflow; the estate is untouched either way.

Convergence is IaC/Apply, started by hand once the validation is green. The job, inside one iac -c '…' per stage:

1. Plans terraform/prd, runs the destroy check against that plan, and applies that saved plan — one stage, one call. The guard has to run against the commit being applied, not a different build, and a second call would plan again in a fresh clone, so the plan applied is the plan checked.
2. Runs site.yml --limit '!iac_agent'.
3. Runs site-openbao.yml.
4. Runs site-k8s.yml --limit k8s_prd.
5. Converges srvk8sdev last, in a stage that can only ever downgrade the build to UNSTABLE. srvk8sdev is off by default, so IaC/Scheduled Update is UNSTABLE most weeks on that stage alone. The operator reads that as green and wants it left as it is (2026-10-04): no card, no change. Any other stage UNSTABLE or failed is still a finding.

All ansible stages pass --skip-tags os_update: patch posture belongs to IaC/Scheduled Update.

On failure, jenkins-telegram-bot reports the build — job name, build link, and whatever the job put in the build description. It is subscribed to every build, so no iac-* pipeline carries a failure handler of its own. The bot deliberately stays quiet about UNSTABLE and ABORTED, so each job raises those itself through JenkinsPipelineUtils' notify var. The dev stage warns when srvk8sdev was up and the run genuinely failed. Every iac-* file ends with a post { aborted } block that raises <job> #<n> aborted (timeout or hand): a build is aborted by hand or by its timeout, four hours on the srviac jobs and 60 minutes on IaC/IaC Docker Image. An apply cut off that way is left half converged.

The split exists so that pushing a commit is not the same act as applying it to production — an unattended agent pushing a branch must not be able to roll the prd fleet. The cost is that prd no longer converges on its own after a push: if you push and do not start IaC/Apply, the change sits unapplied until a scheduled job or a later apply picks it up.

Routine: the iac image rebuild #

A push still starts IaC/IaC Docker Image, but the job decides for itself whether to build. It rebuilds only when the push's changeset touched something the image is built from — support/iac-image/, the root pyproject.toml or poetry.lock whose venv is baked in, ansible/roles/baseline/files/homelab-root.crt, ansible/files/known_hosts.d/homelab, or Jenkinsfile.iac-image itself, which carries the Dockerfile path, the context and both tags. Any other push reports the Build iac image stage as skipped for conditional and pushes no tag; registry:5000/iac:latest stays where it was.

The gate reads the build's own changeset (utils.hasChanges), so a build that fails on an image-input push is not retried by the next unrelated push: the input no longer appears in that build's changeset. Restart the failed build with image=iac, or push again.

The job takes an optional image parameter, as DockerImages does: all, or a comma-separated list of image names. A build whose list contains iac builds whatever its changeset holds. The build stamps image=iac in the image's org.webathome.poller.params label, and the version poller passes it back as the build parameter when it starts the weekly rebuild. So that rebuild still builds when its changeset carries unrelated commits.

So: to force a rebuild, start the job with Build with Parameters and image=iac. A plain replay, or a build with no image-input change, skips.

IaC/SecretRotator starts it the same way. A push to SecretRotator's main that passes its lint and tests resets SecretRotator's prd branch to that commit, then starts IaC/IaC Docker Image with image=iac. Every build installs SecretRotator from prd's tip, so every rebuild, the weekly one included, carries the rotator's last commit whose lint and tests were green, and a SecretRotator build red at those changes nothing on srviac. Its last stage, the dashboard publish, runs after the reset and the image build's start, so a build red there has already moved prd.

Routine: manual run from srviac #

SSH in and use iac. Two forms:

ssh srviac
iac                           # interactive bash inside the container
iac -c 'cd /work/Ansible/ansible && ansible-playbook playbooks/site.yml --limit srvxxx --check'

Neither takes a host lock. Terraform still interlocks with a running Jenkins job, through the backend's per-state lock branches, but hand-run Ansible on srviac no longer interlocks with a running job — check for a running IaC/* build before converging by hand.

Inside the container: /work/Ansible is a fresh clone, terraform state flows through the local terraform-backend-git daemon (the backend.tf http block in each config — no symlinks, no /work/TerraformState checkout in the container), every env entry from secrets.yaml is exported, and every file entry has been written at its declared mode. Edits inside an iac shell are lost on exit unless committed and pushed before exiting — same constraint Jenkins jobs run under.

Break-glass / srviac mutation: from wrkdev #

The orchestrator cannot orchestrate its own replacement. Anything that mutates srviac itself runs from wrkdev:

• Initial creation: cd terraform/prd && terraform apply then cd ansible && poetry run ansible-playbook playbooks/site.yml --limit srviac.
• Subsequent agent VM changes (disk resize, role refresh, image bump): same.
• True break-glass (CI down, controller unreachable): wrkdev can still run terraform apply and ansible-playbook directly. Terraform there still takes the backend's lock branches, but nothing interlocks Ansible between the workstation and srviac, so don't mix routine work between the two.

First-time cutover (one-off) #

This is the sequence to stand srviac up the first time, after all the source code is in place. Each step is run by the operator from wrkdev unless stated otherwise.

1. Create the VM via Terraform.

   cd terraform/prd && terraform apply
   

2. Apply Ansible to srviac — bootstrap, baseline (including node_exporter + unattended-upgrades), the iac_agent and dhcp_probe roles.

   cd ansible && poetry run ansible-playbook playbooks/site.yml --limit srviac
   

   The role will fail loudly at the secrets step with "you need to populate secrets" — that's expected on a fresh host.

3. Populate /etc/iac/secrets.yaml on srviac.

   ssh srviac
   sudo cp /etc/iac/secrets.example.yaml /etc/iac/secrets.yaml
   sudo chmod 0600 /etc/iac/secrets.yaml
   sudo $EDITOR /etc/iac/secrets.yaml
   

   Fill in every REPLACE_ME value. The id_ed25519_ansible entry is a !bao ref to kv/iac/ansible-ssh-key#private, resolved at each iac start, so it needs no fill; with OpenBao down, iac-cold-boot.md gives its literal. The JENKINS_AGENT_SECRET comes from the controller — register the agent ("IaC Agent", label iac-controller, remote root /work) on https://jenkins.webathome.org/ first.

   See proxmox-credentials.md for the TF_VAR_proxmox_* values.

4. Re-apply the role to verify it converges cleanly.

   poetry run ansible-playbook playbooks/site.yml --limit srviac
   

   The agent container should reach the controller; systemctl status jenkins-agent on srviac shows it running.

5. Bootstrap TerraformState from the workstation's current tfstate. State now lives behind the terraform-backend-git http backend and is sops+age-encrypted at rest; the first apply through the backend writes it there. The one-time seed from the workstation's plaintext files:

   cp terraform/prd/terraform.tfstate /work/TerraformState/prd/
   cp terraform/scratch/terraform.tfstate /work/TerraformState/scratch/
   cd /work/TerraformState && git add prd scratch && git commit -m 'bootstrap from wrkdev' && git push
   

   These seed files are plaintext; the backend re-encrypts each on its next write.

6. Smoke-test iac on srviac.

   ssh srviac
   iac -c 'cd /work/Ansible/terraform/prd && terraform plan'           # should be no-op
   iac -c 'cd /work/Ansible/ansible && ansible-playbook playbooks/site.yml --limit "!iac_agent" --check'
   

   Both clean → green light.

7. Wire the iac-controller jobs listed under "What lives where" on the controller — each is a pipeline job with SCM pvginkel/Ansible and Script Path Jenkinsfile.iac-<name> in the repo root. Verify each runs against a no-op change (a comment-only push) before unleashing. IaC/Scheduled Secret Rotation is the exception: its first build runs the rotator, so it is created last, by secret-rotator-go-live.md step 7.

8. Cutover. Stop running Terraform and Ansible from wrkdev as the routine path. Delete the workstation-local terraform/{prd,scratch}/terraform.tfstate{,.backup,.<timestamp>.backup} files — state is reached only through the backend (encrypted in TerraformState) from now on.

Recovery #

srviac is unreachable #

• If the host is up but iac won't run, run iac -v -c true to see where iac-impl's setup stops, and check docker ps.
• If the systemd Jenkins agent is failing, journalctl -u jenkins-agent -n 100. Most failures are stale JENKINS_AGENT_SECRET (controller regenerated it) or controller unreachable.

Rebuild srviac from scratch #

From wrkdev, not IaC/Apply, which runs on srviac. Terraform refuses to replace a prd VM, so destroy srviac on Proxmox first (vm_id 920 on pve, from terraform/prd/vms.tf); the apply then finds it gone and recreates it:

ssh root@pve 'qm shutdown 920 ; sleep 5 ; qm destroy 920'
cd terraform/prd && terraform apply
cd ../../ansible && poetry run ansible-playbook playbooks/site.yml --limit srviac
# then re-populate /etc/iac/secrets.yaml as in step 3 of cutover

Cloud-init re-bakes; the role re-applies; the operator re-populates secrets. srviac's own copy of the Ansible key goes with the VM's disk. Once SecretRotator's ssh-key kind is in its nightly run's kinds_enabled, seed it again as the go-live does; until then, or until the next rotation writes it, iac-cold-boot.md has no Ansible key to take while OpenBao is down. The VM's MAC is pinned in Terraform so the dnsmasq reservation keeps the same IP.

Lost wrkdev (extreme case) #

Bootstrap any Ubuntu box: install Poetry + the standard SSH keys from the cloud-synced attachments (once SecretRotator's ssh-key kind is enabled, the Ansible key is not among them: take it from OpenBao, as operator-workstation.md gives), clone pvginkel/Ansible — which carries the host glue at support/iac-agent/, so that one clone is the whole controller side. For break-glass terraform, run scripts/tf-backend.sh (the same backend in a local docker run --network host) and have the age private key from OpenBao (kv/iac/tf-backend#age_secret_key) so the backend can decrypt state — wrkdev doesn't clone TerraformState for normal use. The script runs the patched build the iac image copies, registry:5000/terraform-backend-git:<build>. With the registry down it starts the copy cached on the workstation, which an earlier run while the registry was up leaves there; with neither, it starts the stock upstream release behind a warning on stderr, and that one can fail with non-fast-forward update beside another writer of TerraformState (AnsibleSpecs decisions.md, "Concurrency control"). Its last line names the image it started. From there wrkdev's workflows resume. The orchestrator-self-applicable guarantee stops here — there is no zero-touch recovery for the case where both the workstation and srviac are lost simultaneously.

Secret rotation #

SecretRotator never rotates JENKINS_AGENT_SECRET, GIT_API_TOKEN or the state keypair's private half: they are external keys, the markers rotator/bootstrap/jenkins-agent-secret and rotator/bootstrap/git-api-token and iac/tf-backend#age_secret_key. Each falls due 365 days after its last stamp. Rotate it by its section below, then press Done on its box in secret-rotator ui (external-key-due.md).

OPENBAO_SECRET_ID (the iac-agent AppRole) and SecretRotator's entries #

SecretRotator rotates the iac-agent secret_id every 90 days, with the operator (openbao.md §5). The nightly run never starts that rotation: once approle is among its enabled kinds, it announces the rotation in Telegram as it falls due. Run it on srviac in secret-rotator ui, or with secret-rotator run rotator/approle/iac-agent. It shows the new secret_id; paste it over the OPENBAO_SECRET_ID literal in /etc/iac/secrets.yaml. No restart: each iac call starts a fresh container that reads the file. The three SECRET_ROTATOR_* entries are !bao refs to leaves the rotator rewrites itself, so they need no edit.

JENKINS_AGENT_SECRET #

Regenerate on the controller, paste into /etc/iac/secrets.yaml, systemctl restart jenkins-agent.

GIT_API_TOKEN (GitHub PAT for TerraformState) #

Mint a new PAT with the same scopes as the current one, update /etc/iac/secrets.yaml. Now consumed by the terraform-backend-git daemon (as its GITHUB_TOKEN) to pull/push TerraformState. No restart needed; iac-impl reads the file at every invocation and starts the daemon fresh per run.

TF_VAR_proxmox_password #

SecretRotator's pve-root-password kind rotates it from secret-rotator ui, on the PVE nodes and in OpenBao (proxmox-credentials.md § Rotation). /etc/iac/secrets.yaml holds it as !bao kv/iac/proxmox#password, so it needs no edit: each iac call takes the current value. A workstation's terraform.tfvars is updated by hand, from Roboform.

State encryption keypair (SOPS/age) #

TF_BACKEND_HTTP_SOPS_AGE_RECIPIENTS (the public half, a literal in /etc/iac/secrets.yaml) and SOPS_AGE_KEY (the private half, !bao kv/iac/tf-backend#age_secret_key) are one keypair. D32 makes that an estate invariant rather than a detail: iac and the Argo CD PreSync hook both write pvginkel/TerraformState, so a second keypair would leave state one side cannot decrypt.

Where the tools are. sops (pinned from its GitHub release) and age/age-keygen (from the archive) are in the iac image, support/iac-image/Dockerfile — so on srviac via iac -c, in the IaC/* pipelines, and on wrkdev via docker run registry:5000/iac. No separate install anywhere. Note the image carries no bao CLI — iac-impl resolves OpenBao through its Python client — so anything needing bao and age-keygen in one pipeline runs from the KubeCoder iac toolchain (cexec iac) instead, which has both. terraform-backend-git bundles its own sops+age and uses those; these CLIs exist for the by-hand paths below.

Reading the recipient is the usual need — every new consumer takes it as a plaintext literal, and it is a public key, not a secret:

ssh srviac 'sudo grep -A1 TF_BACKEND_HTTP_SOPS_AGE_RECIPIENTS /etc/iac/secrets.yaml'

Prefer that over deriving it. age-keygen -y recovers the same string from the private half, but it means a credential read to obtain something already sitting in plaintext. Tooling is no longer the obstacle; the credential handling is.

Match-check — prove the two halves are one pair. Needs bao and age-keygen together, so run it from the KubeCoder toolchain:

cexec iac sh -c 'bao kv get -field=age_secret_key -mount=kv iac/tf-backend | age-keygen -y'

Pipe it; never bao kv get first and paste, which puts the private key in scrollback and history. Compare the age1… against the literal above. A mismatch is D32's failure mode — state written by one side is undecryptable by the other.

Decrypting a state file by hand is TerraformState/README.md §"Decrypting state by hand". It needs sops, not age: state is a sops envelope encrypted to an age recipient, and age -d cannot parse it. On srviac the key should already be present — secrets.yaml carries SOPS_AGE_KEY as a !bao ref and iac-impl resolves it into every run's environment — so iac -c 'sops -d --input-type json --output-type json prd/terraform.tfstate' ought to need no by-hand credential read at all. Verify that on first use; it has not been exercised.

Rotation is not a paste. Every tfstate already in TerraformState is encrypted to the current recipient, so swapping both halves at once orphans all existing state. The shape to use instead is a transition: encrypt to old and new (the variable is RECIPIENTS, plural — sops takes a comma-separated age list, and SOPS_AGE_KEY takes multiple identities), let every state get rewritten by a normal apply, then drop the old. Dry-run it against one throwaway state before touching prd — this path has not been exercised either.

Ansible SSH key (id_ed25519_ansible) #

Until SecretRotator's ssh-key kind is in its nightly run's kinds_enabled, the key is rotated by hand, by the bootstrap role's SSH key rotation, which writes the new private half to OpenBao and pushes the new public key with the site.yml apply. Once the kind is enabled, it rotates the key every 14 days with no operator step, srviac's own copy of the key included (secret-rotator-go-live.md). Either way secrets.yaml needs no edit: it holds the key as !bao kv/iac/ansible-ssh-key#private, which each iac call resolves.