Rebuilding a Terraform-managed VM #

How to destroy and recreate a managed VM end-to-end. Rebuild is the canonical path for upgrading the OS, recovering from drift that adoption can't reach, and (eventually) for routine cluster upgrades. Per /work/AnsibleSpecs/decisions.md "Adoption is a waypoint; rebuild is the parity event," every adopted VM has a planned rebuild that ends the transition.

What persists across rebuild:

• VMID, name, MAC, SSH host key (Terraform-generated tls_private_key survives in tfstate; cloud-init re-embeds it).
• Disk identities — the LVM-thin volumes (vm-<vmid>-disk-N) survive when a VM is destroyed via the bpg provider only if they're declared with import_from or attached as path_in_datastore; a plain disk { datastore_id = "local-lvm", size = N } is reformatted on recreate. For the production VMs in terraform/prd/, the boot and data disks are reformatted on rebuild; their content lives in Ansible roles + Ceph/ZFS, not in PVE storage volumes that need to survive.
• Inventory (host_vars/<host>.yml) — never touched by rebuild.

What does not persist:

• Anything written to the rootfs by hand outside the role definitions. If it isn't in the role, it dies at rebuild — that's the point.

When to rebuild #

Scenario	Action
Scratch VM after a role change	Rebuild on demand; that's its job.
k8s node	k8s-rebuild.md — evict → qm destroy → terraform apply → rebuild-k8s.yml, one node at a time.
Ceph node	Phase 5 — noout → drain → rebuild → reattach OSDs, serial: 1.
wrkdev	Operator-scheduled.
pve hosts	Never. Bare metal, fidelity-only.

pve hosts have no scheduled rebuild; their drift is reconciled in place against proxmox_host. Anything below assumes a guest VM, not a PVE node.

Rebuild flow — scratch VM (the validated path) #

The simplest case. No cluster impact. Use this exercise to sanity-check role changes before pointing them at a real cluster member. The example below uses wrkscratchk8s1; substitute any host in the scratch fleet.

1. Baseline check before destroying anything #

cd terraform/scratch
terraform plan -detailed-exitcode ; echo "exit=$?"        # expect 0

cd ../../ansible
poetry run ansible-playbook playbooks/site.yml \
    -i inventories/scratch --limit wrkscratchk8s1 --check
# expect: changed=0, failed=0

If the baseline isn't clean, fix what surfaces before rebuilding — otherwise the post-rebuild verification can't tell role drift from rebuild drift.

2. Rebuild via Terraform #

cd ../terraform/scratch
terraform apply -replace='proxmox_virtual_environment_vm.scratch["wrkscratchk8s1"]'

-replace destroys and recreates only that VM resource. The Ubuntu cloud image (proxmox_download_file), the per-VM cloud-init snippet, and the per-VM tls_private_key are independent resources — those for other scratch VMs aren't touched, and the targeted VM's host key is preserved across the rebuild, so the ansible/files/known_hosts.d/scratch entry stays valid.

Terraform blocks until qemu-guest-agent reports an IP back to PVE — typically 1–3 minutes including cloud-init.

3. Apply roles to the fresh VM #

cd ../../ansible
poetry run ansible-playbook playbooks/site.yml \
    -i inventories/scratch --limit wrkscratchk8s1

First run on a fresh VM has changed > 0 — that's the role landing. Watch for failures.

4. Verify zero residual #

poetry run ansible-playbook playbooks/site.yml \
    -i inventories/scratch --limit wrkscratchk8s1 --check
# expect: changed=0

Non-zero changed here means either a baseline imperfection in a role (idempotency bug — fix the role) or an artefact of the rebuild flow itself. Either way, fix before declaring the rebuild successful.

Rebuild flow — k8s and Ceph cluster members #

k8s nodes: use k8s-rebuild.md. Phase 4 rebuilt the k8s fleet from scratch (from_scratch = true in terraform/prd/vms.tf), and that runbook is the concrete, playbook-backed flow: evict-k8s.yml, qm destroy, terraform apply, rebuild-k8s.yml. The outline below does not apply to k8s nodes.

Ceph nodes are still adopted. srvceph1–srvceph3 keep their legacy VMIDs (113–115) and carry no from_scratch, so a rebuild requires switching the VM's entry to the from-scratch shape first — that's a deliberate commit, not a transparent operation. The full procedure lands with Phase 5 (decisions.md "Ceph rebuild path"). Outline so the constraints are visible now:

1. Pre-rebuild drain. ceph osd set noout and stop the OSD/mon. Owned by Ansible; Phase 5 builds the playbook.
2. Configuration update commit. The VM's entry in terraform/prd/vms.tf switches from "model what's there" to the from-scratch shape:
   • Set from_scratch = true. terraform/prd/main.tf then builds the per-VM scaffolding — the Ubuntu cloud image (proxmox_download_file), the cloud-init snippet (proxmox_virtual_environment_file), the host keypair (tls_private_key) — and hands managed-vm the cloud_init wiring that adds the initialization block.
   • Switch from BC:24:11:... MAC to deterministic 02:A7:F3:VV:VV:EE (decisions.md "MAC addressing"). VMID likely also moves into the 900-and-up range; if so, the deterministic MAC moves accordingly.
   • Keep or add passthrough_disks on the VM's TF entry — TF attaches them atomically with the VM, no staged Ansible reattach.
3. dnsmasq reservation update. New MAC → new reservation (or new IP allocation). Must land before terraform apply so the first DHCP lease on the rebuilt VM lands correctly.
4. Destroy the VM on Proxmox, then terraform apply. qm destroy on the PVE node that owns it; Terraform's refresh finds the VM gone and recreates it under its vm_id — the shape of k8s-rebuild.md steps 2 and 4. The scratch flow's -replace does not carry over: Terraform refuses to destroy or replace a prd VM (prevent_destroy in managed-vm), so until the destroy, any plan in which step 2's commit replaces the VM fails with Error: Instance cannot be destroyed. A Ceph node's OSD disks are passthrough /dev/disk/by-id paths, not Proxmox volumes: qm destroy leaves them in place and the new VM reattaches them from passthrough_disks.
5. site.yml — bootstrap + baseline + microceph role lands the cluster bits.
6. Re-join the cluster. noout lifted, OSDs come back, wait for HEALTH_OK.
7. Verify zero residual with --check against the rebuilt host.

Phase 5 will produce concrete playbooks for steps 1, 6. Until then, this outline is forward-looking — don't try to rebuild a Ceph node by hand without the playbook backing.

Removing a prd VM — the first one proves V11 #

Removal follows the same destroy-on-Proxmox-first path (decisions.md "Production execution model"): qm destroy <vm_id> on the owning PVE node, drop the VM's vms.tf entry, push, and read the IaC/Apply plan — the refresh should find the VM gone, plan no destroy for it, and delete only its companions (the DNS reservation, the cloud-init snippet). That removal half has only been reasoned, never run. Slice 017's verification.json still carries V11 as owed (AnsibleSpecs slices/completed/017_iac_pipeline_safety_rails_and_signalling/), and the operator asked to be reminded rather than to schedule a proof. So the first real removal doubles as the proof: hand back the full plan output, and tick V11 off. If the plan instead fails with Error: Instance cannot be destroyed, decisions.md's removal path and terraform/prd/README.md "Rebuilding or removing a VM" are wrong and need fixing before the apply.

Disk passthrough — replacing a failing OSD disk #

When a Ceph OSD's underlying SSD fails and is swapped for a new physical disk, the /dev/disk/by-id/<serial> path changes. Procedure (operator runs as root on the PVE host that owns the VM):

# 1. Identify the new disk's by-id path
ls /dev/disk/by-id/                          # find the new ata-Samsung_..._<newserial>

# 2. Detach the old, attach the new — VM running, hot-swap
qm set <vmid> --delete scsi2                 # detach old passthrough
qm set <vmid> --scsi2 /dev/disk/by-id/<newpath>,backup=0,size=<bytes>K

Capacity in size= matches what qm config <vmid> shows on the surviving Ceph nodes (the Samsung 870 EVO 2TB drives report 1953514584K).

After PVE has the new path:

# 3. Update the VM's entry in terraform/prd/vms.tf
$EDITOR terraform/prd/vms.tf
# change passthrough_disks[*].path_in_datastore to the new path under the VM's key

# 4. Catch tfstate up to PVE
cd terraform/prd
terraform refresh -target='module.vm["<host>"]'
terraform plan -target='module.vm["<host>"]'  # expect zero diff

terraform refresh reads PVE current state into tfstate without modifying anything. After it runs, both state and module declare the new path.

The Ceph side (re-adding the OSD on top of the new disk, balancing) is owned by the microceph role — operator workflow today, role-driven once Phase 5 lands.

If a rebuild goes sideways #

Scratch VM (terraform apply -replace). -replace is destructive once Terraform commits to the destroy phase. If it fails after destroy and before create, the VM is gone but state may be partially written.

• TF errors before destroy: safe — no live impact, fix the error and retry.
• TF errors during create: state may have a tainted resource. terraform plan will surface a -/+ (replace) on the next run. Fix the underlying cause (image download, network, snippet upload) and terraform apply again.
• VM created but cloud-init never finished: Terraform times out waiting for the IP. SSH to the PVE host and qm console <vmid> or check journalctl -u cloud-init on the VM. Usually qemu-guest-agent failing to install — fix the snippet, terraform apply -replace again.

Prd VM (destroyed on Proxmox, then terraform apply). The old VM is gone before Terraform runs; recovery is forward.

• TF errors before the VM is created: fix the error and terraform apply again.
• TF errors during create, or VM created but cloud-init never finished: a VM the failed create left on Proxmox is tainted in state, and Terraform refuses the replace it needs (Error: Instance cannot be destroyed). Every terraform/prd plan fails on it, the daily drift run's included, until it is cleared. Find the cause as for scratch (qm console <vmid>, journalctl -u cloud-init) and fix it, then qm destroy <vmid> on the PVE host and terraform apply again.

site.yml fails on the rebuilt VM (either root): the VM exists, just isn't fully roled. Fix the role and re-run; cloud-init has done its part (ansible user + host key) and bootstrap can re-run idempotently.

For cluster members, leave the node cordoned/drained until site.yml --check reports zero changes. Only then bring it back to the cluster.