PeriodicP6
Restore drills, yearly
- When
- Once a year, and after a change to a backup path. A round restores every backup somewhere harmless and proves the result is usable.
The procedures and their Drill logs are the Ansible runbooks, mirrored on this site. This card is the round: every drill once, then a row in the run log below. Nothing announces that the year is up.
Before you start #
- Roboformthe age private key, the s3-mirror crypt password and salt, the Shamir recovery keys (3 of 5), the ansible-vault passphrase
- The Google Drive login, for
gdrive-pieter:Homelab Backups - A host with rclone, age, Docker, kubectl, jq and a browser. The KubeCoder pod does not qualify: it has no rclone or Docker.
- For the S3 mirrorsrvk8sdev (VM 919 on
pve, off by default) can be started - For YouTracka local YouTrack administrator's password
- Each drill the morning after a successful nightly run of its backup, so production has changed as little as possible since
The round #
- OpenBao, whole-cluster lossopenbao.md §3 on the real srvvault VMs, through the re-issue of every AppRole, to its step 9. Record the timings in its Drill log. Last drilled 2026-05-23.
- OpenBao, single-node lossopenbao.md §2: replace one srvvaultN, let the roles converge, watch Raft stream the snapshot from the leader. Never drilled; its timings are the TBD in the Drill log.
- S3 mirrors3-mirror.md §5:
iot-prd-attachmentsfrom Drive into a scratch bucket on srvk8sdev, checked byte for byte against the live bucket. Record it in its Drill log. First drill still owed. - YouTrackyoutrack-restore.md §3: the newest Drive backup into a throwaway container, with the chart's start arguments. Record it in its Drill log. First drill still owed.
- Add a row per drill to the run log below.
Backed up, but nothing to drill
The postgres-pas dumps (90 kept on Drive, nightly pg_dump through backup-server) and
the PVE vzdump of the VMs on pve (local-backup, retain three) have no restore procedure
written. Until one exists they are not on this card, and a complete round does not prove them.
Run log #
| Date | Drill | Result | Log |
|---|---|---|---|
| 2026-05-23 | OpenBao, whole-cluster loss (a single drill, before this card) | Pass: converge 6m32s, restore and verification about 5 min more | openbao.md Drill log |
End of procedure