Terraform state lock stuck

Every Terraform here reaches its state through terraform-backend-git: the daemon iac-impl starts on srviac, the KubeCoder sidecar, wrkdev's scripts/tf-backend.sh, and the Argo CD PreSync hook. The backend locks a state by pushing a branch locks/<state path> to pvginkel/TerraformState, holding <state path>.lock with Terraform's lock metadata. Unlocking deletes the branch. A run killed without a clean shutdown leaves it: an aborted Jenkins build, a destroy-stage Job hit by its 30-minute deadline, a dropped cexec.

Identify #

<Repo> is the deploy repo as GitHub spells it, FieldnotesDeploy and not fieldnotesdeploy. <path>/ is there only for an app in a directory of a monorepo; <path> is that directory, the registry entry's path:. An app <app> of PlatformAddOnsDeploy locks locks/argocd/PlatformAddOnsDeploy/<app>/<stage>/terraform.tfstate.

Steps #

1. Read the lock Terraform printed: who took it, when, and for which operation.

2. Is it still alive? A running buildhttps://jenkins.webathome.org/job/IaC/

3. A hook or destroy Jobkubectl -n argocd-hooks get jobs,pods

4. Another shella cexec iac terraform in some environment, or tf-backend on wrkdev

5. Alive: wait for it, or queue behind it. Nothing below.

6. List the lock branches; the stuck state's should be the only one:

   git ls-remote --heads https://github.com/pvginkel/TerraformState 'locks/*'
   

7. Delete that branch. That is the force-unlock:

   git push origin --delete locks/<state path>
   

   from a clone of TerraformState, or the branch's delete button on GitHub. Only the locks/ branch; main holds the encrypted states.

8. Re-run what failed. A Destroy Stage build: again with the same parameters, each step skips what is done (argocd § A build that stopped halfway).