AbnormalA4

Terraform state lock stuck

When: A Terraform run fails with "Error acquiring the state lock" and nothing is running that could hold it.

Every Terraform here reaches its state through terraform-backend-git: the daemon iac-impl starts on srviac, the KubeCoder sidecar, wrkdev's scripts/tf-backend.sh, and the Argo CD PreSync hook. The backend locks a state by pushing a branch locks/<state path> to pvginkel/TerraformState, holding <state path>.lock with Terraform's lock metadata. Unlocking deletes the branch. A run killed without a clean shutdown leaves it: an aborted Jenkins build, a destroy-stage Job hit by its 30-minute deadline, a dropped cexec.

Identify #

State	Who writes it	Lock branch
`prd/terraform.tfstate` (`terraform/prd`)	`IaC/Build-Main` and `IaC/Scheduled Drift` (plan), `IaC/Apply`, a `cexec iac terraform` in a KubeCoder environment, break-glass from wrkdev	`locks/prd/terraform.tfstate`
`scratch/terraform.tfstate` (`terraform/scratch`)	the same runners	`locks/scratch/terraform.tfstate`
`argocd/<Repo>/[<path>/]<stage>/terraform.tfstate`	the app's hook Job `tf-presync-<app>-<stage>` in `argocd-hooks`; `IaC/Destroy Stage` as `destroy-stage-<build#>` there, with `APP_PATH` set to `<path>`	`locks/argocd/<Repo>/[<path>/]<stage>/terraform.tfstate`

<Repo> is the deploy repo as GitHub spells it, FieldnotesDeploy and not fieldnotesdeploy. <path>/ is there only for an app in a directory of a monorepo; <path> is that directory, the registry entry's path:. An app <app> of PlatformAddOnsDeploy locks locks/argocd/PlatformAddOnsDeploy/<app>/<stage>/terraform.tfstate.

Steps #

Read the lock Terraform printed: who took it, when, and for which operation.
Is it still alive? A running buildhttps://jenkins.webathome.org/job/IaC/
A hook or destroy Jobkubectl -n argocd-hooks get jobs,pods
Another shella cexec iac terraform in some environment, or tf-backend on wrkdev
Alive: wait for it, or queue behind it. Nothing below.

List the lock branches; the stuck state's should be the only one:

git ls-remote --heads https://github.com/pvginkel/TerraformState 'locks/*'

Delete that branch. That is the force-unlock:
```
git push origin --delete locks/<state path>
```
from a clone of TerraformState, or the branch's delete button on GitHub. Only the locks/ branch; main holds the encrypted states.
Re-run what failed. A Destroy Stage build: again with the same parameters, each step skips what is done (argocd § A build that stopped halfway).

Only when nothing holds it

Deleting a live run's lock puts two writers on one state. Steps 2 to 4 come first, every time.

What a clean repo looks like

On 2026-10-06, idle, TerraformState had only main. A locks/* branch while Jenkins and Argo are quiet is stale. terraform force-unlock has not been exercised against this backend; the branch delete is the documented path (AnsibleSpecs decisions.md, "Concurrency control").

Not this

IaC/* builds waiting on each other is the IaC Agent node's single executor, which serialises the IaC jobs on purpose. A queue there is not a state lock.

End of procedure