Idempotent, Self-Verifying Operations
The principle in one line: an operation should be safe to run again and should prove it worked — never “ran without error, so it must be fine.”
What it means (plain English)
Two different promises, both essential.
Idempotent means running it twice does no harm — like pressing a “save” button that just makes the file match what’s on screen. Press it once or five times, same result. The opposite is a button that appends, so every extra press corrupts things. Idempotent operations are safe to retry, safe to run “just in case,” safe to schedule.
Self-verifying means the operation checks its own work before declaring success. A backup script that uploads a file and then re-reads it to confirm it’s actually there and non-empty. A deploy that curls the site afterward and only says “done” if it got a real response. The opposite trusts that “no error was thrown” equals “it worked” — and those are very different things.
Why it matters
Silent success is the most expensive kind of failure. A backup that runs nightly for a year and quietly uploaded zero bytes is worse than no backup, because you believed you were covered. An “apply” that returns exit 0 but didn’t actually change the running system sends you debugging the wrong layer for hours (see verify-the-artifact-under-test). And a non-idempotent operation makes recovery scary: you can’t just re-run it, so every incident becomes a delicate one-shot.
Where it showed up in QinCloud
-
x3-verified-edge-deploys — the apply that proves itself.
deploy-edge.shis the poster child. It validates the config before touching anything, checksums the container’s file against the on-disk one and self-heals a stale mount, reloads, restores every route, and then curls each app host and exits non-zero if any didn’t come back (deploy-edge.sh). Run it twice with no change: same result. It cannot lie about whether it worked — the exact failure mode (verify-the-artifact-under-test) it was built to end. -
m2-data-and-backups — a verified backup.
backup.shdoesn’t trust the upload. After copying each dump to R2 it re-lists the object and requires a positive byte count, dying loudly otherwise — “an empty offsite backup is a silent disaster.” It publishes a success metric only on full completion, so m8-disaster-recovery’sBackupStalealert fires if the pipeline ever quietly stops. And the restore was rehearsed (restore-drill.sh), because the only proof a backup works is putting it back. -
m0-host-baseline — rerunnable from zero.
bootstrap.shis written to be run again cleanly: it checks for each rule/package before adding it, so a second run is a no-op, not a pile of duplicate firewall rules. That idempotence is why the whole box can be rebuilt from scratch (m8-disaster-recovery) without fear.
How to apply it
Write every operation so that (a) running it twice is safe — check-then-act, “make it so” not “do it again”; and (b) it ends by observing the state it claims to have produced, not by observing that no exception occurred. The end of the script should be an assertion: re-read the file, curl the endpoint, query the row, and fail loud (fail-loud-at-boundaries) if reality doesn’t match intent. If it’s expensive to verify, verify anyway — the cost of a silent lie is always higher.
Signs you’re violating it
- “It exited 0, so it worked” is the only evidence you have.
- You’re nervous to re-run something because a second run might double-apply.
- A scheduled job that has “succeeded” for months but you’ve never confirmed its output.
- A backup you’ve never restored; a config apply you’ve never curled.
Related: verify-the-artifact-under-test · fail-loud-at-boundaries · the-box-is-disposable