Verify

A verification stands on two legs. Remove either one and it is not a verification.

• Observation at the real surface. The claim is exercised where a user meets it, and what the software actually emitted is captured.
• A controlled contrast. A second capture, from the unchanged state, measured identically — so the difference is attributable to the change and not to the weather.

One leg alone has a name. A treatment capture with no control is a demo. A control/treatment pair measured somewhere other than the real surface is a laboratory result about code nobody runs.

What does not count

Three substitutes, each of which feels like evidence and is not:

1. Suites and typecheckers. They re-assert what someone already wrote down. They cannot speak to a claim nobody has encoded yet — which is precisely the claim under verification. Read a test as a specification if it helps; then go exercise the software.
2. Calling the changed function directly. Reaching past the public boundary to invoke an internal and printing the result is a test you authored moments ago, graded by its author. The function behaved as written; reading it told you that. Nothing that ships was exercised.
3. Narrating the diff. A description of what changed is a claim awaiting verification, never its discharge.

1. Fix the claim, and fix what would refute it

Two sentences, written before anything is measured, and not revised afterwards:

Claim. Under <condition>, at <surface>, <observable> <holds>. Refuted if. <the specific observation that would end this as a failure>.

Worked examples:

Claim: invoked with an empty --from, the CLI exits non-zero and names the flag in its message. Refuted if: it exits 0, or the message omits --from.

Claim: the twelfth request inside a minute receives 429 carrying Retry-After. Refuted if: a twelfth request succeeds, or the header is absent or unparseable.

Claim: p95 cold start on this host is at or under 400 ms across 10 trials. Refuted if: p95 exceeds 400 ms, or the trials scatter too widely to place it.

The refutation sentence is the whole point of writing this down. Fixing the failure condition in advance is what stops the bar from drifting to wherever the output happened to land. Quote it verbatim in the report.

Where the claim comes from. Intended behaviour is owned by the request, the issue, or the acceptance criteria — not by the diff. The diff is authoritative only about what changed. Where the two disagree, that disagreement is the finding. If no external statement of intent exists, derive the claim from the diff and label it inferred: you may then verify that the observed change is real, but you may not present it as proof the software does what was wanted.

Unmeasurable claims ("cleaner", "more robust", "better structured") carry no refutation condition and so cannot be verified. Obtain a measurable one, or ask; if you cannot ask — you are running as a subagent — report INCONCLUSIVE naming the claim you could not pin down.

2. Establish scope

Resolve the real base branch first, then measure from the merge base. A branch may be many commits, and the change may not be committed at all:

# The base is the PR's own, else whatever the remote calls its default.
# Test emptiness rather than chaining with ||: a pipeline's exit status is the
# LAST command's, so `base=$(git ... | sed ...)` succeeds even when git failed.
base=$(gh pr view --json baseRefName --jq .baseRefName 2>/dev/null)
[ -n "$base" ] || base=$(git symbolic-ref --short refs/remotes/origin/HEAD 2>/dev/null | sed 's#^origin/##')

# A named base is authoritative: use it, fetch it, or stop. Never substitute a
# trunk for a base you were actually told — that compares a different branch.
ref=""
if [ -n "$base" ]; then
  for c in "origin/$base" "$base"; do
    git rev-parse --verify -q "$c" >/dev/null 2>&1 && { ref="$c"; break; }
  done
  if [ -z "$ref" ] && git fetch -q origin "$base" 2>/dev/null; then
    git rev-parse --verify -q FETCH_HEAD >/dev/null 2>&1 && ref=FETCH_HEAD
  fi
else
  # Only with no base named anywhere is a conventional trunk a fair guess.
  for c in origin/main main origin/master master; do
    git rev-parse --verify -q "$c" >/dev/null 2>&1 && { ref="$c"; break; }
  done
fi

mb=""
[ -n "$ref" ] && mb=$(git merge-base HEAD "$ref" 2>/dev/null)

if [ -z "$mb" ]; then
  echo "STOP: could not resolve base '${base:-unnamed}' (tried '${ref:-nothing}')."
  echo "Ask which branch to compare against; do not guess one."
else
  git rev-list --count "$mb"..HEAD            # how many commits
  git --no-pager diff --stat "$mb"..HEAD      # committed work
  git --no-pager diff --stat HEAD             # staged + unstaged
  git ls-files --others --exclude-standard    # untracked — also part of the change
  git rev-parse "$mb" HEAD                    # the two SHAs, for the report
fi

If it prints STOP, stop. An unresolved base is not a scope of zero commits — it is an unknown scope, and continuing past it reports an empty diff for a change that may be substantial. Repositories built on master, or with no remote and a differently-named trunk, land here legitimately: ask which branch to compare against rather than guessing.

Two traps those commands exist to avoid. @{u} is the wrong base: an upstream is usually the remote copy of this same branch, so diffing against it hides every commit already pushed. git diff HEAD is not the whole change: it omits untracked files, so a newly added file is invisible in it.

Carry both SHAs and whether the tree was dirty into the report — those are what let someone else reproduce the comparison. A large diff that truncates goes to a file you then read. No repository → the scope is whatever the requester named; ask if they did not.

3. Locate the surface

The surface is wherever something outside the change first observes it. Pick the row; the mechanics live in reference/surfaces.md.

The change surfaces at	Observe by
A terminal (CLI, TUI)	issuing the command, capturing the screen
A socket (server, API)	sending the request, capturing the whole response
A window (GUI, web)	driving it headless, then looking at the screenshot
A package boundary	consuming the published export as an outside caller would
A compiler	compiling an external consumer against the shipped declarations
An agent (prompt, role spec)	running the agent, capturing what it did
A CI runner	triggering the real event, then confirming the run's head SHA is yours — a manual dispatch defaults to the default branch and can grade a different version of the workflow

Internals are not surfaces. Every internal has a caller, and following the callers terminates at one of the rows above. The observable behaviour of a commit-blocking hook is not what its function returns — it is whether the commit is refused when you type it.

Nothing observes it → SKIP, one line saying why. Prose, comments, and declarations that emit nothing and that no consumer compiles against qualify. A diff of only tests qualifies too: those are the author's evidence, and re-running them re-runs CI. Where a diff carries both source and tests, verify the source.

Do not stretch SKIP to cover a change you merely found inconvenient to reach. A public type with no runtime emission still has the compiler as its surface, and an external consumer that fails to compile before and succeeds after is a real control/treatment pair.

4. Obtain a handle on the software

Getting the software built and running is /run's subject, not this one. Invoke it. It already knows the per-project-type recipes and will find a project skill that supersedes them.

Before falling back to anything generic, look for what this repository has already committed — at its root and at every directory level the diff touches, because within a multi-package tree whatever makes a given package runnable tends to be stored next to it:

ls .claude/skills/
ls <each-touched-dir>/.claude/skills/

A verifier-* skill matching your surface is the repository's own evidence-capture protocol; prefer it, because a reviewer can replay what it records. Mismatched surface, try the next. Where it misdirects you over plumbing that has nothing to do with what you are checking, it has rotted — say so, and never let its decay become the change's verdict.

Nothing to lean on, and /run cannot get there either? Timebox the cold start to roughly fifteen minutes, then report BLOCKED with the exact point of failure.

If a cold start did succeed, write the recipe down after reporting — .claude/skills/verifier-<surface>/SKILL.md, beside the code it applies to. Commands, worthwhile flows, traps; nothing more. Amend an existing one only where it misled you.

Two rules about that file. Never write during a verification — a new file lands in the very diff being measured, and for a prompt or config change it can alter the behaviour under observation. And never name it verify: a project-scope skill of that name shadows this one, so the next run would load a build recipe stripped of the claim, the control, the probe, and the verdicts — strictly worse than having nothing. The verifier- prefix is also what the search above actually matches.

5. Capture the control

Skipped more often than any other step, and the reason unfounded PASSes ship. How to obtain one, what to record before starting the treatment, and the state that leaks between runs: reference/control.md.

Name the contrast before measuring it. Old-breaks-new-works is one shape among several, and assuming it universally rejects correct work:

The change claims	Its contrast
A defect removed, a capability added	control violates the claim; treatment satisfies it
Behaviour preserved (refactor, swap, migration)	the two agree, inside a margin you state
Failures made rarer	the rate over repeated trials falls — one clean control run refutes nothing
Better performance	named statistic across enough trials to separate the distributions
Depth of defence	the guard is reached under an injected fault that otherwise masks it
Wider compatibility	previously-working environments still work and the newly-targeted one flips

Where the control contradicts the contrast you named, that is information, not a stop sign: confirm you are reaching the changed path at all, and that the measurement could detect the effect if it were there. Then either re-state the contrast or report what you actually found.

Measure both sides the same way — same invocation, same inputs, same environment, same warm-up. A control obtained differently is not a control; it is a second variable wearing one's coat.

Do not let measuring the control disturb what the treatment needs. Quotas, rate-limit counters, one-shot migrations, caches, seeded rows, and first-invocation warm-up all persist across runs — consume the counter while measuring the control and the 429 you then attribute to the fix was your own doing. Re-seed between runs, keep the two builds separate, and say what you reset. The full list, and the countermeasures, are in reference/control.md.

Control genuinely unobtainable? Say so and report INCONCLUSIVE, unless the change is purely additive — then the absence is the control, and it is capturable: the unrecognised flag, the 404, the compile error naming the missing export. Assert an absence and you have asserted, not verified.

6. Drive the treatment

Shortest route that puts the changed lines into execution. Then read your own plan back: if every step builds, typechecks, or runs a suite, you have planned a CI re-run. Replace one of them with something an observer at the surface could witness, or report BLOCKED.

Go through the interface, never around it. Components passing in isolation says nothing about the joins, and joins are where this fails. Where users press a button, press the button.

Stochastic surface — agents, concurrency, anything network-adjacent — is not settled by one run each. Fix inputs and scoring in advance, run the pair repeatedly, and report trials and rates (reference/measurement.md). One fortunate control failure beside one fortunate treatment success is a fully compliant false PASS.

Prove you reached the changed code. A control that fails and a treatment that passes is also consistent with two different environments, or a stale build, or a cache. Keep something that could only come from the new path: a log line it alone emits, a version string, an error only it raises. Without that receipt the pair is circumstantial.

Irreversible operations — deleting, publishing, sending, writing outside the workspace — do not get driven live without a dry-run or a disposable target. Exercise what is safe, then name the path you left alone. A required destructive effect you never exercised is BLOCKED or INCONCLUSIVE, never PASS.

7. Compare

Set the two captures beside each other as artefacts — exit statuses, response bodies, screen dumps, images, trial series. Not as your recollection, which is not evidence.

Numbers carry four figures: control, treatment, difference, and the threshold from §1. Two samples cannot distinguish a real effect from cache warming, thermal drift, or a noisy neighbour — if the difference does not clear the spread, it has not been shown. Declare the statistic and trial count before collecting, alternate which side runs first, and keep cold and warm trials apart: reference/measurement.md.

Name the confounds you can think of — carried-over state, unrelated commits riding along in the treatment, an environment that shifted between captures — rather than waiting to be asked. Where the treatment contains changes beyond the one claimed, either isolate the target or say plainly that the verdict covers the whole branch.

8. Probe the edges

Confirming the claim is the first half. The claim is what the author already believed; your value is what they had not thought to check.

You know exactly what moved, so push on what sits beside it, at the same surface:

• A new flag or option — empty, doubled, contradicted, misspelt (does the error name it?)
• A new route or handler — wrong verb, malformed body, absent required field, oversized payload
• A reworked error path — the neighbouring errors it did not touch: did the rework reach them, or only the one in the diff?
• Anything interactive — interrupt mid-operation, resize, paste rubbish, hammer a key, escape at the wrong moment
• Anything stateful — twice in a row, on top of stale state, from two sessions at once

Choose what the change points at; this is not a list to exhaust. At least one probe, and its result, always. One that finds nothing still earns its line — recording that empty --from produces a clean error: --from requires a value and exit 2 tells the author what is now known to hold, which a bare PASS cannot.

A probe that misbehaves gets re-run against the control before it can drive a FAIL. Identical misbehaviour on both sides is pre-existing, and pre-existing is a finding, not a regression.

Evidence discipline

What was captured is evidence; what you remember is not. Something surprising appears — capture it, record it, and decide whether it belongs to the change or the environment. Breakage you did not cause is still a finding.

Isolate shared state: private tmux socket, port 0, a fresh temporary directory. The host is not yours alone.

Artefacts on disk are optional, one directory per claim, holding the claim, the two captures, and the verdict. Where they would contain credentials, personal data, prompts, or images of private material, keep the minimum inline and leave the rest unwritten unless the requester agrees.

Evidence has to arrive. A path is evidence only to someone who can open it. Screen dumps and response bodies travel inside the report; a bare path works only where the reader shares your filesystem. On a remote surface with a file-sending tool available, send the images and say in the report what was sent.

Report

Inline, in the final message:

## Verification: <what changed, one line>
**Verdict:** PASS | FAIL | INCONCLUSIVE | BLOCKED | SKIP
**Claim:** <verbatim from §1>
**Refuted if:** <verbatim from §1 — unchanged since>
**Scope:** <base SHA>..<head SHA>, <n> commits, <clean|dirty tree>
**Handle:** <verifier or run skill used, or cold start; what was launched>
**Reached the changed code:** <the receipt — log line, version, unique error>

### Control vs treatment
| | Control (<ref>) | Treatment (<ref>) |
|---|---|---|
| <observable> | <artefact> | <artefact> |
<numeric claims add difference + threshold + trial count>

### Steps
Things done to running software, and what came back. Building, installing and
checking out are preparation, not steps.
1. ✅/❌/⚠️/🔍 <what was done> → <what was observed>
   <the software's own output>

### Findings
<What stood out. Bugs, and also friction, surprises, whatever a newcomer would
stumble over. The bar is low: a pause while exercising the software earns a
line. It has to be your pause, from running it — relaying a red check or
someone else's comment is not an observation. Claim/diff disagreements,
pre-existing breakage and environment notes belong here. Every probe gets a
line, including the ones that held. Lead with ⚠️ where the reader should stop
and look.>

🔍 marks a probe. A report with no 🔍 cannot be PASS — it is an incomplete verification, whatever the happy path showed.

Verdicts

Decide on the evidence's standing, not on how confident you feel. In order:

1. Treatment unreachable → BLOCKED
2. Nothing observes the change → SKIP
3. Either capture untrustworthy → INCONCLUSIVE
4. Trustworthy captures meet the named contrast → PASS, else → FAIL

• PASS — software exercised at its surface, trustworthy captures satisfy the §5 contrast, the refutation condition did not occur, and at least one probe is recorded.
• FAIL — exercised, captures trusted, and the contrast is not met: wrong direction, short of threshold, collateral breakage, or intent and diff disagreeing about the very observable claimed.
• INCONCLUSIVE — it ran, but the evidence cannot bear a verdict: no valid control, noise larger than the difference, environments that drifted apart, state carried between runs, a measurement that failed. Name what would settle it — the access, data, trial count, or rerun that decides. Without that sentence this is an abandoned verification wearing a verdict's clothes, and it is the exit an agent takes when it would rather be finished.
• BLOCKED — the treatment could not be reached at all: build broken, dependency absent, nothing would start. Says nothing about the change. A control you cannot obtain is INCONCLUSIVE (§5), not this. Do not claim it before searching the skills beside the touched code, where the unlock usually is. Say where it stopped.
• SKIP — nothing observes the change. One line why.

Nothing partial passes, but partial does not automatically mean FAIL — route it by the same ladder. Where several claims are in scope: any one of them validly falsified makes the whole thing FAIL. Where none was falsified but one was never measured, the aggregate is INCONCLUSIVE, or BLOCKED if that one's surface could not be reached at all — and name which claim is outstanding. Only an author placing a claim outside the scope removes it from the count.

Ambiguity never resolves upward into a PASS. Sound captures falling short is FAIL, stated plainly. Unsound captures is INCONCLUSIVE with the settling step named. A shipped false PASS costs far more than another look — and downgrading a real defect to INCONCLUSIVE to avoid delivering the news is the same failure in the other direction.