qa

QA

Core Principle: Nothing ships without proof, and the fixer never certifies its own fix. qa drives the real application, attacks it, and — if it fails — owns the diagnose→fix→re-verify loop through independent agents until the surface is actually green.

Overview

Pure dynamic adversarial-e2e verification skill. qa reads the change to author high-coverage scenarios and proves them by execution: static document-vs-code auditing (Security/Data-Integrity checklists, MUST-DO compliance tables, Completeness prose audits) stays code-review's job; the behavior-invisible PRE-FLIGHT contract gate below is a narrow exception, not a static-audit stand-in.

qa is standalone and stateful. A single invocation owns the whole cycle — detection, diagnosis, fix, and re-verification — through to a final verdict, persisting its phase/cycle to a state file so an interrupted run can resume with continue. Activation is unchanged: invoke qa through its existing skill trigger/tool path; the enforcement below governs an invoked session rather than changing when qa activates.

Standards: The application actually runs, survives hostile probing across all 6 adversarial categories, and any regression introduced while fixing it is caught by a fresh full re-run, not the fixer's own say-so. Setup cost—including starting multiple local apps or seeding local databases—is never a reason to skip adversarial scenarios: run every authored scenario and retain its evidence proving correct development.

QA REQUEST Format

The caller composes a QA REQUEST using this structure:

# QA REQUEST

## Spec
[WHAT to verify — requirements, criteria, constraints, MUST-NOT-DO scope]

## Required Verification
[HOW to verify — verification commands, QA scenarios, evidence paths to collect. Optional but standard for sisyphus-orchestrated QA requests.]

## Scope
- Changed files:
  - [explicit file paths]
- Summary: [what the implementer claimed]

• # QA REQUEST → ## Spec / Required Verification / Scope → ### internal subsections
• The content of Spec is PLAN's input: it determines the verification targets and the adversarial scenarios PLAN derives.
• Required Verification is used when sisyphus explicitly passes verification commands and evidence paths — BASELINE and ADVERSARIAL E2E execute the section's commands verbatim and store evidence at the declared paths.
• When a delegation prompt is included, its sections become ### headings under ## Spec

To understand what changed, use git diff $(git merge-base HEAD main) -- <path> for context. If main does not exist, substitute master. To verify correctness, read the actual files directly (Read tool). Do not independently discover which files changed — use the file list from the QA REQUEST Scope.

---

The Cycle

qa runs a single stateful cycle, in order:

PRE-FLIGHT → PLAN → BASELINE → ADVERSARIAL E2E → CHECK → [DIAGNOSIS → FIX → RE-VERIFY loop, ≤5 cycles] → EXIT → CLEANUP → ROLLBACK → STATE

Every phase below runs once per pass, except the bracketed loop, which repeats on CHECK failure until an EXIT condition fires.

PRE-FLIGHT

A behavior-invisible contract check — a narrow exception to qa's dynamic-only posture, because no amount of running the app surfaces a scope violation. Gates on exactly two things:

1. MUST-NOT-DO scope membership. A changed file violates the contract iff it matches the QA REQUEST's MUST-NOT-DO scope — no positive allowlist, no per-invocation judgment call. Tests and config files are NOT special-cased: they are violations only if the MUST-NOT-DO explicitly names them, and clean otherwise.
2. B ⊆ A scope boundary. Expected files (from EXPECTED OUTCOME) = A; Changed files (from QA REQUEST Scope) = B. PASS if B ⊆ A. When the QA REQUEST carries no EXPECTED OUTCOME, A does not exist: record this gate as not-evaluable and proceed on gate 1 alone. Never fill A from the Scope list — B ⊆ B is true by construction and turns the gate into a rubber stamp that reads like a PASS.

On violation: immediate REQUEST_CHANGES, cycle NOT executed — fail-fast. The expensive cycle below never runs against a change that already fails its own declared contract.

PRE-FLIGHT also captures the ROLLBACK safety baseline (used only if the loop below runs): snapshot git status --porcelain as user_dirty_set (the user's pre-existing dirty/uncommitted files) plus current HEAD — each entry is a porcelain status line (XY <path>); the file path is the portion after the status code (accounting for rename old -> new syntax).

At cycle entry, create or re-enter the guarded state with bun ${CLAUDE_SKILL_DIR}/scripts/qa-state.ts start --target "<what is being verified>". A second qa invocation in the same session must run start again so it receives a fresh chain and re-armed runtime gates.

PLAN

Two ordered outputs. The roster comes first because it fixes where every scenario must be entered and what its evidence has to show — scenarios authored before it drift inward toward whatever is easiest to call.

PLAN.1 — Actor Roster, before any scenario

Enumerate every actor the changed surface serves and pin each one's boundary, as actor · boundary · driver · reachable:

• actor — who acts: an end user on a named path, a specific role (household owner vs payer), an operator/admin, a calling client system, an attacker.
• boundary — the exact thing that actor touches with its own hands: this screen in this app, this HTTP endpoint, this CLI command, this queue's trigger. A function, a class, or an internal module is never a boundary — nobody touches those.
• driver — the tool that reaches that boundary (agent-device / agent-browser / curl / bash).
• reachable — yes, or the named obstacle plus the deepest point toward the boundary that IS reachable (see Boundary substitution).

A change with no UI still has actors. When the changed code is internal, trace the call graph outward from it until you reach something a human or an external system touches — that is the boundary, not the function that changed. Emit the result as the ## Actor Roster output section.

Record the roster in state before authoring scenarios. First capture the acceptance criteria — the concrete pass conditions this change must meet, taken from the QA REQUEST Spec (or the derived expected outcome) — with qa-state.ts set-acceptance --json '["…","…"]'; the command accepts only a JSON array of non-empty strings, and the report renders its Acceptance Criteria section from this record. Then add each actor with qa-state.ts add-actor --id … --name … --boundary … --driver agent-device|agent-browser|curl|bash --reachable unknown, and update --reachable after the PLAN.1 probe. Add at least one story per actor with add-story; the roster and stories are the referential base for every scenario cell.

The roster spans the actor's journey, not the diff (CRITICAL). Never QA only the platform where the change landed. A changed surface is verified from every platform where an actor observes it, and every platform holding one of its preconditions — the admin web that toggles the flag, the operator tool that seeds the state — enters the roster too, as a boundary this cycle will actually launch and drive during setup.

When a scenario needs an account, auth, or a data state, read the project's provisioning protocol before authoring — that path is documented, never improvised (CRITICAL). A project that has QA accounts and seedable data documents how QA obtains them: a list of pre-provisioned test accounts and the data state each carries (already has a Program/reports vs. a bare household), the admin/operator QA tool that seeds programs/reports/fixtures, the auth method for each account, and the one command that stands the local stack up with its env prerequisites. When a scenario carries such a precondition, locate and read that protocol (mine README/CONTRIBUTING, docs/, rules/, Makefile, docker-compose*.yml, scripts/ per [stage1-commands.md]) and pick the account/fixture and QA tool it prescribes for that scenario's required data state, before you touch a driver. If the search turns up no such protocol, record that in PLAN and proceed to the bootstrap ladder's fallback — never invent provisioning details, and never block on this lookup a change that has no account/auth/data precondition at all. Hand-rolling signup/onboarding to manufacture data, minting a token, or injecting dummy credentials before trying a pre-provisioned account or QA seeding tool the project does document for that state is a wrong detour, not a bootstrap — where a documented path exists it is what you attempt first, and the improvise options are the fallback for when it is absent or, once tried, unusable (see the bootstrap ladder's Documented protocol first rung). A precondition the documented protocol can satisfy is not an obstacle you improvise around; it is a step you execute as written.

PLAN.2 — Scenarios, per actor

Parse the QA REQUEST's Spec/AC into concrete verification targets: what BASELINE must run green, what ADVERSARIAL E2E must attack from each actor's boundary, and what CHECK will judge against. MUST-DO tables and Completeness sub-checks are code-review's static-audit territory, not PLAN's.

See [scenario-authoring.md] for the risk/coverage-gap derivation framework.

Author every story's eight cells before leaving PLAN: the six bare classes plus cls1/hang-timeout and cls5/flaky-green. Use qa-state.ts author-cell --story … --cls … [--sub …] --attack-point "…" --priority H|M|L. advance-phase BASELINE (or set --phase BASELINE) and every later phase are refused until chainComplete is true, so PLAN cannot be left with an empty or content-free attack plan.

BASELINE

Build/test/lint green baseline.

1. Discover project commands: check ~/.omt/{project}/project-commands.md cache first, then CLAUDE.md/README.md/build files, then ask the user. Save discovered commands back to the cache.
2. Run: Build (fast build/typecheck) → Tests → Lint. Slow native build (e.g. an RN bundle) runs only when native code changed or this is a release build (native-code-or-release) — otherwise skip it.
3. Save the full output of each check as an evidence file (see Evidence Saving Protocol below).
4. ANY failure = immediate REQUEST_CHANGES.

See [stage1-commands.md] for command-discovery detail, special cases (no tests for changed code, no build system), and output format — the content there is BASELINE's detail target, not a separate stage.

ADVERSARIAL E2E

Drive the changed surface for real and attack it. Two parts, both required when the change touches a risk surface — user-facing OR an internal risk surface (feature-flag-gated logic, payment/notification resolver internals, permission/state transitions), per [stage3-handson.md] ### Decision Logic; only a genuinely inert refactor that touches no risk surface skips:

1. Execute caller-provided scenarios verbatim, with per-scenario evidence. ANY provided-scenario failure = immediate REQUEST_CHANGES. Caller-provided scenarios always run verbatim, unchanged — the derivation framework below governs only scenarios qa self-authors; it never rewrites what the caller handed in.
2. Self-author the 6-axis adversarial matrix for the changed surface, in this order — breadth before depth:
   1. Derive candidate scenarios by breadth via [scenario-authoring.md]: Layer A impact-map → coverage-gap → H/M/L priority, then Layer D product use-case breadth (arrival paths · adjacent state transitions · lifecycle stances) from a product-context map built from the repo.
   2. Attack each derived scenario with the applicable depth rows, working highest-priority (H) first, from the 6 coverage axes: failure paths, boundary/malformed input, injection, interruption-resume + dirty state, misleading success, idempotency. See [stage3-handson.md] ## Adversarial Scenario Matrix for the full matrix and the lifecycle/applicability detail (start → verify → stop). Rows 7–9 (stale-state, dirty-worktree, flaky-rerun) are per-run checks recorded separately with record-run-check.

When a caller-provided scenario fails, record its failing cell and record every remaining unrun cell as na with the halt reason before declaring REQUEST_CHANGES. Apply the same sweep after any EXIT fired following a FIX dispatch (max-cycles, Same-Failure-3x, or Safety): inc-cycle invalidates prior-cycle records, including baseline and run-check records in the current view, so record the current cycle's remaining cells explicitly rather than leaving the gate with unrecorded work. Prior-cycle records remain in the raw state/history for audit.

For a genuinely inert refactor with no risk surface, still author the roster and all cells, record the story baseline and all three run checks, record every cell as na with the no-risk-surface reason, and run qa-state.ts declare-inert --reason "<why nothing is reachable>" once. Without that declaration an H-priority na blocks APPROVE.

Inline modality drivers, no tmux. qa itself drives the modality-appropriate tool inline — it is not delegated to a separate driver subagent:

Change Type	Driver
API endpoint	curl
Frontend / UI	agent-browser (fallback: playwright, if available)
Mobile / native UI	agent-device
CLI / TUI	interactive bash

For mobile/native UI work, load the agent-device skill first and derive the current concrete commands from its runtime agent-device help <topic> guidance. Do not copy or invent concrete driving command syntax in this skill.

Enter at the actor's boundary. Every self-authored scenario is executed by entering at its actor's boundary from the Actor Roster and observing what that actor observes. Calling the changed function, class, or module directly is a unit check, not a scenario run — it proves the code in isolation and leaves every layer between the actor and that code unexercised.

A caller-provided scenario is exempt from relocation and stays verbatim (part 1 above) whatever layer it enters at — the caller owns that choice. It is not exempt from disclosure: record its driven-at as the layer it actually entered, and it supports no claim above that layer. A caller-provided command that runs at an inner layer never satisfies an actor-boundary scenario on the same surface.

Precondition bootstrap (CRITICAL) — exhausted before any hop is called unreachable. A missing precondition is work to do, not an obstacle to record. Bootstrap it, rung by rung:

Local-first stance — stand up an isolated stack you own, and give it maximum freedom. For a source change not yet deployed, run QA against a local stack you stand up and fully control, so you are free to seed any account/data state, mutate it, force failure branches, interrupt mid-flight, and reset between scenarios. Stand up whatever the scenario needs to run and to give you that control — never a fixed checklist. A component comes up because a scenario depends on it, not because it was the thing that changed: a server-logic change still brings its database up locally — a real datastore you can freely seed and mutate beats a shared or deployed one you cannot — along with the backend and anything else that scenario exercises, while a change whose actor boundary is a command or job trigger may have no service, database, or bundler behind it at all, and running the local command is the local instance. Read the project's environment-setup commands and docs (the boot command, required env vars, local config) and apply them; a local process that fails to start on a missing or misconfigured env is a config to supply, not a reason to stop. Isolate the instance (its own ports, data directory, containers) so a shared or fragile local environment is neither a blocker — you bring up your own rather than surrender — nor something you corrupt by driving against a stack another session or developer is using. The one exception is when the QA REQUEST verifies the deployed artifact itself (rung 1's carve-out): there the deployed environment is the boundary, and a local stack proves nothing about it.

Documented protocol first — before any rung below. When the missing precondition is an account, authentication, or a data state, the project's documented QA provisioning protocol (read at PLAN.1) decides the move: use the pre-provisioned account it lists for that data state, or the admin/operator QA tool it prescribes for seeding, before you improvise. The improvise options in rungs 2–3 (hand-rolled seed rows, signup/onboarding, minted tokens, injected dummy credentials) apply only when the project documents no such account or tool, or when the documented path was attempted and found unusable — expired credentials, the tool unavailable or broken, or it cannot produce the required state. Reached for before the documented path is tried, they are a wrong detour that also risks driving the actor through a flow the scenario never meant to test (onboarding, consent) and around a config gap you never diagnosed against the documented setup. Attempted-and-unusable is not the same as absent: the ladder still gets exhausted, just after the prescribed path fails rather than instead of it.

1. Environment not deployed / not accessible (branch not on stage, endpoint 404, no deploy permission) → first read what the QA REQUEST verifies. For a source change not yet deployed, the deployed environment was never the boundary: stand up locally whatever the scenario needs to run — the backend and its database, the bundler, or, for a command-boundary change, just the local command. When HOW to stand it up is not already known, mine the project's own docs and scripts for it — README.md/CONTRIBUTING.md, Makefile, docker-compose.yml/docker-compose.*.yml, scripts/ — before asking the user or declaring the precondition unreachable (see [stage1-commands.md] Discovery Order). Then re-point or re-build the app against that local backend; non-deployment is an environment choice, not a boundary obstacle. A local stack that fails to boot on a missing or misconfigured env (an unset env var, absent local config, an unsigned local credential) is the same rung: read the documented env-setup, supply the missing configuration on your own isolated instance, and retry — a startup config gap is bootstrap work, not a stop, and "the local backend won't start" or "the stack is shared, so I left it alone" is env surrender, not an unreachable boundary. When the QA REQUEST verifies the deployment itself — a release, deploy config, routing, migration, packaged artifact — the deployed environment IS the boundary: its 404 is the scenario's observed FAIL (or NOT-RUN when truly unreachable), never a precondition to bootstrap around, and a local stack proves nothing about the deployed artifact ([stage3-handson.md] stale-state row).
2. No data → read the schema and create seed data; when the seeding procedure itself is not already known, mine README/Makefile/docker-compose.yml/scripts/ for a seed script or documented seed procedure first.
3. No account / credential → first use the pre-provisioned QA account the documented protocol prescribes for the data state you need (e.g. an account that already carries a Program, or the admin QA tool that seeds one). Only when the project documents no such account — or the prescribed account/tool was tried and proved unusable (expired credentials, a broken/unavailable tool, or it cannot produce the required state) — do you run the signup flow as its own scenario step, or mint a test token and inject it (cookie/header/session).
4. Precondition satisfiable only on another platform (a flag toggled in an admin web, a state set from an operator tool) → launch that platform too, satisfy the precondition there for real, then verify the feature on the platform under test. Launching the precondition platform is setup cost, and setup cost never skips a scenario.
5. Genuinely external dependency outside your control (third-party API off-network, physical hardware absent) → only this rung enters Boundary substitution below: fake that hop alone while every other layer runs real.
6. Required tool missing locally (a CLI, browser driver, emulator/simulator toolchain, or other dependency the verification needs is not installed) → install it, don't skip the scenario. Try project-local first — a devDependency, a local bin, a project-scoped install — no machine mutation. Only when a tool genuinely cannot be project-local (e.g. a system emulator/simulator toolchain) try a global install. If the global install fails (offline, locked-down) — substitute only that hop and record what was substituted and why, the same discipline as rung 5's Boundary substitution; never treat the failed install as a reason to skip the scenario outright. Never use rm -rf or a force flag to clear the way for an install.

Declaring an obstacle without recording which rungs were attempted and why each failed is boundary evasion, not a coverage delta.

Boundary substitution. An unreachable boundary — one still unreachable after the bootstrap ladder above, like absent physical hardware or an off-network third-party dependency — never relocates the scenario inward while the claim stays where it was. In order:

1. Drive from the actor's boundary anyway, replacing only the unreachable hop with a fake or stub (fake transport, stubbed external API, seeded local data), so every layer between the actor and that hop still executes.
2. Record the scenario's driven-at as the deepest point actually entered plus what was substituted — e.g. app dispense screen → hardware command (USB transport faked).
3. Only when even that is impossible is the scenario NOT-RUN, not PASS. A recorded coverage delta states what remains unproven; it is never a substitute for running it.

NOT-RUN is reserved for physical impossibility, never for setup cost. A boundary is NOT-RUN only when it stays unreachable after you have stood up every component you control and created every data/account/membership state the scenario needs — absent physical hardware, or an off-network third party you cannot run locally (rung 5). The amount of bootstrap work — full local stack, manual DB inserts, minting QA accounts and memberships, toggling flags on another platform — never makes a boundary NOT-RUN; it is the work the ladder requires. "Focused time-box", "beyond a reasonable attempt", or a caller telling you to time-box do not change this.

Substitution replaces one unreachable hop — it never swaps the boundary for the app's test suite. Running vitest/jest/pytest on the changed service/function (even against a real local DB) is not substitution: it relocates the entire scenario off the actor's boundary and down to a unit check, and its output is a test-runner report, which is never cell evidence (see the HARD RULE above). When the true boundary is a UI or API the actor touches and you cannot drive it here, the scenario is NOT-RUN and the requirement it covers is unverified — recorded loudly, never green.

Depth honesty. A verdict claims exactly the boundary its scenarios were driven from. Evidence sets collected at different depths never merge into a deeper claim: "the internal function behaves" plus "the app launches" is not "the actor's path works".