test

THE IRON LAW: Write code before test? DELETE IT. Start over.

• Do NOT keep it as "reference"
• Do NOT "adapt" it while writing tests
• Do NOT look at it to "inform" test design
• Delete means delete. git checkout -- <file> or remove the changes entirely. This is not negotiable. This is not optional. "But I already wrote it" is a sunk cost fallacy.

ROLE BOUNDARY: Test writes TEST FILES only. NEVER modify source/implementation files.

• Do NOT "quickly fix" a broken import in source to make tests run
• Do NOT refactor source code to be "more testable"
• Do NOT add missing exports to source files
• If source needs changes → hand off to rune:fix. Test's job ends at the test file. This separation ensures test never writes code biased toward passing its own tests.

VERTICAL SLICING (Iron Law extension): one test → GREEN → one test → GREEN. Never bulk.

• bulk_test_count MUST stay <= 1 before the first GREEN in a session
• After each GREEN, bulk_test_count resets to 0; writing 2+ tests before the next GREEN = HORIZONTAL VIOLATION
• Each cycle MUST emit a commit pair: test(scope): <behavior> + feat(scope): <behavior>
• Claim "I did TDD" is verified by completion-gate against git log --oneline — no paired commits = REJECTED
• Horizontal slicing produces tests-of-imagination, not tests-of-behavior. See references/vertical-tdd.md.
• Emit signal tdd.horizontal.violation when triggered; preflight blocks merge until cycles are unwound. Exceptions (narrow, must be documented in test header): retrofitting characterization tests for legacy untested code; spec-driven scaffolding where the contract is external (OpenAPI, wire protocol).

Instructions

Phase 1: Understand What to Test

1. Read the implementation plan or task description carefully
2. Use Glob to find existing test files: **/*.test.*, **/*.spec.*, **/test_*
3. Use Read on 2-3 existing test files to understand:
   • Test framework in use
   • File naming convention (e.g., foo.test.ts mirrors foo.ts)
   • Test directory structure (co-located vs __tests__/ vs tests/)
   • Assertion style and patterns
4. Use Glob to find the source file(s) being tested

TodoWrite: [
  { content: "Understand scope and find existing test patterns", status: "in_progress" },
  { content: "Detect test framework and conventions", status: "pending" },
  { content: "Write failing tests (RED phase)", status: "pending" },
  { content: "Run tests — verify they FAIL", status: "pending" },
  { content: "After implementation: verify tests PASS (GREEN phase)", status: "pending" }
]

Phase 2: Detect Test Framework

Use Glob to find config files and identify the framework:

• jest.config.* or "jest" key in package.json → Jest
• vitest.config.* or "vitest" key in package.json → Vitest
• pytest.ini, [tool.pytest.ini_options] in pyproject.toml → pytest
  • Async check: If pytest detected AND source files contain async def:
    • Check if pytest-asyncio is in dependencies (pyproject.toml [project.dependencies] or [project.optional-dependencies])
    • Check if asyncio_mode is set in [tool.pytest.ini_options] (values: auto, strict, or absent)
    • If async code exists but no asyncio_mode configured → WARN: "pytest-asyncio not configured. Async tests may silently pass without executing async code. Recommend adding asyncio_mode = \"auto\" to [tool.pytest.ini_options] in pyproject.toml."
• Cargo.toml with #[cfg(test)] pattern → built-in cargo test
• *_test.go files present → built-in go test
• cypress.config.* → Cypress (E2E)
• playwright.config.* → Playwright (E2E)

Verification gate: Framework identified before writing any test code.

Phase 3: Write Failing Tests

Use Write to create test files following the detected conventions:

1. Mirror source file location: if source is src/auth/login.ts, test is src/auth/login.test.ts

2. Structure tests with clear describe / it blocks (or language equivalent):

   • describe('Feature name')
     • it('should [expected behavior] when [condition]')

3. Cover all three categories:

   • Happy path: valid inputs, expected success output
   • Edge cases: empty input, boundary values, large input
   • Error cases: invalid input, missing data, network failure simulation

4. Use proper assertions. Do NOT use implementation details — test behavior:

   • Jest/Vitest: expect(result).toBe(expected)
   • pytest: assert result == expected
   • Rust: assert_eq!(result, expected)
   • Go: if result != expected { t.Errorf(...) }

5. For async code: use async/await or pytest @pytest.mark.asyncio

Python Async Tests (pytest-asyncio)

When writing tests for async Python code:

1. Verify setup before writing tests:

   • Confirm pytest-asyncio is in project dependencies
   • Confirm asyncio_mode is set in pyproject.toml [tool.pytest.ini_options] (recommend "auto")
   • If neither is configured, warn the caller and suggest setup before proceeding

2. Writing async test functions:

   • With asyncio_mode = "auto": just write async def test_something(): — no decorator needed
   • With asyncio_mode = "strict": every async test needs @pytest.mark.asyncio
   • Without asyncio_mode set: always use @pytest.mark.asyncio decorator explicitly

3. Async fixtures:

   • Use @pytest_asyncio.fixture (NOT @pytest.fixture) for async setup/teardown
   • Scope rules: async fixtures default to function scope — use scope="session" carefully with async

4. Common pitfalls:

   • Tests that pass without await — they run but don't execute the async path
   • Missing pytest-asyncio makes async def test_* silently pass as empty coroutines
   • Mixing sync and async fixtures can cause event loop errors

Phase 3.5: Cycle Discipline (Vertical Slicing Gate)

Before running the new test in Phase 4, count what's about to be added to the working tree:

bulk_test_count = (test files staged + test files unstaged but new) since the last GREEN commit

Gate: bulk_test_count MUST be exactly 1.

State	Action
bulk_test_count == 1	Proceed to Phase 4 (run RED).
bulk_test_count >= 2 AND no prior GREEN this session	HORIZONTAL VIOLATION — pause, keep one test, defer the rest.
bulk_test_count >= 2 AND last GREEN exists in git log	HORIZONTAL VIOLATION — same.
bulk_test_count == 0	No test to run; this phase is a no-op.

Cycle audit log — append to TEST.md (or create) one line per cycle:

cycle 1 — RED: test_validates_email_rejects_empty | GREEN: validateEmail handles empty | commit: 4f3a1c
cycle 2 — RED: test_validates_email_requires_at_sign | GREEN: validateEmail checks @ | commit: a92e0d

The audit log is the receipt. completion-gate reads it.

When the violation fires, do NOT delete tests automatically — surface to the calling agent: "horizontal slicing detected (N tests before first GREEN). Recommend keeping test K, deferring N-1 to subsequent cycles."

Phase 4: Run Tests — Verify They FAIL (RED)

Use Bash to run ONLY the newly created test files (not full suite):

• Jest: npx jest path/to/test.ts --no-coverage
• Vitest: npx vitest run path/to/test.ts
• pytest: pytest path/to/test_file.py -v (if async tests and no asyncio_mode in config: add --asyncio-mode=auto)
• Rust: cargo test test_module_name
• Go: go test ./path/to/package/... -run TestFunctionName

Hard gate: ALL new tests MUST fail at this point.

• If ANY test passes before implementation exists → that test is not testing real behavior. Rewrite it to be stricter.
• If tests fail with import/syntax errors (not assertion errors) → fix the test code, re-run

Phase 5: After Implementation — Verify Tests PASS (GREEN)

After rune:fix writes implementation code, run the same test command again:

1. ALL tests in the new test files MUST pass
2. Run the full test suite with Bash to check for regressions:
   • npm test, pytest, cargo test, go test ./...
3. If any test fails: report clearly which test, what was expected, what was received
4. If an existing test now fails (regression): escalate to rune:debug

Verification gate: 100% of new tests pass AND 0 regressions in existing tests.

Phase 6: Coverage Check

After GREEN phase, call verification to check coverage threshold (80% minimum):

• If coverage drops below 80%: identify uncovered lines, write additional tests
• Report coverage gaps with file:line references

Phase 6.5: Diff-Aware Mode (optional)

When invoked with mode: "diff-aware" or by cook after implementation:

1. Run git diff main --name-only to get changed files
2. For each changed file, trace its blast radius: what imports it? what routes does it serve? what components render it?
3. Map changed files → affected routes/endpoints/pages
4. Prioritize tests: files with most downstream dependents get tested first
5. Generate targeted test commands that cover ONLY affected paths — skip unchanged modules

This mode is valuable for large codebases where running the full suite is slow. It answers: "what could this diff have broken?"

Input:  git diff main --name-only
Output: Prioritized test plan targeting only affected paths

Test Types — 4-Layer Methodology

Tests are organized in 4 layers. Each layer catches a different failure class. Higher layers are slower but catch integration issues lower layers miss.

Layer	Type	What It Catches	Framework	Speed
L1	Unit	Logic bugs, boundary violations, pure function errors	jest/vitest/pytest/cargo test	Fast
L2	Integration	API contract breaks, DB query errors, service interaction failures	supertest/httpx/reqwest	Medium
L3	True Backend	Real tool/service output correctness (not just exit 0)	Same + real software invocation	Medium-Slow
L4	E2E / Subprocess	Full workflow from user/agent perspective, installed app works	Playwright/Cypress/subprocess	Slow

Layer rules:

• L1 (Unit): Synthetic data, no external deps. Every function tested in isolation. Fast, deterministic, CI-friendly
• L2 (Integration): Tests service boundaries — API endpoints, DB operations, message queues. May need test DB or mock server
• L3 (True Backend): Invokes the REAL tool/service and verifies output programmatically. No graceful degradation — if the dependency isn't installed, tests FAIL (not skip). Verify: magic bytes, file size > 0, content structure. Print artifact paths for manual inspection
• L4 (E2E/Subprocess): Tests the installed command/app via subprocess or browser automation. Full user workflow: input → process → output → verify

"No graceful degradation" rule (L3/L4): Hard dependencies MUST be installed. Tests MUST NOT skip or produce fake results when the dependency is missing. A silently skipping test is worse than a loudly failing test.

Additional modes:

Type	When	Speed
Regression	After bug fixes	Fast
Diff-aware	After implementation, large codebases (Phase 6.5)	Fast (targeted)

TEST.md — Test Plan + Results Document

For non-trivial features (3+ test files or 20+ test cases), create a TEST.md in the test directory. This is BOTH a planning doc (written BEFORE tests) and results doc (appended AFTER tests pass).

Before writing tests — write the plan:

# Test Plan: [Feature Name]

## Test Inventory
- `test_core.py`: ~XX unit tests planned (L1)
- `test_integration.py`: ~XX integration tests planned (L2)
- `test_e2e.py`: ~XX E2E tests planned (L3/L4)

## Unit Test Plan (L1)
| Module | Functions | Edge Cases | Est. Tests | Req IDs |
|--------|-----------|------------|------------|---------|
| `core/auth.py` | login, register, refresh | expired token, invalid creds, rate limit | 12 | REQ-001, REQ-003 |

## E2E Scenarios (L3/L4)
| Workflow | Simulates | Operations | Verified | Req IDs |
|----------|-----------|------------|----------|---------|
| User signup | New user onboarding | register → verify → login | Token valid, profile created | REQ-005 |

## Realistic Workflow Scenarios
- **[Name]**: [Step 1] → [Step 2] → verify [output properties]

After tests pass — append results:

## Test Results
[Paste full `pytest -v --tb=no` or `npm test` output]

## Summary
- Total: XX | Passed: XX | Failed: 0
- Execution time: X.Xs | Coverage: XX%

## Requirement Coverage
| Req ID | Test File(s) | Status |
|--------|-------------|--------|
| REQ-001 | `test_auth.py::test_login` | ✅ Covered |
| REQ-002 | — | ❌ Not covered |

## Gaps
- [Areas not covered and why]

Why TEST.md: Planning tests before code catches missing edge cases early. Appending results creates permanent evidence. One document = complete testing story.

Skill Behavior Tests (Eval Scenarios)

For testing SKILL.md behavior (not code), use Eval Scenarios — unit tests for skill files, not code files.

Eval Scenario Format

## Eval: E[NN] — [scenario name]

### Prompt
[The exact situation/message an agent receives]

### Expected Reasoning
[Step-by-step reasoning the agent SHOULD follow]

### Must Include
- [Assertion 1: what the output MUST contain or do]
- [Assertion 2]

### Must NOT
- [Anti-pattern 1: what the output MUST NOT do]
- [Anti-pattern 2]

### Category
happy-path | adversarial | edge-case | jailbreak | credential-leak

Eval Coverage Requirements

A skill is behavior-tested when it has evals covering:

Category	Min Evals	Purpose
Happy path	1	Core workflow executes correctly
Edge case	1	Empty input, missing context, unusual state
Adversarial	1	Time pressure, sunk cost, authority pressure
Jailbreak / injection	1	Prompt injection attempt, "ignore instructions"

Minimum: 4 evals per skill (1 per category). Security-critical skills (sentinel, safeguard): 8+ evals.

Eval Storage

Save eval files as skills/<name>/evals.md. Each eval is a numbered scenario (E01–E24 range). skill-forge Phase 7 checks for evals presence before ship.

Error Recovery

• If test framework not found: ask calling skill to specify, or check package.json devDependencies
• If Write to test file fails: check if directory exists, create it first with Bash mkdir -p
• If tests error on import (module not found): check that source file path is correct, adjust imports
• If Bash test runner hangs beyond 120 seconds: kill and report as TIMEOUT

Called By (inbound)

• cook (L1): Phase 3 TEST — write tests first
• fix (L2): verify fix passes tests
• review (L2): untested edge case found → write test for it
• deploy (L2): pre-deployment full test suite
• preflight (L2): run targeted regression tests on affected code
• surgeon (L2): verify refactored code
• launch (L1): pre-deployment test suite
• safeguard (L2): writing characterization tests for legacy code
• review-intake (L2): write tests for issues identified during review intake
• scaffold (L1): generate initial test suite for new project
• graft (L2): write integration tests for grafted code
• skill-forge (L2): write tests for new skill functionality
• mcp-builder (L2): write tests for MCP server tools
• debug (L2): write regression test capturing the bug
• plan (L2): reference test requirements in implementation plan

Calls (outbound)

• verification (L3): Phase 6 — coverage check (80% minimum threshold)
• browser-pilot (L3): Phase 4 — e2e and visual testing for UI flows
• debug (L2): Phase 5 — when existing test regresses unexpectedly

Data Flow

Feeds Into →

• cook (L1): test results (pass/fail/coverage) → cook's Phase 5 quality gate evidence
• completion-gate (L3): test runner stdout → evidence for "tests pass" claims
• fix (L2): failing test output → fix's target (what to make green)

Fed By ←

• plan (L2): phase file test tasks → test's RED phase targets (what to test)
• review (L2): untested edge cases found during review → new test targets
• fix (L2): implemented code → test's GREEN phase verification target

Feedback Loops ↻

• test ↔ fix: test writes failing tests (RED) → fix implements to pass → test verifies (GREEN) → if new failures emerge, loop continues
• test ↔ debug: test discovers regression → debug diagnoses root cause → test writes regression test to prevent recurrence

Anti-Rationalization Table

Excuse	Reality
"Too simple to need tests first"	Simple code breaks. Test takes 30 seconds. Write it first.
"I'll write tests after — same result"	Tests-after = "what does this do?" Tests-first = "what SHOULD this do?" Completely different.
"I already wrote the code, let me just add tests"	Iron Law: delete the code. Start over with tests. Sunk cost is not an argument.
"Tests after achieve the same goals"	They don't. Tests-after are biased by the implementation you just wrote.
"It's about spirit not ritual"	Violating the letter IS violating the spirit. Write the test first.
"I mentally tested it"	Mental testing is not testing. Run the command, show the output.
"This is different because..."	It's not. Write the test first.
"I'll batch the tests since they're related"	Batched tests = tests of imagination. Each cycle reacts to the prior. Write one, GREEN it, then the next.
"All five tests are already written, let me just review them with you"	Same fallacy as code-before-test. Keep the first one, defer the others to subsequent cycles.

Advanced: Oracle-Injection E2E Testing

For data pipelines, AI workflows, and multi-stage processing where comparing full output structures is impractical, use oracle injection:

1. Generate a UUID oracle token: const oracle = crypto.randomUUID()
2. Inject into synthetic input: embed the oracle in realistic test data that flows through the pipeline
3. Run the full pipeline: input → all stages → output
4. Search for oracle in output: if found → data flowed end-to-end correctly

// Example: testing a document processing pipeline
const oracle = "ORACLE-" + crypto.randomUUID();
const testDoc = `Meeting notes: discussed ${oracle} integration timeline`;
const result = await pipeline.process(testDoc);
assert(result.output.includes(oracle), "Oracle not found — pipeline lost data");

When to use: E2E tests for pipelines with 3+ stages, LLM-based processing, ETL workflows, or any system where output structure is complex/non-deterministic but data preservation is critical.

When NOT to use: Unit tests, simple CRUD, or when exact output comparison is feasible.

Spec→Test Traceability

When a spec or plan with acceptance criteria exists (.rune/features/<name>/requirements.md, plan.md, or phase file), every criterion MUST map to at least one test case.

Spec Acceptance Criteria → Test Case → Implementation

US-1/AC-1.1: "User can reset password via email" → test_password_reset_sends_email()
US-1/AC-1.2: "Rate limit: max 3 reset attempts/hour" → test_password_reset_rate_limit()
FR-3: "Expired tokens rejected" → test_expired_reset_token_rejected()

Validation step (after writing tests): Cross-check acceptance criteria against test names. For each criterion:

• Has test → OK
• No test → flag as UNTESTED REQUIREMENT (more serious than uncovered lines)

Cross-boundary minimum: every user story whose path crosses the UI↔data boundary (story has both a UI task and an endpoint/data task) needs ≥1 L2 integration test exercising handler → service/endpoint with the REAL route wired (contract-level is fine; mocking the ENTIRE chain does not count). A story covered only by unit tests with the handler mocked = the story is NOT covered — that's how dead buttons reach 80% coverage.

Contract tests first: when plan emitted .rune/features/<name>/contracts/, each contract gets a failing contract test (request/response shape, error cases) BEFORE the endpoint is implemented — the RED phase for the API surface.

Why this is stronger than coverage: Coverage checks that lines were EXECUTED. Traceability checks that INTENT was VERIFIED. You can have 100% coverage but miss a requirement if the test doesn't assert the right behavior.

Skip if: No spec AND no plan exists (ad-hoc fix), or neither has an acceptance criteria section. A requirements.md with US/AC IDs makes this section MANDATORY — "no plan file" alone is not a skip reason.

Browser click-through (advisory): when a UI story completes, SUGGEST a browser-pilot run of the story's Independent Test (from requirements.md) — one real click-path beats ten mocked renders. Advisory, not a gate: skip freely in headless/CI-only environments.

Eval-Driven Development

Define capability evals and regression evals BEFORE writing implementation code. Evals go beyond unit tests — they verify that the agent/system can handle the feature's intent, not just its mechanics.

Two Eval Types

Type	Purpose	Pass Criteria	When
Capability eval	Can the system do this new thing?	pass@k: ≥1 success in k attempts (k=3-5)	Before implementation
Regression eval	Did we break existing behavior?	pass^k: ALL k attempts must pass	After implementation

pass@k (capability): At least 1 of k runs succeeds. Used for new features where some variance is acceptable. Threshold: ≥90% pass@3 for standard features, ≥95% pass@5 for critical paths.

pass^k (regression): ALL k runs must pass. Used for existing behavior that must never break. If ANY run fails, it's a regression. Threshold: 100% pass^3.

Eval File Format

Store evals in .rune/evals/<feature>.md:

# Eval: <feature name>

## Capability Evals (pass@k)
| ID | Description | k | Threshold | Status |
|----|-------------|---|-----------|--------|
| CAP-1 | [what the system should be able to do] | 3 | 90% | pending |

## Regression Evals (pass^k)
| ID | Description | k | Status |
|----|-------------|---|--------|
| REG-1 | [existing behavior that must not break] | 3 | pending |

Anti-Pattern: Eval Overfitting

Do NOT overfit evals to specific prompts or known examples. Evals should test the capability, not the exact input.

• BAD: "When user says 'hello', respond with 'Hi there!'" — tests exact string match
• GOOD: "When user greets, respond with a greeting" — tests capability

Integration with TDD

1. Write eval definitions (capability + regression) → .rune/evals/<feature>.md
2. Write unit/integration tests (RED phase) → test files
3. Implement feature (GREEN phase) → source files
4. Run evals to verify capability achieved + no regressions
5. Preflight checks eval results as part of quality gate

Red Flags — STOP and Start Over

If you catch yourself with ANY of these, delete implementation code and restart with tests:

• Code exists before test file
• "I already manually tested it"
• "Tests after achieve the same purpose"
• "It's about spirit not ritual"
• "This is different because..."
• "Let me just finish this, then add tests"

All of these mean: Delete code. Start over with TDD.

Constraints