Eval

The runner for the harness fitness function. It discovers .agro/evals/probes/*.sh, runs each against real state, and writes the .agro/evals/RESULTS.md scoreboard. A rectification is provably "done" when its probe is green; a recurrence shows up as a REGRESSION (was-PASS, now-fail) naming the # source: lesson. The full contract — 3-state exit oracle, header convention, correction-surface triage — is in .agro/evals/README.md.

Usage

bash .claude/skills/eval/run.sh                 # run the whole suite, rewrite RESULTS.md
bash .claude/skills/eval/run.sh --probe <id>    # run one probe, update only its row
bash .claude/skills/eval/run.sh --tier A        # run only Tier-A probes

Exit-code oracle (per probe): 0=PASS, 1=REGRESSION, 2=SKIPPED (not applicable — excluded from pass-rate), 124=TIMEOUT, other=ERROR. Each probe is wrapped in timeout 30s. Runner aggregate exit (the process $? of run.sh itself): 0 when no new green→red regression occurred this run, 1 when one or more new regressions were detected (${#regressions[@]} > 0). When invoked via the Bash tool as bash .claude/skills/eval/run.sh, the agent caller reads $? directly to gate on success — the printed REGRESSIONS (...) stdout block and per-probe stderr lines remain the human-readable signal. Note: the eval-weekly cron is an intentional legacy caller that appends || true then greps stdout; it does not consume the exit code by design — this is not a bug.

What the runner does

1. Discover + run every probe matching the filters; extract # tier: / # source: via the exact header grep.
2. Compute the delta vs the prior RESULTS.md row. First run (no prior row) emits new-pass/new-fail and raises NO regression without prior state.
3. Surface regressions — any PASS → (REGRESSION|TIMEOUT|ERROR) transition is printed first, naming the probe's source.
4. Rewrite RESULTS.md atomically — build the full scoreboard into a temp sibling file (RESULTS.md.tmp.$$) and replace the live file in one mv -f (never truncate-then-append in place), so a crash or concurrent run can't leave a partial scoreboard. Overwrite the row for each probe run; carry prior rows for probes not run this invocation from a pre-write snapshot (RESULTS_ORIG) captured before the rewrite — not the live file — so a filtered run never erases untouched rows and the scoreboard stays complete.

When NOT to use

• Tier-B behavioral evals (sub-agent + LLM-judge of judgment-call behavior) are deferred — /eval is deterministic only. Never hard-gate on a noisy metric.
• For scoring context files for staleness/budget, that is /audit context and /audit skills — /eval checks behavior/state, not prose quality.