Harness Improvement Loop

What this is

A tight feedback loop for fixing harness failure modes using the framework's own diagnostic tooling (@reactive-agents/diagnose). Optimized for short iteration cycles — minutes per pass, not days. Every change lands with an empirical before/after trace as evidence.

Probe → Diagnose → Hypothesize → Fix → Verify → Commit
  ↑                                              │
  └──────────────────────────────────────────────┘
                    iterate

This skill replaces the previous "wide-scan probe + giant report template" workflow. The diagnose CLI now does the heavy lifting that hand-rolled jq queries used to do.

Prerequisites — the framework's diagnostic system

Tracing is on by default since Sprint 3.6. Every agent run writes a typed JSONL to ~/.reactive-agents/traces/<runId>.jsonl. The rax-diagnose CLI is the primary inspection surface:

bun run rax:diagnose list                     # recent runs (most-recent first)
bun run rax:diagnose replay latest            # pretty timeline of the most recent run
bun run rax:diagnose replay <runId> --only=verifier-verdict,harness-signal-injected
bun run --silent rax:diagnose grep <runId> "<js-expr>" # JSONL filter — pipe to jq for further work
bun run rax:diagnose diff <runIdA> <runIdB>   # structural diff: stats, kinds, verdicts, final state

If you're going to remember one thing: rax:diagnose replay <runId> --only=kernel-state-snapshot,harness-signal-injected,verifier-verdict tells the failure narrative of any run at a glance.

The eval system — primary probe + diagnosis substrate (2026-06, canonical)

Since the canonical evaluation system shipped (Phases 1–4b — see wiki/Architecture/Design-Specs/2026-06-24-canonical-evaluation-system.md), the benchmark bench is the primary probe, and it carries diagnosis + verdict + ledger inline. You no longer hand-roll a probe script for most sweeps:

• Probe = a 2-variant benchmark session (bare-llm vs ra-full) over real-world tasks, run cross-tier. runSession writes a SessionReport (per-cell scores, ablation harnessLift, per-run RunDiagnosis).
• Diagnosis = RunDiagnosis on every run (honesty label, failure modes, blind spots) — the "why" is attached to each score; aggregate across the report. Drill into a specific failing trace with rax:diagnose as before.
• Verdict = rax eval gate --report <r.json> --baseline bare-llm --candidate ra-full applies the project lift rule (≥3pp ∧ ≤15%tok ∧ ≥2 tiers) → default-on | opt-in | reject.
• Ledger = --ledger <path> records the weakness→hypothesis→verdict chain (wiki/Research/Harness-Reports/improvement-ledger.json); rax eval ledger lists it.

The old probe scripts (Phase 2) remain valid for targeting a single subsystem the bench tasks don't exercise, but the bench is the default for a weakness sweep.

---

Amplified loop mechanics (2026-07-07 — use these, they are the fast path)

Session evidence: one un-aimed fix-verify cycle was costing GPU-hours. These four mechanics cut it to minutes-per-iteration:

1. Start aimed — weakness queue. Fold the latest report into ranked targets (each row: severity, failure modes, evidence, traceIds, and the exact probe command):

   bun run packages/benchmarks/src/weakness-queue.ts <report.json> [--variant ra-full] [--top 10]
   

2. Verify narrow — one cell, not the matrix. --task AND --variant compose; a fix iteration costs 1 variant × 1 task × 1 run:

   bun run packages/benchmarks/src/run.ts --session <s> --task rw-2 --variant ra-full --runs 1 --output /tmp/claude-1000/<fix>-verify.json --timeout 90
   

   Four mechanics that bite (each cost a wasted cycle 2026-07):
   • --session is a PRESET name, not free text — real-world-full, cross-tier-stress, frontier-spot-check, local-models, … (an unknown name prints the list and exits). --provider/--model/--task/--variant/--runs then OVERRIDE the preset on top (e.g. swap in a single --provider anthropic --model claude-haiku-4-5-20251001).
   • ALWAYS pass --output — without it per-cell evidence (check stdout, judge inputs, error causes) is unrecoverable. It OVERWRITES per invocation (it does NOT merge across runs) — a second provider into the same file loses the first; use a distinct file per cell.
   • Run FOREGROUND with --timeout ≤590. A bench cell launched as a background task gets SIGKILLed by the reaper mid-run (silent live-model cells especially); timeout N bun … --timeout <sec> foreground.
   • Per-run scores are Bernoulli (0/1). A 0%→33% on n=3 is a MECHANISM confirmation (the deterministic barrier is gone), NOT a lift claim; never read a finding off the summary table (gaps <~26pp at small n are noise).
3. Trust the clock again — timeouts hard-kill. Cell timeouts abort the agent fiber via AbortSignal (runner.ts, 2026-07-07); zombie-fiber GPU contention no longer degrades later cells. If successful-cell durations climb monotonically across a session, that regression has returned — investigate before trusting aggregates.
4. Judge recipe (memorize): JUDGE_LAYER=live JUDGE_PROVIDER=openai JUDGE_MODEL=gpt-4o-mini bun run packages/judge-server/src/index.ts — without JUDGE_LAYER=live a silent STUB scores 0.95 on everything. Health-check with a REAL /judge POST (/version stays up when the LLM backend is dead). Error/timeout cells are never judged (scoreErrorCell) — evidence says the true cause.

Ledger discipline (opt-in today — pass it anyway). The ledger only writes when you pass --ledger, and improvement-ledger.json is gitignored; nothing enforces it yet (enforcement tracked: task #53 / north-star spec §4). Every fix that touches harness behavior SHOULD record weakness → hypothesis → verdict on the gate run:

rax eval gate --report after.json --baseline bare-llm --candidate ra-full \
  --ledger wiki/Research/Harness-Reports/improvement-ledger.json \
  --weakness "<the failure mode>" --hypothesis "<the structural fix>"

A trace-level fix can be structurally correct yet reject on lift — record both facts.

---

Phase 1 — Orient (5 minutes, do not skip)

Read these before forming hypotheses. Misdiagnosing happens when you reason about runtime behavior without knowing what the kernel is supposed to do.

Code targets

What	File
Kernel main loop	packages/reasoning/src/kernel/loop/runner.ts
Reasoning phases	packages/reasoning/src/kernel/capabilities/{reason,act,verify,decide}/
Verifier (output gate)	packages/reasoning/src/kernel/capabilities/verify/verifier.ts
Strategy adapters	packages/reasoning/src/strategies/
Provider adapters (per-tier guidance)	packages/llm-provider/src/adapters/
Context profiles (per-tier knobs)	packages/reasoning/src/context/context-profile.ts
Diagnostic emit helpers	packages/reasoning/src/kernel/utils/diagnostics.ts
AGENTS.md root + memory	AGENTS.md, ~/.claude/projects/.../memory/MEMORY.md

Wiki targets (knowledge graph queries)

The wiki is the single source of truth for prior diagnostic work. Before probing, query it.

Use claude-obsidian:wiki-query (not raw grep) — it reads the hot cache + index + drills into specific notes, surfacing semantic matches that grep misses. Falls back to grep only if wiki-query unavailable.

claude-obsidian:wiki-query "<symptom keywords> + <model tier> + <subsystem>"

Example queries that pay off:

• wiki-query "verifier rejection cogito:14b retry context" — finds M3 + related debriefs
• wiki-query "tool name hallucination ollama native FC" — finds healing pipeline (M4) work
• wiki-query "harness signal redirect parroting" — finds related FM-A patterns

What	Path	Use for
Recent context	wiki/Hot.md	Current focus, recent fixes
Past harness reports	wiki/Research/Harness-Reports/	Phase reports, baselines, diffs (this is where YOUR reports go)
Failure mode catalog	wiki/Architecture/Specs/02-FAILURE-MODES.md + wiki/Failure-Modes/	FM-A through FM-H taxonomy
Mechanism validations	wiki/Experiments/M*.md	M1-M13 spike reports — does failure relate to a known mechanism?
Past debriefs	wiki/Research/Debriefs/	What was already tried?
Active issues	wiki/Issues/Running Issues Log.md	Known blockers
Decision context	wiki/Decisions/Decision Index.md	Why current architecture exists

If wiki-query returns nothing relevant, the failure pattern may be genuinely new. Consider claude-obsidian:autoresearch "<topic>" to ground your hypothesis in external research (papers, blog posts, GitHub issues) before guessing. The autoresearch result lands in wiki/Research/<topic>/ for future agents.

Also check git log --oneline -20 so you know what shipped recently and aren't reinventing a fix that was reverted last week.

Record before proceeding: which model tier you're targeting (local / mid / large / frontier), the maxIterations default, the verifier check list (agent-took-action, synthesis-grounded, etc.). You'll need these to evaluate what trace evidence actually means.

---

Phase 2 — Probe (run a scenario, capture trace)

The probe is a small TypeScript script that exercises the harness via the public builder API. The point is to surface a failure mode in a real run, not to construct a synthetic test fixture.

Primary probe — the eval bench (cross-tier weakness sweep)

For a broad weakness/failure-mode map, run a 2-variant cross-tier benchmark session instead of a single probe script. One run surfaces harness lift, regressions, honesty, and failure modes across tiers.

Setup (once):

1. Start the frozen judge with a NON-SUT model (Rule 4 — judge ≠ any model under test):

   JUDGE_LAYER=live JUDGE_PROVIDER=ollama JUDGE_MODEL=gemma4:12b \
   JUDGE_MODEL_SHA=gemma4:12b JUDGE_CODE_SHA=dev PORT=8910 \
   bun run packages/judge-server/src/index.ts &
   # smoke: curl -s localhost:8910/version  then POST a /judge request
   

2. Use CALIBRATED models only. The bench's preflight honesty guard marks any model NOT in STATIC_CAPABILITIES (packages/llm-provider/src/capability.ts) as inconclusive (2048 fallback ctx) and refuses to score it — you get 0 measured cells, 0 dishonest numbers (guard working as designed). Calibrated local set (verified 2026-07-22): cogito:8b, cogito:14b, qwen3:4b, qwen3:14b, qwen3.5:latest; frontier calibrated incl. claude-haiku-4-5 / -4-5-20251001. gemma4:* is NOT calibrated — don't use it as an SUT (0 cells). Grep the file for the current list before a run; local availability is separate — ollama list shows what's pulled (a calibrated id you haven't pulled will fail at connect, not at the guard). (To bench an uncalibrated model intentionally: RA_BENCH_ALLOW_FALLBACK=1 — scores at fallback ctx, flagged, not representative. Better: add it to STATIC_CAPABILITIES with its real numCtx.)

Run a session (calibrated models, bare-llm + ra-full variants, traceDir set so RunDiagnosis is captured, runs≥3 for the gate's variance/significance):

JUDGE_URL=http://127.0.0.1:8910 \
bun run apps/cli/src/index.ts bench --session <session> --output report.json

Or a one-off runner that builds the session inline via runSession + getVariant("bare-llm")/getVariant("ra-full") — run it from inside the repo so the @reactive-agents/benchmarks workspace import resolves (a script under /tmp will not).

runs:1 makes the gate hair-trigger (variance 0 → any negative tier reads as a significant regression). Use runs≥3 for an authoritative verdict.

Legacy probes (single-subsystem targeting)

Existing probes (.agents/skills/harness-improvement-loop/scripts/):

Script	Purpose
task-quality-gate.ts	5 tasks (T1–T5) covering pure synthesis, single-tool, selective filter, multi-criteria, long-form. Strong general-purpose first probe.
harness-probe.ts	Wider 5-probe baseline (trivial-1step → termination-quality). Use when you don't know which subsystem is broken.
harness-probe-wide.ts	Wide-scan suite — only run when you need cross-strategy evidence.

Pick one model and run it. Examples:

TASK_GATE_MODEL=cogito:latest bun .agents/skills/harness-improvement-loop/scripts/task-quality-gate.ts
PROBE_MODEL=qwen3:14b          bun .agents/skills/harness-improvement-loop/scripts/harness-probe.ts

The probe writes a JSON summary under wiki/Research/Harness-Reports/. Tracing is automatic — one <runId>.jsonl per task run lands in ~/.reactive-agents/traces/.

Adapting probes is expected, not an anti-pattern. If the existing scripts don't exercise the failure mode you suspect, add a task or tweak maxIterations directly. Just commit the change with the run that justifies it.

---

Phase 3 — Diagnose (use the trace, not the source)

This is the phase where the new tooling pays off. The trace JSONL has already captured every state transition, verifier verdict, harness signal, and (when wired) LLM exchange. You don't have to read kernel source to know what happened — read the trace.

Eval-bench diagnosis — aggregate the SessionReport first

When the probe was an eval-bench run, the diagnosis is already in the report — aggregate it before drilling into individual traces:

• Ablation (report.ablation[].harnessLift): per (task × model) ra-full − bare-llm. Positive = harness helps; negative = harness REGRESSES (a prime root-cause target — e.g. an adversarial "nothing to optimize" task where ra-full over-engineers a false change).
• Honesty (RunDiagnosis.honestyLabel across runs): a high rate of dishonest-success-suspected / claimed-success (unverified) = models claiming success without verified deliverables — the dominant local-tier failure mode (≈95% of runs in the 2026-06 14b sweep).
• Failure modes (RunDiagnosis.failureModes): overlap-storm / nudge-loop / recall-loop / runaway-tokens / max-iter-no-progress, tallied across cells.
• Token overhead: ra-full mean tokens vs bare-llm — the harness tax (the 14b sweep saw +200–900%). A lift that fails the ≤15%tok gate is a real cost finding, not just a win.

THEN take a specific failing cell's traceId (from RunScore.traceId) and drill with rax:diagnose replay below.

For a verifiable task, read the CHECK EVIDENCE first — it names the defect

rax:diagnose replay shows trace events, but for a deterministic verifiable task the highest-signal source is the report's per-run dimension evidence (the hidden-check's stdout), the run output, and the run trust field — the check tells you EXACTLY why the deliverable failed, in one line, without reading a single trace event:

# Per failing run: trust label + the check's own failure line + output head.
python3 -c "
import json; d=json.load(open('report.json'))
tr=[t for t in d['taskReports'] if t['taskId']=='rw-1'][0]
for i,r in enumerate(tr['runs']):
    acc=[x for x in r['dimensions'] if x['dimension']=='accuracy'][0]
    print(f'run {i}: trust={r.get(\"trust\")} | {acc[\"evidence\"].splitlines()[0][:80]}')
    print('  output:', r.get('output','')[:100].replace(chr(10),' '))
"

• trust: verified-correct (deliverable passed) · claimed-but-wrong (agent claimed success, deliverable failed — the honesty layer is working; the deliverable is the bug) · unverified.
• dimension.evidence: the check's raw stdout — e.g. exit 1: 0 pass … FAIL: databases.json exists and parses — JSON Parse error: Unrecognized token '\'. That single line located the 2026-07-22 file-write defect (models fence a .jsondeliverable → unparseable file) — a boundary fix, not a trace-spelunk. **When accuracy is 0 butpassRate` is 1, this is where the answer is: the run didn't crash; the deliverable is wrong.**

Only drop to rax:diagnose replay when the check evidence + output don't explain it (the failure is in the run's behavior, not the deliverable's content).

Default first move

bun run rax:diagnose list                                        # which runs are recent?
bun run rax:diagnose replay latest                               # full timeline
bun run rax:diagnose replay latest --only=kernel-state-snapshot,verifier-verdict,harness-signal-injected

The filtered replay surfaces three high-signal narratives:

• kernel-state-snapshot per iteration — step composition, status, output length
• verifier-verdict — every gate firing with check breakdown and reason
• harness-signal-injected — every harness-authored step with file:line origin

If the failure isn't obvious from those three event kinds, broaden:

bun run rax:diagnose replay latest --only=verifier-verdict,strategy-switched,intervention-dispatched,intervention-suppressed

Structured queries — patterns that pay off

# Did the verifier reject any output, and why?
bun run --silent rax:diagnose grep latest "e.kind === 'verifier-verdict' && !e.verified"

# Which harness signals fired and from where?
bun run --silent rax:diagnose grep latest "e.kind === 'harness-signal-injected'" | jq -c '{iter, signalKind, origin, len: .contentLen}'

# Did the agent ever actually call a tool?
bun run --silent rax:diagnose grep latest "e.kind === 'kernel-state-snapshot' && e.toolsUsed.length > 0" | jq '.iter'

# What's the output trajectory across iterations?
bun run --silent rax:diagnose grep latest "e.kind === 'kernel-state-snapshot'" | jq -c '{iter, status, outputLen, terminatedBy}'

# Was strategy-switching responsible for failure?
bun run --silent rax:diagnose grep latest "e.kind === 'strategy-switched' || e.kind === 'intervention-dispatched'"

Cross-run comparison

When you suspect a recent change broke something:

bun run rax:diagnose diff <good-runId> <bad-runId>

The diff prints a stat table (iterations, tool calls, verifier rejections, harness signals, tokens, duration) plus event-kind histogram and final-state comparison. Use it to confirm "did my change move the needle in the right direction" without eyeballing scoring functions that move slowly.

What you're looking for

Signature	Likely cause
kernel-state-snapshot.toolsUsed = [] across all iterations + verifier-verdict.checks[].name = "agent-took-action" passed=false	Model never called any data tool — parrot/hallucination output suppressed by the verifier (working as designed). Investigate WHY model didn't call: prompt, schema, tier-specific FC issue.
Multiple harness-signal-injected with same origin and same content	Harness nudge is firing repeatedly without effect. Either the nudge is wrong, the model can't parse it, or the kernel control flow ignores model compliance.
verifier-verdict.checks[].name = "synthesis-grounded" passed=false	Output contains claims not present in tool observations — fabrication.
kernel-state-snapshot.outputLen > 0 while status="failed"	Output ownership invariant broken — kernel is letting a populated state.output ride past a failure transition.
verifiable task: passRate=1 but accuracy=0, and the dimension evidence shows a parse/format error on the deliverable file (e.g. JSON Parse error: Unrecognized token '\', Expected '}') with trust="claimed-but-wrong"`	The run didn't crash — the deliverable is corrupted. Common: the model fenced a structured answer (```json) or added preamble and the write tool stored it verbatim (fixed at the write boundary 2026-07-22); or the JSON is truncated (token budget) — different class. The honesty layer is fine; fix the deliverable path, not the receipt.
intervention-suppressed repeatedly with reason="below-entropy-threshold" on a stuck run	RI threshold too high for this model tier.
Final kernel-state-snapshot.terminatedBy = "dispatcher-strategy-switch" and the next strategy never recovers	Strategy-switching escape hatch isn't actually escaping — investigate the sub-strategy.

Anti-pattern to retire

The old jq cookbook (jq 'select(.kind == "...")' .reactive-agents/traces/...) is now redundant. Use rax:diagnose grep — it's faster to type and survives schema changes. If you find yourself reaching for raw jq against a ~/.reactive-agents/traces/*.jsonl, ask whether rax:diagnose replay --only=... or a grep predicate would express it more cleanly.

---

Phase 4 — Hypothesize (one structural change, not a band-aid)

Before writing any code, write down:

1. The failure mode in one sentence. "T2/T4/T5 ship harness-redirect text as the final answer."
2. The mechanism. "When the kernel terminates dispatcher-strategy-switch with state.status=failed but state.output is unset, reactive.ts's lastThought ?? state.error ?? null fallback grabs the parroted recovery nudge from a thought step and ships it."
3. The structural fix candidate. "Establish a kernel invariant: status=failed → output=null (no exceptions). Move the lastThought→state.output synthesis INTO the kernel before the verifier gate so the verifier always sees what the user will see."
4. What you expect the after-trace to show. "T2/T4/T5 → outputLen=0; T3/T5 → verifier-verdict.verified=false on agent-took-action; verifier-verdict event count goes up because the gate now fires on more terminations."

If you can't fill in (4), you don't have a falsifiable hypothesis yet — keep diagnosing.

Call advisor() before committing to the fix when the hypothesis is non-trivial. The advisor sees the full trace evidence; their independent reading often surfaces alternative root causes (H1 vs H2) that look the same in the immediate symptom.

Anti-patterns at this phase

• "Add a guard in execution-engine.ts to filter the parroted text" — band-aid; the leak is the kernel's output ownership being unclear.
• "Bump the recovery nudge budget" — symptom-papering; doesn't address why the model parrots in the first place.
• "Disable strategy switching for this case" — disabling a feature to make a probe pass is reactive, not architectural.
• "Just delete the failing test" — never. Test failures are signal.

---

Phase 5 — Fix (one coordinated change, then stop)

Make the minimum coordinated set of edits that implements the hypothesis. Don't fold in adjacent cleanups — those are a separate commit.

Do NOT reflexively rebuild before re-probing. Under Bun, the bench runner (packages/benchmarks/src/run.ts) and the probe scripts import workspace src via the bun exports condition (added to ~all packages 2026-04-28) — a src edit is LIVE in the next probe with zero rebuild (verified 2026-07-22: a file-operations.ts src fix moved rw-1 0%→33% on the very next bench run, no build). Rebuilding "just in case" wastes minutes per iteration.

Rebuild ONLY when the probe consumes dist rather than src:

• the rax:diagnose CLI after you changed packages/diagnose or its deps → cd packages/diagnose && bun run build;
• a Node-runtime probe (no Bun) or an npm-publish / .d.ts validation.

Then re-run the probe with the same model + scenario as Phase 2. If you genuinely suspect stale code is masking your edit, confirm with git diff on the src file, not a blind rebuild.

---

Phase 6 — Verify (before/after, structurally)

Two checks every fix must pass before commit: