calibrate

Validate agents and skills by measuring outputs against synthetic problems with defined ground truth. Primary signal: calibration bias — gap between self-reported confidence and actual recall. Well-calibrated agent reports 0.9 when it finds ~90% of issues. Miscalibrated: reports 0.9, finds 60%.

Calibration data drives improvement loop: systematic gaps → instruction updates; persistent overconfidence → adjusted re-run thresholds in MEMORY.md.

NOT for: static routing overlap analysis (use /foundry:audit); manually reviewing skill output quality (use /develop:review (requires develop plugin)).

• $ARGUMENTS: parse --flags first, then resolve remaining tokens as scope targets

  Flags (order independent):

  • --fast — 3 problems per target (default when neither pace flag passed)
  • --full — 10 problems per target; mutually exclusive with --fast
  • --ab-test — also run general-purpose baseline and report delta metrics; requires benchmark (default --fast if no pace flag); mutually exclusive with --apply
  • --apply — apply proposals: with --fast/--full: run benchmark then immediately apply; without pace flag: skip benchmark, apply proposals from most recent past run; mutually exclusive with --ab-test
  • --skip-gate — suppress follow-up gate; for programmatic callers
  • --local — resolve target agent/skill files from source tree (plugins/*/) instead of installed plugin cache; for plugin-dev workflows where local edits aren't yet installed; sets LOCAL_MODE=true in all pipeline spawns

  Mutual exclusion validation (check before any work):

  • --ab-test + --apply together → hard error: "--ab-test and --apply are mutually exclusive. Pass one or neither."
  • --fast + --full together → hard error: "Pass --fast or --full, not both."
  • --ab-test without pace flag → default --fast silently (no error)

  Unsupported flag check — after all supported flags extracted (--fast, --full, --ab-test, --apply, --skip-gate, --local, --keep), scan $ARGUMENTS for remaining --<token> tokens. If found: print ! Unknown flag(s): `--<token>`. Supported: `--fast`, `--full`, `--ab-test`, `--apply`, `--skip-gate`, `--local`, `--keep`. then invoke AskUserQuestion — (a) Abort (stop, re-invoke with correct flags) · (b) Continue ignoring (skip unknown flags, proceed). On Abort: stop.

  Legacy positional tokens (ab, apply, fast, full) — hard error: print migration hint and stop. Example: "ab removed — use --ab-test flag: /calibrate curator --ab-test."

  Scope tokens (positional, space-separated — defaults to all):

  • all — all agents + relevant skills + routing + communication + all rules
  • agents — all agents only (full agent list in modes/agents.md)
  • skills — calibratable skills only (/audit and others per modes/skills.md; /oss:review (requires oss plugin) excluded — requires live GitHub PR)
  • routing — routing accuracy test: measures how accurately general-purpose orchestrator selects correct subagent_type for synthetic task prompts (not per-agent quality benchmark; included in all)
  • communication — handover + team protocol compliance: runs foundry:curator against synthetic agent responses and team transcripts with injected protocol violations (missing JSON envelope, missing summary, AgentSpeak v2 breaches); included in all
  • rules — rule adherence test: for each global rule file (no paths:) and each path-scoped rule when matching file is in context, generates synthetic tasks that should trigger rule's key directives, measures whether general-purpose agent with rule loaded correctly applies them; reports rules that are ignored, misapplied, or redundant; included in all
  • plugins — all agents + calibratable skills from all plugins/*/ directories (union of all plugin-namespaced agents and calibratable skills)
  • <plugin-name> — tier 2: bare plugin directory name (e.g. oss, foundry, research, develop) auto-resolved when token matches plugins/<name>/ directory; calibrates all agents + calibratable skills in that plugin
  • <agent-name> — tier 3: single agent (e.g., foundry:sw-engineer); also accepts bare name (e.g. sw-engineer) and resolves via plugins/*/agents/<name>.md
  • /foundry:audit — single skill (pass any calibratable skill name; /oss:review (requires oss plugin) accepted but excluded per modes/skills.md)
  • Multiple scope tokens — space-separated; calibrates union of resolved targets: oss research, agents skills, curator shepherd; each token resolved through same tier hierarchy as /audit scope tokens (reserved keywords first, then plugin-dir lookup, then agent/skill file search)

  Every invocation surfaces report: benchmark runs print new results; --apply without pace flag prints saved report from last run before applying.

• FAST_N: 3 problems per target
• FULL_N: 10 problems per target
• RECALL_THRESHOLD: 0.70 (below → agent needs instruction improvement)
• CALIBRATION_BORDERLINE: ±0.10 (|bias| within this → calibrated; between 0.10 and 0.15 → borderline)
• CALIBRATION_WARN: ±0.15 (bias beyond this → confidence decoupled from quality)
• CALIBRATE_LOG: .notes/logs/calibrations.jsonl (legacy .claude/logs/calibrations.jsonl read-only fallback for historical entries)
• AB_ADVANTAGE_THRESHOLD: 0.10 (delta recall or F1 above this → meaningful advantage; below → marginal or none)
• PHASE_TIMEOUT_MIN: 5 (per-phase budget — if spawned subagents haven't all returned, collect partial results and continue)
• PIPELINE_TIMEOUT_MIN: 10 (hard cutoff — pipeline not notified within 10 min of launch is timed out; extendable if agent explains delay) # tighter than global 15-min cutoff from CLAUDE.md §6 — intentional for calibrate
• PIPELINE_BATCH_SIZE: 5 when one mode category runs alone, 2 while two categories are in flight (max agent/skill pipeline subagents spawned concurrently within one mode — prevents agent count explosion on all; batch: spawn ≤ that many, wait for all results, then spawn next batch; halving keeps peak concurrency at 4 ≤ 5 when paired)
• ROUTING_ACCURACY_THRESHOLD: 0.90 (below → agent descriptions need improvement) # keep in sync with modes/routing.md
• ROUTING_HARD_THRESHOLD: 0.80 (below → high-overlap pair descriptions need disambiguation)
• SPAWN_GATE_THRESHOLD: 50 (spawn estimate = target-count × N; above this, large-fan-out gate fires before Step 2 even when --apply is set — only --skip-gate bypasses)

• PROBLEM_SET_VERSION: 1.0
• CODEX_PROBLEM_RATIO: 0.6 (fraction of in-scope problems generated by Codex — agents/skills modes only)
• CODEX_SCORER_WEIGHT: 0.49 (Codex scorer weight; Claude = 0.51 — Claude has last word on disagreements)
• SCORER_AGREEMENT_WARN: 0.70 (scorer agreement below this → flag ambiguous ground truth ⚠)
• CODEX_MODES: ["agents", "skills"] (modes where Codex is active; routing/communication/rules excluded — test Claude-specific internals)
• PIPELINE_TIMEOUT_MIN_DUAL: 15 (hard cutoff when Codex active — replaces PIPELINE_TIMEOUT_MIN=10 for dual-source runs)

Domain tables per mode: see modes/agents.md, modes/skills.md, modes/routing.md, modes/communication.md, modes/rules.md.

Task hygiene: load and follow the protocol below.

# loads: compaction-contract.md
# audit-skip: resilience-replication — duplicated; plugin cannot self-locate
cat "$(python "${CLAUDE_PLUGIN_ROOT:-plugins/cc_foundry}/bin/resolve_shared_path.py" foundry skills/_shared 2>/dev/null || echo "plugins/cc_foundry/skills/_shared")/task-hygiene.md"

Task tracking: create tasks at start of execution (Step 1) for each phase that will run:

• "Calibrate agents" — Step 2 (benchmark mode, when target includes agents)
• "Calibrate skills" — Step 2 (benchmark mode, when target includes skills)
• "Calibrate routing" — Step 2 (benchmark mode, when target includes routing)
• "Calibrate communication" — Step 2 (benchmark mode, when target includes communication)
• "Calibrate rules" — Step 2 (benchmark mode, when target includes rules)
• "Analyse and report" — Steps 3–5 (benchmark mode)
• "Apply findings" — Step 6 (apply mode only)

Task marking discipline: create ALL category tasks as pending at the start (before any pipeline spawns). Mark a task in_progress only immediately before spawning its pipeline. Mark it completed immediately after collecting its results. Never mark more than one category task in_progress simultaneously — misrepresents execution state. On loop retry or scope change, create new task.

Step 1: Parse targets and create run directory

From $ARGUMENTS, determine:

• Strip flags first: extract --fast, --full, --ab-test, --apply, --skip-gate, --local, --keep before scope resolution; validate mutual exclusion (error and stop on conflict). Strip all flags from ARGUMENTS before scope token resolution:

  export CSID="${CLAUDE_CODE_SESSION_ID:-$PPID}"
  KEEP_ITEMS=""
  if [[ "$ARGUMENTS" =~ --keep[[:space:]]\"([^\"]+)\" ]]; then
      KEEP_ITEMS="${BASH_REMATCH[1]}"
  fi
  ARGUMENTS=$(echo "$ARGUMENTS" | sed 's/--keep "[^"]*"//g')
  rm -f .temp/state/skill-contract.md  # clear stale contract (compaction-contract.md §Lifecycle)  # timeout: 5000
  LOCAL_MODE=false; [[ "$ARGUMENTS" == *"--local"* ]] && LOCAL_MODE=true
  ARGUMENTS="${ARGUMENTS//--fast/}"; ARGUMENTS="${ARGUMENTS//--full/}"
  ARGUMENTS="${ARGUMENTS//--ab-test/}"; ARGUMENTS="${ARGUMENTS//--apply/}"
  ARGUMENTS="${ARGUMENTS//--skip-gate/}"; ARGUMENTS="${ARGUMENTS//--local/}"
  ARGUMENTS="${ARGUMENTS#"${ARGUMENTS%%[![:space:]]*}"}"
  mkdir -p "${TMPDIR:-/tmp}/calibrate-state-${CSID}"
  echo "$LOCAL_MODE" > "${TMPDIR:-/tmp}/calibrate-state-${CSID}/local-mode"
  echo "$KEEP_ITEMS" > "${TMPDIR:-/tmp}/calibrate-state-${CSID}/keep-items"
  

• Target list — remaining tokens after flag-strip; union of resolved targets:
  • all or omitted → all agents + /audit + routing + communication + all rules
  • agents → all agents (full agent list in modes/agents.md)
  • skills → /audit only (and other non-live-PR skills in modes/skills.md; /oss:review (requires oss plugin) excluded)
  • routing → routing accuracy test only
  • communication → handover + team protocol compliance only
  • rules → rule adherence test (all rule files in .claude/rules/) only
  • plugins → all agents + calibratable skills from all plugins/*/ directories
  • <plugin-name> matching plugins/<name>/ directory → tier 2: all agents + calibratable skills in that plugin
  • Any other token → tier 3: single agent or skill name; search plugins/*/agents/<name>.md, .claude/agents/<name>.md, plugins/*/skills/<name>/SKILL.md, .claude/skills/<name>/SKILL.md; error if no match
  • Multiple tokens → union: e.g. oss research, curator shepherd; each resolved independently

Empty resolution guard: after resolving all scope tokens to target list, if list is empty (e.g. plugin matched but contains no calibratable agents/skills, such as /calibrate codemap), stop with:

! No calibratable agents/skills found for scope: <input-scope>
Verify: (a) plugin name spelled correctly, (b) plugin has agents/*.md or calibratable skills (see modes/skills.md domain table)

Do not proceed to Step 2 — silent no-op produces no report and confuses callers.

• Pace: --full → 10 problems; --fast → 3 problems; neither → default --fast
• A/B flag: --ab-test → also spawn general-purpose baseline per problem
• Apply flag:
  • --apply without pace flag → pure apply mode: skip Steps 2–5; go to Step 6
  • --apply with --fast/--full → benchmark + auto-apply: run Steps 2–5 then continue to Step 6

If benchmark will run (i.e., --fast or --full present, with or without --apply): generate timestamp YYYY-MM-DDTHH-MM-SSZ (UTC, e.g. 2026-03-03T13-44-48Z) explicitly via the Bash tool and persist for downstream steps (fresh-shell state loss between Bash() calls):

export CSID="${CLAUDE_CODE_SESSION_ID:-$PPID}"
TIMESTAMP=$(date -u +%Y-%m-%dT%H-%M-%SZ)
echo "Calibration timestamp: $TIMESTAMP"
mkdir -p "${TMPDIR:-/tmp}/calibrate-state-${CSID}"
echo "$TIMESTAMP" > "${TMPDIR:-/tmp}/calibrate-state-${CSID}/timestamp"

Every subsequent Bash block in Steps 2–6 that uses $TIMESTAMP must re-read it at the top of the block:

export CSID="${CLAUDE_CODE_SESSION_ID:-$PPID}"
IFS= read -r TIMESTAMP < "${TMPDIR:-/tmp}/calibrate-state-${CSID}/timestamp" 2>/dev/null || TIMESTAMP=""
[ -z "$TIMESTAMP" ] && { echo "! TIMESTAMP state lost — re-invoke /foundry:calibrate"; exit 1; }
# never fall back to $(date ...) — generates new timestamp → nonexistent run dir; surface state loss explicitly

All run dirs use this timestamp.

Large fan-out gate — after target list resolves (and before any task creation or pipeline spawn), when --skip-gate not passed:

• Skip entirely in pure-apply mode (--apply without a pace flag) — zero pipelines spawn in this mode (routes straight to Step 6), so no confirmation is needed.
• Mode-category scopes (all, agents, skills, plugins, <plugin-name> tier 2) — the target list here is mode categories, not yet expanded to individual agent/skill files (expansion happens inside Step 2's mode files, per the mode-file table below). An exact spawn count is not knowable at this point — these scopes routinely expand to dozens of agent/skill pipelines. Gate always fires whenever a benchmark pace flag is set (--fast or --full), independent of any count.
• Tier-3 single-target scopes (<agent-name>, <skill-name>) — the target list is already a concrete file (or small union of files), so the count is exact here: SPAWN_ESTIMATE = <resolved-target-count> × (FULL_N if --full else FAST_N). Gate fires only when SPAWN_ESTIMATE > SPAWN_GATE_THRESHOLD.

When gated (either branch), fire even when --apply is set together with a pace flag — --apply only skips the Step 3 proposal-review gate, not this one.

Call AskUserQuestion:

• Mode-category scopes: question: "<scope> expands to dozens of agent/skill pipelines × <N_PROBLEMS> problems each — potentially 100+ spawns. Proceed?"
• Tier-3 scopes: question: "This run resolves to <N> targets × <N_PROBLEMS> problems ≈ <SPAWN_ESTIMATE> pipeline spawns. Proceed?"
• (a) label: Proceed — description: run as specified
• (b) label: Switch to --fast — description: re-run with --fast instead of --full (lowers spawn count ~3.3×) — omit this option when pace is already --fast/default; two-option menu (Proceed / Abort) in that case
• (c) label: Abort — description: stop; narrow scope and re-invoke

On Abort: stop immediately — no tasks created, no spawns. On Switch to --fast: replace pace flag with --fast (mode-category scopes still always-fire at --fast; tier-3 recomputes SPAWN_ESTIMATE), continue to task creation.

Create tasks before proceeding:

• Benchmark only (no --apply): TaskCreate "Calibrate agents" (if target includes agents), TaskCreate "Calibrate skills" (if target includes skills), TaskCreate "Calibrate routing" (if target includes routing), TaskCreate "Calibrate communication" (if target includes communication), TaskCreate "Calibrate rules" (if target includes rules), TaskCreate "Analyse and report" — all created as pending; do NOT mark any in_progress yet
• Benchmark + auto-apply (--fast/--full + --apply): TaskCreate "Calibrate agents" (if target includes agents), TaskCreate "Calibrate skills" (if target includes skills), TaskCreate "Calibrate routing" (if target includes routing), TaskCreate "Calibrate communication" (if target includes communication), TaskCreate "Calibrate rules" (if target includes rules), TaskCreate "Analyse and report", TaskCreate "Apply findings" — all created as pending; do NOT mark any in_progress yet
• Pure apply mode (only --apply, no pace flag): TaskCreate "Apply findings" only

Step 2: Spawn pipeline subagents

Pre-flight: mode files at <plugin-cache>/foundry/<v>/skills/calibrate/modes/ — resolve via plugin cache scan below.

/foundry:setup does NOT symlink these (only rules/*.md and TEAM_PROTOCOL.md); if not found, re-install foundry plugin.

export CSID="${CLAUDE_CODE_SESSION_ID:-$PPID}"
IFS= read -r LOCAL_MODE < "${TMPDIR:-/tmp}/calibrate-state-${CSID}/local-mode" 2>/dev/null || LOCAL_MODE="false"
CALIB_MODES_DIR=$(python "${CLAUDE_PLUGIN_ROOT:-plugins/cc_foundry}/bin/resolve_skill_subdir.py" calibrate modes $([ "$LOCAL_MODE" = "true" ] && echo --local))  # timeout: 5000

Gate: if the bash block above failed (non-zero exit or $CALIB_MODES_DIR empty) — stop immediately; do not proceed to pipeline spawns. Print: ! calibrate/modes/ directory not found — re-install foundry plugin then retry.

For each target mode in resolved target list, read corresponding mode file and execute spawn instructions. At most 2 mode categories in flight at once, and only with $PIPELINE_BATCH_SIZE halved to 2 (floor of 5÷2) for as long as two run concurrently. Rationale: the constraint being protected is peak agent count and context, not ordering — two categories at batch 2 peak at 4 concurrent pipelines, below the 5 a single category reaches on its own, so the resource ceiling is unchanged while wall-clock on all drops. Everything else stays serial: never issue a third category's spawns while two are running, and never restore batch size 5 until one of the two has fully returned its results.

Execution order for all: agents → skills → routing → communication → rules, run as pairs in that order — (agents + skills), (routing + communication), then rules alone at batch 5. For each pair:

1. Guard — call TaskList; if a category task (agents/skills/routing/communication/rules) is in_progress but its mode is not one of the categories currently in flight, call TaskUpdate(that_task_id, completed) before proceeding — corrects a missed completed call from a prior iteration. Never complete the task of a category still running: with two in flight, two category tasks are legitimately in_progress.
2. Mark both in-flight modes' tasks in_progress (all others stay pending)
3. Spawn pipelines for both modes with $PIPELINE_BATCH_SIZE = 2 (= 5 when a category runs alone — see constants)
4. Wait for all batch results from both modes before proceeding
5. Mark each mode's task completed as its own results arrive
6. Only then start the next pair

Target mode	Mode file	Task to mark in_progress
agents	$CALIB_MODES_DIR/agents.md	"Calibrate agents"
skills	$CALIB_MODES_DIR/skills.md	"Calibrate skills"
routing	$CALIB_MODES_DIR/routing.md	"Calibrate routing"
communication	$CALIB_MODES_DIR/communication.md	"Calibrate communication"
rules	$CALIB_MODES_DIR/rules.md	"Calibrate rules"
plugins or <plugin-name> (tier 2)	expand to per-agent + per-skill pipelines: glob plugins/<name>/agents/*.md and calibratable plugins/<name>/skills/*/SKILL.md; spawn one pipeline per resolved target using appropriate mode file (agents.md for agents, skills.md for calibratable skills); task name "Calibrate "	"Calibrate "
<agent-name> / <skill-name> (tier 3)	single-file pipeline: use agents.md or skills.md mode file with <TARGET> = resolved name; task name "Calibrate "	"Calibrate "

For multiple tokens, merge resolved targets into per-mode groups before spawning — one pipeline per unique mode file needed, each carrying full target list.

Before spawning any pipeline (when target includes agents, skills, or all), check cross-plugin availability. When LOCAL_MODE=true, check plugins/ source tree (local edits not yet installed); otherwise check installed plugin cache:

export CSID="${CLAUDE_CODE_SESSION_ID:-$PPID}"
IFS= read -r LOCAL_MODE < "${TMPDIR:-/tmp}/calibrate-state-${CSID}/local-mode" 2>/dev/null || LOCAL_MODE="false"
if [ "$LOCAL_MODE" = "true" ]; then
    [ -d "plugins/cc_oss" ]      && OSS_AVAILABLE="plugins/cc_oss"           || OSS_AVAILABLE=""
    [ -d "plugins/cc_research" ] && RESEARCH_AVAILABLE="plugins/cc_research" || RESEARCH_AVAILABLE=""
    [ -d "plugins/codemap-py" ]  && CODEMAP_AVAILABLE="plugins/codemap-py"   || CODEMAP_AVAILABLE=""
    [ -d "plugins/cc_develop" ]  && DEVELOP_AVAILABLE="plugins/cc_develop"   || DEVELOP_AVAILABLE=""
else
    OSS_AVAILABLE=$(find ~/.claude/plugins/cache -name "oss" -type d 2>/dev/null | head -1)  # timeout: 5000
    RESEARCH_AVAILABLE=$(find ~/.claude/plugins/cache -name "research" -type d 2>/dev/null | head -1)  # timeout: 5000
    CODEMAP_AVAILABLE=$(find ~/.claude/plugins/cache -name "codemap-py" -type d 2>/dev/null | head -1)  # timeout: 5000
    DEVELOP_AVAILABLE=$(find ~/.claude/plugins/cache -name "develop" -type d 2>/dev/null | head -1)  # timeout: 5000
fi

• agents pipeline: exclude oss:cicd-steward and oss:shepherd (requires oss plugin) if $OSS_AVAILABLE empty; exclude research:data-steward and research:scientist (requires research plugin) if $RESEARCH_AVAILABLE empty. Log: "oss/research plugin not installed — skipping calibration"
• skills pipeline: exclude /oss:review (requires oss plugin) always (requires live GitHub PR — not calibratable with synthetic input; see modes/skills.md); exclude /codemap-py:* skills (requires codemap plugin) if $CODEMAP_AVAILABLE empty; exclude /research:plan, /research:judge, /research:verify (requires research plugin) if $RESEARCH_AVAILABLE empty; exclude /develop:review (requires develop plugin) if $DEVELOP_AVAILABLE empty. Log skip message per excluded skill.

Fallback role descriptions for cross-plugin agents (if ever substituted with general-purpose) — run cat "$_FS/agent-resolution.md" (where $_FS is resolved via the cache-resolution block at the start of Step 2; if $_FS is empty, skip — role descriptions unavailable) and apply the matching fallback description.

Each mode file defines <TARGET>, <DOMAIN>, any N overrides, and extra instructions for pipeline subagent. Pipeline template lives at $CALIB_MODES_DIR/../templates/pipeline-prompt.md. N override: communication caps at fast=3 / full=5 (not global FULL_N=10) to prevent pipeline context overflow — run cat "$CALIB_MODES_DIR/communication.md" for details. rules mode spawns one general-purpose subagent per rule file (not standard pipeline template) — run cat "$CALIB_MODES_DIR/rules.md" for direct-spawn approach.