Consistency Audit — the paper vs itself

Audit intra-paper self-consistency for: $ARGUMENTS (requires claims.json from /evidence-ledger). Emit span-anchored consistency-audit.findings.json; the deterministic adjudicator — not this skill — computes the verdict.

🔒 Do not wrap this skill in /loop, /schedule, or CronCreate. It is verdict-bearing input — it proposes the findings the deterministic adjudicator turns into the report. Re-firing it on a wall-clock timer adds no signal: its output changes only when the paper / ledger changes, not with the clock. Schedule the external wait that precedes it — ledger built → audit once. (Mirrors ARIS's external-cadence doctrine.)

The flagship instrument. Internal contradiction is the single most defensible thing you can check on an unknown submission: it needs no external GT, runs at L0 (PDF-only), and is exactly where machine-generated papers crack — they hallucinate local coherence. Recall scales with the ledger: a PDF-text (L0) ledger extracts only number/scope spans, so the table/caption/method-drift checks gain teeth at L1 (LaTeX), where tables and captions are actually extracted. Adapted from ARIS paper-claim-audit, reframed from "paper vs result files" to "paper vs itself." There is no external ground truth in this skill.

Why this exists

An autoresearch pipeline (or rushed human) writes the abstract, the tables, the method section, and the appendix in separate passes and never reconciles them. The result is a paper that disagrees with itself:

• abstract quotes 85.3% accuracy; the best row of its own Table 2 is 84.7%;
• "improves by 16%" when 73.1 → 78.0 is +6.7% relative / +4.9 points;
• "mean over 5 seeds" where the number is the single best seed, and N=3 in the table;
• method section says "no test-time labels"; the experimental-setup paragraph loads gold labels for calibration;
• "comprehensive evaluation across diverse benchmarks" on two datasets, one domain.

None of this needs the code, the data, or a single external fact to detect — only the paper, read against itself. That is why this skill is the flagship and the only auditor that always runs: it runs at L0 (no external GT), the level at which we have the least to work with — though L0/PDF-text recall is bounded by what the extractor recovers (number/scope spans only); the richer table/caption checks need the L1 LaTeX ledger.

Core principle

Ledger-anchored, span-verified, paper-vs-itself, reviewer≠adjudicator. Two passes feed the pipeline:

1. a deterministic arithmetic pass (no model) — reproducible, the eval backbone;
2. a fresh cross-model semantic pass — meaning drift the arithmetic can't see.

Both emit findings conforming to schemas/finding.schema.json. Every above-info finding cites a ledger claim_id + a verbatim span (references/integrity-forensics-contract.md rules 1–2). The model proposes; tools/adjudicate_findings.py decides (references/reviewer-independence.md Layer 2). This skill computes no verdict.

How this differs from the other auditors (route correctly)

Auditor	Question it answers	Level
consistency-audit (this)	Does the paper contradict ITSELF / does described method = evaluated method?	L0 — intra-paper, no external lookup
experiment-forensics	Are the reported numbers what the code actually computes? (fake GT, self-norm, phantom)	L2
baseline-comparison-audit	Are the right baselines present, tuned, and is "SOTA" earned?	L0 stated / L2 verified
citation-forensics	Do the cited papers exist and support the claim made?	L0
proof-derivation-forensics	Does the WRITTEN proof/derivation actually establish its theorem? (gap / circularity / invalid step / symbol drift / smuggled assumption)	L1
presentation-signals	Surface "AI-flavor" hints (auxiliary, surface-class)	L0
adversarial-case-builder	Strongest evidence-bound rejection memo (no verdict weight)	any

Do NOT raise here (hand off instead): code/result-level fraud → experiment-forensics (needs L2); "first / SOTA / beats prior work" external truth → baseline-comparison-audit

• citation-forensics (emit needs_external_check); citation existence/context → citation-forensics; proof / derivation validity (a theoretical relation that must be derived, not measured — gap / circularity / invalid step / symbol drift / smuggled assumption) → proof-derivation-forensics (family G); surface/AI-flavor → presentation-signals; the rejection memo → adversarial-case-builder. This skill never reaches outside the paper.

Constants & Reviewer Calling Convention

REVIEWER_MODEL          = gpt-5.6-sol                  # different family from executor (Claude)
REVIEWER_REASONING      = xhigh                    # always; effort never lowers reviewer quality
REVIEWER_SANDBOX        = read-only                # detect-only; never mutate the paper
REVIEWER_CWD            = <paper-dir>              # so it can read claims.json + sources directly
THREAD_POLICY           = fresh mcp__codex__codex per run; NEVER mcp__codex__codex-reply
TAXONOMY_VERSION        = 0.5                      # references/hack-pattern-taxonomy.md
DETERMINISTIC_FINDINGS  = consistency-audit.deterministic.findings.json   # Step 1 (tool)
SEMANTIC_FINDINGS       = consistency-audit.findings.json                 # Step 4 (validated)
TRACE_POLICY            = forensic (never silently dropped)
TRACE_DIR               = .aris/traces/consistency-audit/<YYYY-MM-DD>_run<NN>/

• Executor (Claude) builds nothing of the judgment: it locates the ledger, passes paths + the ledger + the checklist to the reviewer, validates the reviewer's spans, and writes the findings file. It never summarizes the paper, pre-judges, or leaks an opinion into the prompt (reviewer-independence.md).
• Reviewer (codex / gpt-5.6-sol) reads claims.json and the sources, proposes findings, and self-reports false_positive_risk. It is the evidence-extractor, not the judge.
• Fresh thread per run. If you fan out the checklist into groups for breadth, each group is a new mcp__codex__codex call — never codex-reply carrying one group's conclusions into another (the bias guard). codex-reply is intentionally absent from allowed-tools.

---

Step 0 — Preconditions: locate the ledger, read the run level

The ledger is the only structure this skill reasons over. Resolve it and read the observability level L, paper_id, and the claim count it was built at (each Bash block is self-contained — shell state does not persist between calls, so re-derive paths every step):

ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
# $ARGUMENTS is a paper-dir OR a claims.json path:
LEDGER="$ARGUMENTS"; [ -d "$LEDGER" ] && LEDGER="$LEDGER/claims.json"
# Only the NO-ARGUMENT case defaults to the CWD ledger. An EXPLICIT argument that
# resolves to a missing claims.json must NOT silently fall back to $(pwd) — that
# could audit the wrong paper; let the NO_LEDGER check below fire instead.
[ -z "$ARGUMENTS" ] && LEDGER="$(pwd)/claims.json"
python3 - "$LEDGER" <<'PY'
import json, sys, os
p = sys.argv[1]
if not os.path.isfile(p):
    sys.exit("NO_LEDGER: claims.json not found. Run /evidence-ledger FIRST "
             "(it writes artifact_manifest.json + claims.json).")
d = json.load(open(p, encoding="utf-8"))
print("LEDGER      =", os.path.abspath(p))
print("PAPER_DIR   =", os.path.dirname(os.path.abspath(p)) or ".")
print("PAPER_ID    =", d.get("paper_id", "?"))
print("RUN_LEVEL_L =", d.get("observability_level", 0))
print("CLAIMS      =", len(d.get("claims", [])))
PY

Carry forward the absolute LEDGER / PAPER_DIR, plus L and PAPER_ID, into every step below.

Failure / edge handling.

• NO_LEDGER → stop and tell the user to run /evidence-ledger first. This skill never re-reads the raw PDF and invents its own structure (contract rule 1).
• CLAIMS == 0 → the ledger has nothing to audit. Run Step 1 anyway (it emits []), then skip Steps 2–3 and write the empty semantic file directly — printf '[]\n' > "$(dirname "$LEDGER")/consistency-audit.findings.json" — record a trace note (Step 5), and stop. An empty findings array is a valid, non-silent output — the adjudicator reads it as "this dimension found nothing". Do not call the reviewer on an empty ledger.
• Ledger observability_level is 2 → consistency checks run identically (they are intra-paper); the extra L2 power (paper-number↔result-file match, fake GT) belongs to /experiment-forensics, not here.

Step 1 — Deterministic arithmetic pass (no LLM)

Pure arithmetic over the ledger; reproducible; runs before any model:

ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
LEDGER="<abs path to claims.json from Step 0>"
python3 "$ROOT/tools/check_numeric_consistency.py" \
    --ledger "$LEDGER" \
    --out "$(dirname "$LEDGER")/consistency-audit.deterministic.findings.json"
# statistical self-consistency (GRIM / GRIMMER / statcheck) — also pure arithmetic,
# disjoint GRIM###/VAR###/STAT### ids; the orchestrator globs *.findings.json so this
# file is picked up automatically. z-tests are stdlib; t/F/chi2/r use optional scipy.
python3 "$ROOT/tools/check_stat_consistency.py" \
    --ledger "$LEDGER" \
    --out "$(dirname "$LEDGER")/consistency-audit.stat.findings.json"

This emits five pattern types, each already schema-valid, span-anchored, reviewer.deterministic: true, and observability_level_required: 0:

Pattern	id namespace	Severity / FP	What it catches
HP-DELTA-ERROR	NUM###	major / low	stated "improves by X%" contradicts its own two operands (new−old)/old, incl. relative-vs-absolute and reduction-direction confusion (±0.6 pt rounding tolerance).
HP-NUM-INFLATE	HL###	minor / high	a metric-bearing headline %-number in abstract/intro/conclusion appears in no extracted table cell. Only fires when ≥1 table_cell claim exists; a "look here" signal, not a verdict.
HP-GRANULARITY-IMPOSSIBLE	GRIM###	minor (major if headline) / low	a proportion reported over integer N that is not round(k/N) at the stated precision (GRIM); excludes macro/weighted/relative and non-count metrics.
HP-VARIANCE-IMPOSSIBLE	VAR###	major / low–med	a reported SD larger than a bounded metric can have at that mean (Bhatia–Davis); requires an explicit SD label, skips SEM/CI.
HP-STAT-INCONSISTENCY	STAT###	major (critical if headline) / med	a reported p that overstates the .05 significance its own test statistic supports (statcheck); z = stdlib, t/F/χ²/r = optional scipy.

Worked example — real output on the delta_inflate fixture. Ledger claim C003 text is "FooNet reaches 78.0% accuracy, improving from a 73.1% baseline to 78.0% accuracy, a 16.7% relative improvement." The tool emits (span shown exactly as it appears in the JSON — LaTeX escapes preserved verbatim, which is what the anchor gate matches against):

{
  "finding_id": "NUM001",
  "skill": "consistency-audit",
  "pattern_id": "HP-DELTA-ERROR",
  "title": "Stated improvement contradicts its operands",
  "description": "Text states a 16.7% relative change, but 73.1->78 is 6.7% relative (+4.9 absolute points).",
  "severity": "major",
  "observability_level_required": 0,
  "evidence": [{
    "claim_id": "C003",
    "span": "FooNet reaches 78.0\\% accuracy, improving from a 73.1\\% baseline to 78.0\\% accuracy, a 16.7\\% relative improvement.",
    "location": {"file": ".../delta_inflate.tex", "line": 9, "section": "abstract"},
    "artifact_hash": "a0a6b6…e45804"
  }],
  "verdict_local": "fail",
  "reviewer": {"deterministic": true},
  "false_positive_risk": "low",
  "recommended_reviewer_action": "Ask the authors to reconcile the stated delta with the reported operands; verify relative-vs-absolute convention."
}

These deterministic findings are anchored and observability_level_required: 0 by construction, so they survive every adjudicator gate — do not re-validate, re-ID, or mutate them.

Failure handling. If the tool errors, fix the invocation (run python3 "$ROOT/tools/check_numeric_consistency.py" --help) — do not hand-fabricate deterministic findings. An empty output ([]) is a valid result (no arithmetic contradiction, or no tables were extracted to compare against); keep the file.

Step 2 — Cross-model semantic pass (reviewer ≠ adjudicator)

The arithmetic layer cannot see meaning drift. First create this run's trace dir (Step 5's layout) so the reviewer call can be persisted as it happens. Shell state does not persist between Bash calls, so read the printed TRACE_DIR and reuse that absolute path verbatim in Steps 2 and 5:

LEDGER="<abs path to claims.json from Step 0>"; D="$(dirname "$LEDGER")"
TBASE="$D/.aris/traces/consistency-audit/$(date +%F)"
NN=1; while [ -e "${TBASE}_run$(printf '%02d' "$NN")" ]; do NN=$((NN+1)); done
TRACE_DIR="${TBASE}_run$(printf '%02d' "$NN")"; mkdir -p "$TRACE_DIR"
echo "TRACE_DIR = $TRACE_DIR"   # carry this absolute path into Steps 2 and 5

Then open a fresh mcp__codex__codex thread (the Reviewer Calling Convention above) and send it the checklist. The reviewer reads claims.json from its cwd; it may re-open a source file to confirm a span, but every finding must anchor to a ledger claim_id. Send EXACTLY (substitute the absolute PAPER_DIR and the L value from Step 0):

mcp__codex__codex:
  model: gpt-5.6-sol
  config: {"model_reasoning_effort": "xhigh"}
  sandbox: read-only
  cwd: <absolute PAPER_DIR from Step 0>
  prompt: |
    You are an integrity-forensics reviewer auditing a research paper for INTERNAL
    self-consistency only — the paper against ITSELF. You have NO external ground
    truth and you do NOT judge whether results are "real": you find places where the
    paper contradicts itself, or where the method DESCRIBED differs from the method
    EVALUATED.

    INPUTS (in your working directory, read them directly):
      - claims.json  — the evidence ledger: the authoritative, span-anchored list of
        every checkable claim. This is the ONLY structure you reason over for
        navigation. Each claim has {claim_id, type, text_span (VERBATIM source text),
        location, value?}.
      - the source files the ledger references — you MAY re-open them to confirm a
        span is real, but you may NOT introduce a claim that is not in the ledger.
    RUN OBSERVABILITY LEVEL L = <L from Step 0>.

    HARD RULES (a finding that breaks any of these is worthless):
    1. ANCHOR. Every finding above severity "info" MUST carry >=1 evidence entry
       {claim_id, span}, where claim_id EXISTS in claims.json and span is a VERBATIM
       substring of THAT claim's text_span — copied character-for-character INCLUDING
       LaTeX escapes like \% and \, (do NOT unescape, normalize, or paraphrase). The
       anchor check is a whitespace-normalized substring match (`span in claim`); an
       unescaped or reworded span will fail and be demoted to info. If you cannot
       quote such a span, keep the finding at "info" or drop it.
    2. DISCREPANCY, NOT ACCUSATION. description and recommended_reviewer_action
       describe what a human should CHECK or ASK. Never write "reject", "fabricated",
       or "the authors faked X".
    3. OBSERVABILITY. Set observability_level_required = the LOWEST tier at which the
       discrepancy is DECIDABLE. A purely textual contradiction (text vs text/table
       inside the paper) = 0. Anything that needs the CODE or RESULT FILES to confirm
       = 2 (a PDF-only run reports it as needing L2 — that is correct, not a loss).
    4. HONEST FP RISK. Set false_positive_risk truthfully. Legit "best config"
       labels, deliberately-labeled ablations, standard rounding, deterministic
       metrics, and honestly-labeled pilots are COMMON false positives — say so.
    5. HAND OFF EXTERNAL CLAIMS. For "first / SOTA / state-of-the-art / beats all
       prior work" that internal text cannot settle: do NOT rule. Set
       verdict_local = "needs_external_check" and requires_external_check = true.
    6. pattern_id MUST be one of the HP-* ids listed in the checklist.