/anti-autoresearch — the orchestrator

Run a full substantive-integrity forensic pass on $ARGUMENTS and hand a human reviewer / area chair an evidence-first Integrity Forensics Report.

🔒 External cadence: the verdict is not a poll. The pipeline is the verdict-producing path — its output changes only when the paper / repo / ledger changes, not with the clock. Do not wrap /anti-autoresearch (or any auditor sub-skill) in /loop / /schedule / CronCreate to "re-check". A heartbeat may only wait on the external steps that precede the verdict (an arXiv download, a citation web lookup) — never re-fire the adjudicated verdict, and never "decide the paper is fine now." Re-run the sweep when the inputs change; that is the only honest trigger.

🛡️ The dual of ARIS. ARIS ships an internal audit stack so its own autoresearch output stays honest; Anti-Autoresearch is that same audit DNA pointed outward at a third party's submission. This is decision support for a human — it surfaces span-anchored discrepancies to investigate. It is not an AI-text detector and it does not judge misconduct (DESIGN.md §1; references/).

Why this exists

A machine-driven research pipeline (or rushed human) writes the abstract, the tables, the method section, the bibliography, and the appendix in separate passes and never reconciles them. The result is a paper that disagrees with itself, cites papers that do not exist or argue the opposite, claims SOTA while omitting the obvious baseline, or reports numbers the code never computed. Six LLMs each re-reading the PDF would hallucinate six different structures and invite the obvious dismissal — "an LLM grading another LLM's paper is just slop."

This orchestrator answers that structurally. One deterministic pass turns the paper into a hashed, span-anchored evidence ledger; six auditors read only that ledger and propose findings; a deterministic adjudicator decides the verdict by fixed rules; observability levels are recorded on every finding so a claim that needs code cannot pass as confirmed from a PDF; and every proposal is reported rather than silently dropped. Same artifacts → same ledger → same summary.

Pipeline role (what this run does, and never does)

[0] ingest          arxiv-id | pdf | dir  →  working dir (+ pdftotext text for L0)
        │
        ▼
[1] /evidence-ledger   tools/build_manifest.py + tools/build_claim_ledger.py
        │              → artifact_manifest.json (derives observability level L)
        │              → claims.json   (span-anchored, hashed; the ONLY structure auditors read)
        ▼
[2] fan out auditors (each reads the ledger, emits <skill>.findings.json):
        consistency-audit          (always · flagship · deterministic arithmetic + semantic)
        citation-forensics         (if ≥1 citation claim)
        baseline-comparison-audit  (if ≥1 comparison / SOTA scope claim)
        experiment-forensics       (always · L0/L1 = info "could-not-verify" · L2 = full code audit)
        presentation-signals       (always · AUXILIARY · surface-class label)
        ai-style-impressions       (always · AIS track · NOT integrity · zero verdict weight)
        proof-derivation-forensics (if ≥1 theorem/proof/derivation claim · verdict-bearing · dim=proof · L1 source, CAN reach HARD_FLAGS; L0 PDF-only → info)
        eval-design-forensics      (if ≥1 comparison/eval claim · family H · dim=evaluation · L0/L1 stated-tells: leakage / judge-validity / selective-reporting)
        │
        ▼
[3] advisory memos (each reads the ledger + merged findings · NO verdict weight):
        /adversarial-case-builder      → adversarial-case-builder.memo.md       (strongest evidence-bound objection)
        /novelty-duplication-advisory  → novelty-duplication-advisory.memo.md   (if ≥1 contribution claim · MEMO-ONLY · prior-work overlap · zero verdict weight)
        │
        ▼
[4] tools/adjudicate_findings.py  --ledger REQUIRED  → report.json + REPORT.md
        │   anchor check moves severity; observability / FP-risk / surface /
        │   needs-external-check / zero-weight are recorded as annotations
        │   overall_verdict ∈ {CLEAN_GIVEN_EVIDENCE, SOFT_FLAGS, HARD_FLAGS}  (rules, no model)
        ▼
[5] present REPORT.md to the human (verdict + level first; state what could NOT be checked)

• Auditors propose; the adjudicator decides. No auditor (and not this orchestrator) ever computes overall_verdict; only tools/adjudicate_findings.py does, by fixed rules (references/reviewer-independence.md Layer 2; DESIGN.md §3).
• Detect-only. No step ever edits the audited paper or repo. This is a third-party forensics tool, never a co-author (no Edit is granted; the reviewer sandbox is read-only).
• Observability caps everything. L0 (PDF-only) / L1 (source) cannot assert code/result-level fraud — those need L2 (repo + results). The adjudicator demotes any finding above the run's level (references/observability-levels.md; DESIGN.md §4).

Constants & conventions

REVIEWER_MODEL     = gpt-5.6-sol  (via mcp__codex__codex) — a DIFFERENT model family from the executor (Claude)
REVIEWER_REASONING = xhigh    — always; the effort knob never lowers reviewer quality
REVIEWER_SANDBOX   = read-only — detect-only; the reviewer never mutates the paper
THREAD_POLICY      = FRESH mcp__codex__codex per dimension / per cited key; NEVER codex-reply across them
                     (codex-reply is deliberately NOT in allowed-tools — the bias guard)
RUN_THREADS        = serial   — one codex thread at a time; concurrent codex MCP calls can hang
LEDGER_VERSION     = 0.1       (stamped by tools/build_claim_ledger.py)
TAXONOMY_VERSION   = 0.5       (references/hack-pattern-taxonomy.md; 46 integrity patterns A–H + 13 AIS impressions + 2 ADV advisory; stamped into every report)
OBSERVABILITY      = derived   L0 (pdf/text) · L1 (latex, no results) · L2 (repo + results); L3 NEVER (no reproduction)
ADJUDICATOR        = deterministic-rules-v2   (tools/adjudicate_findings.py — the ONLY verdict source)
DETECT_ONLY        = true · EMITS_VERDICT = true (computed by code, not by a model)
REAL TOOLS (resolve ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)):
   tools/build_manifest.py · tools/build_claim_ledger.py · tools/check_numeric_consistency.py
   · tools/check_presentation.py · tools/adjudicate_findings.py   (confirm flags with --help; never invent flags)
OUTPUTS (in PAPER_DIR) — artifact_manifest.json · claims.json · <skill>.findings.json (+ *.deterministic.findings.json)
   · adversarial-case-builder.memo.md · report.json · REPORT.md · .aris/traces/<skill>/<date>_run<NN>/

Overrides (pass after —): effort: balanced (default) | max (auditors fan the checklist into more fresh threads for breadth) · enrich: true (default; the ledger's additive semantic pass) | false · human checkpoint: false (default) | true (pause after Step 1 to confirm L, and before Step 5). Example: /anti-autoresearch ~/papers/submission — effort: max, human checkpoint: true.

⚠️ Shell state does not persist between Bash calls (cwd + env reset each call). Every block below re-derives ROOT and PAPER_DIR at its top and reads L / PAPER_ID from claims.json (the authoritative source after Step 1). Never rely on a variable set in an earlier block, and never cd into the paper dir. Keep PAPER_DIR space-free (simplest way to stay safe across the inline *.tex / *.findings.json globs).

Execution modes & the Reviewer Calling Convention

Pick the mode automatically — both produce the identical file set, and Step 4 (the verdict) is byte-deterministic given those files + the ledger.

• Delegation (default — if the Skill tool is available): invoke /evidence-ledger, the six auditors, and the two advisory memos (/adversarial-case-builder + /novelty-duplication-advisory). Each sub-skill runs its own deterministic tools + its exact codex reviewer prompt + its own anchor validation, and writes its <skill>.findings.json. The orchestrator validates the emitted files and adjudicates. This mirrors ARIS /research-pipeline delegating all GPT review to /auto-review-loop rather than prompting GPT itself.
• Inline (fallback — only shell + Read/Write + codex MCP): run the deterministic tools yourself with the exact commands below; for each cross-model dimension open a fresh mcp__codex__codex thread, send the verbatim reviewer-prompt block — the fenced prompt: / checklist inside that sub-skill's ## Step … — Cross-model … section in skills/<dim>/SKILL.md (Read it — the single source of truth; fill its [...] inputs from claims.json), save the raw reply, then run the shared anchor gate (Step 2) to produce the validated <dim>.findings.json.

The orchestrator's job is to guarantee the envelope and the independence invariants every reviewer call obeys:

• Envelope (every auditor instantiates this exact shape):

  mcp__codex__codex:
    model: gpt-5.6-sol
    config: {"model_reasoning_effort": "xhigh"}
    sandbox: read-only
    cwd: <absolute PAPER_DIR>          # so the reviewer reads claims.json + sources directly
    prompt: |
      <the auditor's per-dimension checklist + the SIX HARD RULES below>
  

• The executor passes only structured inputs — paths + the ledger (claims.json) + the per-dimension checklist + the run level L. It never summarizes the paper, pre-judges, leaks a hunch, or whispers "this is probably AI-generated" (references/reviewer-independence.md Layer 1; the tool is agnostic to authorship).
• Six hard rules the orchestrator requires in every auditor's prompt (a finding that breaks one is structurally worthless): (1) ANCHOR — every above-info finding carries ≥1 evidence{claim_id, span} where claim_id exists in claims.json and span is a verbatim, whitespace-normalized substring of that claim's text_span (span in claim, never claim in span); (2) DISCREPANCY, NOT ACCUSATION — describe what to check/ask, never "reject"/"fabricated"; (3) OBSERVABILITY — set observability_level_required to the lowest tier at which the discrepancy is decidable (code/result confirmation ⇒ 2); (4) HONEST FP RISK — set false_positive_risk truthfully (legit "best config", labeled pilots, rounding, deterministic metrics are common FPs); (5) HAND OFF EXTERNAL CLAIMS — "first / SOTA" the text cannot settle ⇒ verdict_local: needs_external_check + requires_external_check: true; (6) pattern_id ∈ taxonomy v0.5 only.
• Reviewer ≠ adjudicator. The reviewer proposes findings; only tools/adjudicate_findings.py decides the verdict (Layer 2). The model is demoted from judge to evidence-extractor.
• Fresh thread per dimension, serial. A new mcp__codex__codex call per auditor (and per cited key inside citation-forensics); never codex-reply carrying one dimension's conclusions into another (the bias guard). Keep the calls serial — concurrent codex threads can hang; fan-out buys breadth of dimensions, not parallelism. On a stall, re-invoke the identical prompt in a fresh thread (never codex-reply); if it still fails, write [], record "<skill>": "review_unavailable" in $PAPER_DIR/coverage.json, and continue — a dead reviewer must never become a fabricated finding, and an empty file must never masquerade as a completed review (the adjudicator's --coverage gate turns a would-be CLEAN into REVIEW_UNAVAILABLE when any verdict-bearing dimension never ran; zero-weight tracks only add a limitation). Skills that legitimately find nothing to audit record not_applicable; completed dimensions record completed.

Re-entrancy: resuming a partial / re-run sweep

The pipeline is a pure function of PAPER_DIR's contents — every step writes a predictable file, so a crashed or repeated run resumes by probing what already exists. There is no run-state file and no "accepted vs done" gate, because Step 4 recomputes the verdict deterministically from whatever findings are present every time. Probe completeness and staleness before redoing work:

PAPER_DIR="<from Step 0>"
python3 - "$PAPER_DIR" <<'PY'
import json, os, glob, sys
D = sys.argv[1]
def is_array(p):
    try: return isinstance(json.load(open(p, encoding="utf-8")), list)
    except Exception: return False
def is_ledger(p):
    try:
        d = json.load(open(p, encoding="utf-8")); return isinstance(d, dict) and "claims" in d
    except Exception: return False
def newest_source(d):
    s = []
    for ext in ("*.tex", "*.txt", "*.bib"):
        s += glob.glob(os.path.join(d, ext)) + glob.glob(os.path.join(d, "**", ext), recursive=True)
    return max((os.path.getmtime(p) for p in s), default=0.0)
led = os.path.join(D, "claims.json"); have = is_ledger(led)
stale = have and os.path.getmtime(led) < newest_source(D)
print(f"STEP1 ledger      : {'present' if have else 'MISSING'}{'  ⚠ STALE → rebuild (sources changed)' if stale else ''}")
covp = os.path.join(D, "coverage.json")
try: cov = json.load(open(covp, encoding="utf-8"))
except Exception: cov = {}
for f in ("consistency-audit.deterministic", "consistency-audit", "citation-forensics",
          "baseline-comparison-audit", "experiment-forensics",
          "presentation-signals.deterministic", "presentation-signals",
          "proof-derivation-forensics", "eval-design-forensics",
          "ai-style-impressions.deterministic", "ai-style-impressions"):
    p = os.path.join(D, f + ".findings.json")
    skill = f.replace(".deterministic", "")
    # a findings file alone is NOT completion — the coverage entry decides.
    # (deterministic sub-files ride on their skill's coverage key)
    done = (os.path.isfile(p) and is_array(p)
            and cov.get(skill) in ("completed", "not_applicable"))
    print(f"STEP2 {f:<32}: {'ok' if done else 'todo'}")
print(f"STEP3 adversarial memo            : {'present' if os.path.isfile(os.path.join(D,'adversarial-case-builder.memo.md')) else 'todo'}")
print(f"STEP3 novelty advisory memo       : {'present' if os.path.isfile(os.path.join(D,'novelty-duplication-advisory.memo.md')) else 'todo'}")
print(f"STEP4 report.json : {'present' if os.path.isfile(os.path.join(D,'report.json')) else 'todo'}")
PY

Rule: a stale ledger forces a full rebuild (Step 1 → re-fan Step 2 → re-adjudicate) — stale findings anchored to an old ledger are worse than none. A full rebuild also deletes coverage.json (the Step-2 init re-creates it all-review_unavailable; stale completed entries must not survive a rebuild). If the ledger is fresh, skip Step 1 and only re-run the auditors marked todo. Always re-run Step 4 (cheap, deterministic, the only verdict source). Re-running from scratch is always safe — same inputs → same outputs.

---

Step 0 — Ingest: resolve the input to a working dir

Turn $ARGUMENTS (a directory, a PDF, or an arXiv id) into a single working directory (PAPER_DIR) that contains the best source the level allows. Prefer LaTeX (stable file:line spans → L1) over PDF text (best-effort → L0); leave any code/ + results/ in place so the manifest can derive L2.

ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
test -f "$ROOT/tools/adjudicate_findings.py" || { echo "FATAL: not inside the Anti-Autoresearch checkout (point ROOT at it)."; exit 1; }
ARG="$ARGUMENTS"

if [ -d "$ARG" ]; then                       # ---- DIRECTORY ----
  PAPER_DIR="$(cd "$ARG" && pwd)"
  # if only a PDF is present, extract text so the ledger has an L0 source.
  # Deterministic primary-PDF pick (issue #11) — never lexicographic-first: a dir
  # holding only figure/supplement PDFs yields NO pick (exit 1, honest) instead of
  # extracting a figure as the paper. Only exit 1 is tolerated.
  if ! ls "$PAPER_DIR"/*.tex >/dev/null 2>&1 && ! ls "$PAPER_DIR"/*.txt >/dev/null 2>&1; then
    P=$(python3 "$ROOT/tools/select_primary_pdf.py" "$PAPER_DIR"); rc=$?
    [ "$rc" -le 1 ] || { echo "FATAL: select_primary_pdf.py failed (rc=$rc)"; exit 1; }
    [ -n "$P" ] && pdftotext -layout "$P" "$PAPER_DIR/paper.txt"
  fi

elif [ -f "$ARG" ] && case "$ARG" in *.pdf) true;; *) false;; esac; then   # ---- PDF FILE ----
  PAPER_DIR="$(cd "$(dirname "$ARG")" && pwd)"
  pdftotext -layout "$ARG" "$PAPER_DIR/paper.txt" \
    || echo "WARN: pdftotext failed/missing — try: mutool draw -F txt, or pip install pdfminer.six (pdf2txt.py)."

else                                          # ---- ARXIV ID (e.g. 2401.01234) ----
  ID=$(printf '%s' "$ARG" | grep -oE '[0-9]{4}\.[0-9]{4,5}(v[0-9]+)?' | head -1)
  [ -n "$ID" ] || { echo "FATAL: '$ARG' is not a dir, a .pdf, or an arXiv id."; exit 1; }
  PAPER_DIR="$(pwd)/aar-$ID"; mkdir -p "$PAPER_DIR"
  # LaTeX source first (best spans → L1). e-print may be a tarball OR a single gzipped .tex.
  if curl -fsSL "https://arxiv.org/e-print/$ID" -o "$PAPER_DIR/src.tgz"; then
    tar -xzf "$PAPER_DIR/src.tgz" -C "$PAPER_DIR" 2>/dev/null \
      || gunzip -c "$PAPER_DIR/src.tgz" > "$PAPER_DIR/main.tex" 2>/dev/null
  fi
  if ! ls "$PAPER_DIR"/*.tex >/dev/null 2>&1; then     # fallback: PDF → text (L0)
    curl -fsSL "https://arxiv.org/pdf/$ID.pdf" -o "$PAPER_DIR/paper.pdf" \
      && pdftotext -layout "$PAPER_DIR/paper.pdf" "$PAPER_DIR/paper.txt"
  fi
fi

echo "PAPER_DIR = $PAPER_DIR"
ls -1 "$PAPER_DIR"/*.tex "$PAPER_DIR"/*.bib "$PAPER_DIR"/*.txt "$PAPER_DIR"/*.pdf 2>/dev/null
ls -d  "$PAPER_DIR"/code "$PAPER_DIR"/src "$PAPER_DIR"/results "$PAPER_DIR"/outputs 2>/dev/null   # L2 candidates

Validation gate. PAPER_DIR must now contain anchorable text — at least one *.tex or *.txt. A bare *.pdf is not enough: the ledger builders accept only --latex / --pdf-text, so Step 0 must have extracted paper.txt. If extraction failed or no source is present, stop and report exactly what was searched — there is nothing to anchor spans against, so there can be no ledger (see Failure handling below).

Failure handling.

• No pdftotext → fall back to mutool draw -F txt in.pdf out.txt or pip install pdfminer.six && pdf2txt.py in.pdf > out.txt. If text extraction is impossible, you cannot build an L0 ledger from a PDF — ask the user for the LaTeX.
• arXiv unreachable (network gated / proxy needed) → say so and ask the user to drop the source or PDF into a local dir, then re-invoke with that paper-dir. Never fabricate a ledger from an absent paper. (A heartbeat may wait on this download — a pre-verdict external step — but only the sweep, run once the source lands, produces the verdict.)
• arXiv .tex extracted but no obvious main.tex → fine; /evidence-ledger globs *.tex. (PAPER_ID is set authoritatively by the ledger in Step 1 — do not hand-pick it here.)

Step 1 — Build the evidence ledger (always first; the spine)

Delegate to /evidence-ledger, which runs the real deterministic tools (tools/build_manifest.py → artifact_manifest.json + observability level L; tools/build_claim_ledger.py → claims.json) and, by default, one additive span-anchored semantic-enrichment pass. Everything downstream reads these two files.

/evidence-ledger "<PAPER_DIR>"            # add "— enrich: false" to ship the deterministic backbone alone

Then read L and PAPER_ID back from the ledger — it is the source of truth for every later block (re-derive, never persist):

PAPER_DIR="<from Step 0>"
test -f "$PAPER_DIR/claims.json" || { echo "FATAL: /evidence-ledger did not produce claims.json — re-run Step 1."; exit 1; }
L=$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1]))["observability_level"])' "$PAPER_DIR/claims.json")
PAPER_ID=$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1]))["paper_id"])' "$PAPER_DIR/claims.json")
python3 - "$PAPER_DIR/claims.json" "$PAPER_DIR/artifact_manifest.json" <<'PY'
import json, sys, collections
L = json.load(open(sys.argv[1], encoding="utf-8")); M = json.load(open(sys.argv[2], encoding="utf-8"))
assert L["observability_level"] == M["observability_level"], "ledger level != manifest level"
t = collections.Counter(c["type"] for c in L["claims"])
print(f"PAPER_ID={L['paper_id']}  L={L['observability_level']}  claims={len(L['claims'])}  types={dict(t)}")
PY

Validation gate.

• claims.json exists and claims is non-empty. Zero claims on a paper that visibly has numbers/citations means the ingest source was wrong (e.g. a PDF with no usable text, or a dir with no .tex/.txt) → fix Step 0 and re-run /evidence-ledger.
• The ledger's observability_level matches the artifacts present (a PDF-only run MUST be L=0; never hand a higher level to anything because a repo "exists somewhere else"). The level is the honesty contract for the whole run — carry this L forward; it caps every finding's severity in Step 4.
• — human checkpoint: true → present L, the claim-type histogram, and the source list, and pause for confirmation before fanning out.

Failure handling. If /evidence-ledger is unavailable, build the ledger directly with the real tools (exact flags — confirm via --help):

ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd); PAPER_DIR="<from Step 0>"
SLUG=$(basename "$PAPER_DIR" | tr -cs 'A-Za-z0-9.' '-')
TXT=(); [ -f "$PAPER_DIR/paper.txt" ] && TXT=(--pdf-text "$PAPER_DIR/paper.txt")
python3 "$ROOT/tools/build_manifest.py" --paper-id "$SLUG" --dir "$PAPER_DIR" "${TXT[@]}" --out "$PAPER_DIR/artifact_manifest.json"
L=$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1]))["observability_level"])' "$PAPER_DIR/artifact_manifest.json")
if ls "$PAPER_DIR"/*.tex >/dev/null 2>&1; then
  python3 "$ROOT/tools/build_claim_ledger.py" --paper-id "$SLUG" --latex "$PAPER_DIR"/*.tex --observability-level "$L" --out "$PAPER_DIR/claims.json"
else
  python3 "$ROOT/tools/build_claim_ledger.py" --paper-id "$SLUG" --pdf-text "$PAPER_DIR/paper.txt" --observability-level "$L" --out "$PAPER_DIR/claims.json"
fi

Step 2 — Fan out the six auditors (breadth; the verdict is NOT decided here)

Decide which auditors apply from the ledger's claim types, then run each applicable skill. Each reads claims.json, opens a fresh codex thread per dimension (a different model family from the executor), proposes span-anchored findings, validates its own spans, and writes <skill>.findings.json (the deterministic auditors also write <skill>.deterministic.findings.json). The orchestrator only sequences these calls and enforces the Reviewer Calling Convention above — it authors no finding.

Resolved-pair provenance. Right after each dimension's reviewer call succeeds, export the pair that actually ran (export ARIS_RESOLVED_MODEL=... ARIS_RESOLVED_REASONING=... — the fallback pair if one fired); the validator blocks read these and crash if unset (references/reviewer-independence.md).

Coverage state machine (init FIRST, before any auditor runs). Initialize $PAPER_DIR/coverage.json with EVERY expected skill pre-marked review_unavailable, then let each skill's terminal branch overwrite its own key (completed on a finished sweep — including a genuinely empty []; not_applicable when its Step-1 gate says there is nothing to audit; leave review_unavailable when the reviewer died). A key that is never overwritten therefore reads as never ran — the fail-closed default. On re-entry, treat a missing key or review_unavailable as TODO (an existing [] findings file alone does NOT mean the dimension completed). A skill whose per-item sub-calls partially failed for good (e.g. one citation key still unparseable after the retry) also stays review_unavailable — a partial sweep must not read as a completed one.