Citation Forensics — are the references real and honestly used?

Audit citation integrity for: $ARGUMENTS (requires claims.json from /evidence-ledger; reasons over its type:"citation" claims). Emit span-anchored citation-forensics.findings.json. This skill computes no verdict.

🔒 Do not wrap this skill in /loop, /schedule, or CronCreate. It is verdict-bearing input — it proposes the citation findings the deterministic adjudicator turns into the report. Re-firing it on a wall-clock timer adds no signal: its output changes only when the paper / ledger / bibliography changes, not with the clock, and each run spends real cross-model + lookup budget per cited key. Schedule the external wait that precedes it — bibliography finalized → ledger rebuilt → audit once. (Mirrors ARIS's external-cadence doctrine.)

Adapted from ARIS citation-audit, re-wired onto this repo's evidence ledger and the reviewer≠adjudicator contract, and reframed from "audit + rewrite the bib" to "emit ledger-anchored findings, never touch the paper." Three layers, ported verbatim: existence → metadata → context. Following the repo's baseline-comparison-audit pattern, the executor gathers the canonical facts (DBLP / arXiv / DOI) as neutral evidence; a fresh cross-model reviewer judges existence + metadata + context over those facts. The reviewer never grades — the deterministic adjudicator does.

Why this exists

An autoresearch pipeline (or a rushed human) generates a bibliography in a separate pass from the prose and never reconciles the two. The failure modes are not wildly fake entries — those are easy to spot. The dangerous ones are:

• Hallucinated reference — no paper exists at the claimed arXiv id / DOI / venue; authors, title, or year are fabricated. (HP-CITE-HALLUC, critical)
• Metadata drift — a real paper cited with the wrong year, wrong venue (the arXiv preprint number used after the work appeared at CVPR/ICML/NeurIPS, or vice versa), or a v1 title silently merged with a v3 retitle. (HP-CITE-HALLUC, major)
• Wrong-context citation — a real paper used to support a claim it does not make, or argues against (e.g. citing a self-refinement paper to support "self-feedback yields correlated errors" when the cited paper argues the opposite). (HP-CITE-CONTEXT, major)

None of this needs the code, the data, or a re-run — only the citing sentence (from the ledger) checked against the cited work's public record (DBLP / arXiv / publisher). That is why this layer is L0-decidable (observability-wise — no repo or result files needed) and independently defensible. (The citing-sentence claims it anchors to still enter the ledger only via the LaTeX path; a pure PDF-text ledger yields none — see Step 0.)

Core principle

Ledger-anchored, span-verified, canonical-fact-checked-or-handed-off, reviewer≠adjudicator, detect-only.

1. The cite keys and the citing sentences come from the deterministic ledger (claims.json, type:"citation"), never from re-reading the PDF.
2. The executor assembles a neutral per-key dossier (citing spans + the claimed .bib metadata) and gathers a reproducible resolution snapshot of the canonical record (DBLP / arXiv / DOI). It gathers facts, never a judgment — whether a mismatch is fabrication, a typo, or a preprint→venue migration is the reviewer's call.
3. A fresh cross-model reviewer (gpt-5.6-sol xhigh, one thread per cited key) proposes findings, judging existence + metadata + context over those facts.
4. Every above-info finding cites a ledger claim_id + a verbatim span of the citing sentence (references/integrity-forensics-contract.md rules 1–2). The bib entry, the canonical record, and any URL live in the finding's description, never as the anchor span.
5. The model proposes; tools/adjudicate_findings.py decides (references/reviewer-independence.md Layer 2). This skill emits findings only.
6. Fact-or-hand-off honesty. Existence/metadata MUST be settled against a real source (snapshot record + URL in description). Cannot settle → verdict_local: needs_external_check — never a guessed "fabricated". A false hallucination flag is a serious error.

The anchor is the citing sentence, not the bib line. tools/build_claim_ledger.py puts type:"citation" claims in the ledger whose text_span is the sentence that contains \cite{key} and whose refs are the cite keys — it does not parse the .bib. So a citation finding always anchors to that citing sentence (quote a verbatim substring of it, e.g. \cite{smith2024bar} or the surrounding phrase). What the .bib claims and what DBLP/arXiv actually return go in description — there is no ledger claim for the .bib line to anchor to. A bib key that is in the .bib but never \cited has no citation claim, cannot be anchored, and is therefore out of scope (detect-only; not flagged).

How this differs from the other auditors (route correctly)

Auditor	Question it answers	Level
citation-forensics (this)	Do the cited papers exist, with correct metadata, and support the claim they are used for?	L0
consistency-audit	Does the paper contradict ITSELF / described method = evaluated method?	L0
baseline-comparison-audit	Are the right baselines present, tuned, and is "SOTA" earned?	L0 stated / L2 verified
experiment-forensics	Are the reported numbers what the code actually computes? (fake GT, self-norm, phantom)	L2
presentation-signals	Surface "AI-flavor" hints (auxiliary, surface-class)	L0
adversarial-case-builder	Strongest evidence-bound rejection memo (no verdict weight)	any

Do NOT raise here (hand off instead): numeric self-contradiction / method drift → consistency-audit; "first / SOTA / beats prior work" as an empirical claim → baseline-comparison-audit (emit needs_external_check); code/result-level fraud → experiment-forensics (needs L2); surface / AI-flavor of the prose → presentation-signals; the rejection memo → adversarial-case-builder. Stay in lane: this skill judges only whether the cited work exists, is described correctly, and supports the citing sentence — not whether the citing paper's own claim is true.

Constants & Reviewer Calling Convention

REVIEWER_MODEL       = gpt-5.6-sol                  # different family from executor (Claude)
REVIEWER_REASONING   = xhigh                    # always; effort never lowers reviewer quality
REVIEWER_SANDBOX     = read-only                # detect-only; never mutate the paper or .bib
REVIEWER_CWD         = <PAPER_DIR>              # so it can re-open claims.json + the .bib to confirm a span
THREAD_POLICY        = ONE fresh mcp__codex__codex per CITED KEY; NEVER mcp__codex__codex-reply across keys
TAXONOMY_VERSION     = 0.5                      # references/hack-pattern-taxonomy.md §E
PATTERNS             = HP-CITE-HALLUC (existence/metadata) | HP-CITE-CONTEXT (wrong context) | HP-CITE-RETRACTED (retracted/withdrawn)
OBS_REQUIRED         = 0 for all three patterns # decidable at L0 (text + canonical / retraction sources)
FACT_GATHERING       = executor, Step 2: DBLP MCP + WebSearch/WebFetch -> resolution.json (FACTS, never a verdict)
DOSSIER              = <PAPER_DIR>/.aris/citation-forensics/dossier.json      # Step 1 (rehashable; NOT /tmp)
RESOLUTION           = <PAPER_DIR>/.aris/citation-forensics/resolution.json   # Step 2 (rehashable; NOT /tmp)
FINDINGS             = <PAPER_DIR>/citation-forensics.findings.json           # Step 4 output
TRACE_POLICY         = forensic (never silently dropped)
TRACE_DIR            = <PAPER_DIR>/.aris/traces/citation-forensics/<YYYY-MM-DD>_run<NN>/

• Executor (Claude) builds none of the judgment: it pulls the citation claims from the ledger, assembles the per-key dossier, gathers the canonical resolution facts, validates the reviewer's spans, and writes the findings file. It never summarizes the paper, pre-judges "this reference is fake", or leaks an opinion into the prompt — only structured inputs (the dossier + the neutral resolution snapshot
  • the checklist). (reviewer-independence.md Layer 1.)
• Reviewer (codex / gpt-5.6-sol) judges existence + metadata + context per key over the executor's facts and self-reports false_positive_risk. It is the evidence-weigher, not the judge. Like the other auditors it runs config: {"model_reasoning_effort": "xhigh"}, sandbox: read-only; the lookups are the executor's job (Step 2), not the reviewer's.
• Fresh thread per cited key. codex-reply is intentionally absent from allowed-tools; never carry one key's conclusion into another (the bias guard).
• Detect-only. No Edit in allowed-tools — this skill never rewrites the .bib or any .tex (that is ARIS citation-audit's job, not a forensics tool's).

---

Step 0 — Preconditions: locate the ledger, read the level, find the bib, set the run dir

The ledger is the only structure this skill reasons over. Resolve it and read the run's observability level L, paper_id, the set of cited keys, and the bibliography path(s) (each Bash block is self-contained — shell state does not persist, so re-derive paths every time):

ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
# $ARGUMENTS is a paper-dir OR a claims.json path:
LEDGER="$ARGUMENTS"; [ -d "$LEDGER" ] && LEDGER="$LEDGER/claims.json"
# Only the NO-ARGUMENT case defaults to the CWD ledger. An EXPLICIT argument that
# resolves to a missing claims.json must NOT silently fall back to $(pwd) — that
# could audit the wrong paper; let the NO_LEDGER check below fire instead.
[ -z "$ARGUMENTS" ] && LEDGER="$(pwd)/claims.json"
python3 - "$LEDGER" <<'PY'
import json, sys, os, glob
p = sys.argv[1]
if not os.path.isfile(p):
    sys.exit("NO_LEDGER: claims.json not found. Run /evidence-ledger FIRST "
             "(it writes artifact_manifest.json + claims.json).")
d = json.load(open(p, encoding="utf-8"))
paper_dir = os.path.dirname(os.path.abspath(p)) or "."
cites = [c for c in d.get("claims", []) if c.get("type") == "citation"]
keys  = sorted({k for c in cites for k in (c.get("refs") or [])})
bibs  = glob.glob(os.path.join(paper_dir, "**", "*.bib"), recursive=True)
print("LEDGER      =", os.path.abspath(p))
print("PAPER_DIR   =", paper_dir)
print("PAPER_ID    =", d.get("paper_id", "?"))
print("RUN_LEVEL_L =", d.get("observability_level", 0))
print("CITE_CLAIMS =", len(cites))
print("CITE_KEYS   =", len(keys), keys)
print("BIB_FILES   =", bibs or "NONE (metadata layer will be partial)")
PY

Then create the run directories (start NN at 01, bump if it exists):

PAPER_DIR="<PAPER_DIR printed above>"
DATE=$(date -u +%Y-%m-%d)
TBASE="$PAPER_DIR/.aris/traces/citation-forensics"
NN=01; while [ -d "$TBASE/${DATE}_run$NN" ]; do NN=$(printf '%02d' $((10#$NN + 1))); done
RUN="${DATE}_run$NN"
mkdir -p "$TBASE/$RUN" "$PAPER_DIR/.aris/citation-forensics"
echo "RUN        = $RUN"
echo "TRACE_DIR  = $TBASE/$RUN"
echo "DOSSIER    = $PAPER_DIR/.aris/citation-forensics/dossier.json"
echo "RESOLUTION = $PAPER_DIR/.aris/citation-forensics/resolution.json"
echo "FINDINGS   = $PAPER_DIR/citation-forensics.findings.json"

Failure / edge handling.

• NO_LEDGER → stop; tell the user to run /evidence-ledger first. This skill never re-reads the raw PDF and invents structure (contract rule 1).
• CITE_CLAIMS = 0 → the ledger has no type:"citation" claims. This is expected on an L0 PDF-text-only run: extract_from_text in tools/build_claim_ledger.py extracts numbers + scope language but not citations (only the LaTeX path extract_from_latex emits citation claims). Skip to Step 4, write citation-forensics.findings.json = [], and note: "no citation claims in ledger — re-run /evidence-ledger with the LaTeX source, or the paper has no \cite." Silent skip is forbidden — the file must exist.
• BIB_FILES = NONE (or only a .bbl) → continue. The citing sentences still anchor the audit, and existence + context are checkable from the cite key + the sentence + canonical sources. The metadata layer is then partial (no claimed authors/year/venue to compare) — say so in the dossier and tell the reviewer the bib entry is unavailable for that key. That is honest, not a failure.

Carry the absolute LEDGER, PAPER_DIR, PAPER_ID, L, RUN, TRACE_DIR, DOSSIER, and RESOLUTION paths into the steps below.

Step 1 — Build the per-key citation dossier (deterministic; no judgment)

Group every type:"citation" claim by cite key (a \cite{a,b} claim contributes to both a and b) and, best-effort, attach the claimed .bib metadata for each key by balanced-brace extraction from any *.bib under the paper dir. Pure assembly — no web, no opinion. Stage it under .aris/ so the verifier can rehash it (never /tmp):

LEDGER="<abs LEDGER>"; PAPER_DIR="<abs PAPER_DIR>"
OUT="$PAPER_DIR/.aris/citation-forensics/dossier.json"
python3 - "$LEDGER" "$PAPER_DIR" "$OUT" <<'PY'
import json, os, re, sys, glob
ledger_path, paper_dir, out = sys.argv[1], sys.argv[2], sys.argv[3]
L = json.load(open(ledger_path, encoding="utf-8"))
cites = [c for c in L.get("claims", []) if c.get("type") == "citation"]

# 1) group citing spans by cite key
keys = {}
for c in cites:
    for k in (c.get("refs") or []):
        keys.setdefault(k, []).append({
            "claim_id": c["claim_id"],
            "span":     c.get("text_span", ""),
            "location": c.get("location", {}),
        })

# 2) best-effort: parse claimed metadata per key from any .bib (balanced-brace)
bibtext, bibfiles = "", []
for p in glob.glob(os.path.join(paper_dir, "**", "*.bib"), recursive=True):
    try:                                         # best-effort: skip an unreadable .bib
        txt = open(p, encoding="utf-8", errors="replace").read()
    except OSError:
        continue
    bibfiles.append(p)
    bibtext += txt + "\n"

def bib_entry(key):                          # @type{key, ... } with brace matching
    m = re.search(r"@\w+\s*\{\s*" + re.escape(key) + r"\s*,", bibtext)
    if not m:
        return None
    i, depth = m.end(), 1
    while i < len(bibtext) and depth:
        depth += (bibtext[i] == "{") - (bibtext[i] == "}")
        i += 1
    return bibtext[m.start():i][:1200]

entries = [{"key": k, "n_cites": len(keys[k]),
            "bib_entry": bib_entry(k), "citing": keys[k]} for k in sorted(keys)]
os.makedirs(os.path.dirname(out), exist_ok=True)
json.dump({"n_keys": len(entries), "bib_files": bibfiles, "entries": entries},
          open(out, "w", encoding="utf-8"), indent=2, ensure_ascii=False)
missing = [e["key"] for e in entries if not e["bib_entry"]]
print(f"dossier: {len(entries)} cited keys from {len(cites)} citation claims; "
      f"bib_files={bibfiles or 'NONE'}; no-bib-entry keys={missing} -> {out}")
PY

Gate — sanity. The printed cited keys count should match what you expect for the paper. 0 keys with CITE_CLAIMS > 0 (rare) means the refs were empty — spot-read a citation claim in claims.json. Keys flagged no-bib-entry (no .bib, or the key missing from it) are audited existence + context only — the metadata layer is partial because there is no claimed authors/year/venue to compare. That is honest; the reviewer infers the intended work from the cite key + the citing sentences.

Step 2 — Canonical-source resolution (executor gathers FACTS; non-blocking)

For each cited key, gather a reproducible snapshot of what the canonical record actually is — so a finding is reproducible against staged facts even if DBLP changes later, and so the reviewer has the evidence it needs to judge. This step records facts, never a verdict: whether a mismatch is fabrication, a typo, or a preprint→venue migration is the reviewer's call in Step 3. (Same executor-gathers / reviewer-weighs division as baseline-comparison-audit Step 2.)

For each key, seed the queries from the dossier's bib title / authors / venue. If bib_entry is null (no .bib, or only a .bbl), seed instead from the cite-key tokens (author / year / keyword) plus distinctive words from that key's citing sentence(s) in the dossier — existence + context stay checkable; the metadata layer is then partial (nothing claimed to compare against), which is honest, not a failure:

• DBLP fuzzy title — mcp__mcp-dblp__fuzzy_title_search with title=<bib title>, similarity_threshold=0.7 (lower to ~0.5 only if no hit). Returns canonical {title, authors, venue, year, doi, ee, url} for the top matches.
• DBLP boolean cross-check — mcp__mcp-dblp__search with query="<first-author surname> and <distinctive title word>" (the query supports only and/or, no parentheses). Confirms author↔title↔year coherence; venue_filter / year_from / year_to narrow it.
• Venue exists — mcp__mcp-dblp__get_venue_info with venue_name=<bib venue> to confirm the venue is real and of the claimed type (Conference or Workshop / Journal / Repository).
• arXiv / DOI resolve + content — WebFetch the arXiv abstract page built from the bib eprint/arxiv id (https://arxiv.org/abs/<id>) and capture the returned title + authors and the abstract text (the abstract is what lets the reviewer judge context); WebFetch https://doi.org/<doi> to confirm the DOI resolves to this work. Use WebSearch as a fallback for very recent papers (< ~2 weeks) not yet in DBLP.

Write one neutral record per key to the staged snapshot (use Write), and copy it into the trace dir (Step 5):

// .aris/citation-forensics/resolution.json
{
  "smith2024bar": {
    "dblp_fuzzy_top": [{"title": "...", "authors": ["..."], "venue": "...",
                        "year": 2024, "doi": "...", "url": "https://dblp.org/..."}],
    "venue_info": {"venue": "...", "type": "Conference or Workshop"},
    "arxiv": {"id": "2401.01234", "resolves": true, "title_returned": "...",
              "abstract": "... fetched abstract text, for the context layer ..."},
    "doi": {"doi": "10.1145/...", "resolves": true},
    "notes": "facts only — not a verdict"
  }
}

Failure handling (non-blocking). If a DBLP MCP call errors, or web access is unavailable, write {"<key>": {"status": "unavailable", "reason": "..."}} for that key and continue — the Step 3 reviewer then falls back to verdict_local: needs_external_check for that key rather than guessing. Note the skip in the trace. Never hand-author a "fabricated" conclusion from a failed lookup.

Step 3 — Per-entry cross-model audit (existence → metadata → context)

Existence and metadata are mechanical fact-checks against the snapshot; context needs judgment. Read dossier.json + resolution.json. For each entry, issue one fresh mcp__codex__codex call (key order; tag 001, 002, …), cwd = PAPER_DIR so the reviewer can re-open claims.json / the .bib to confirm a span. Inject that one key's dossier record and resolution record in place of the two bracketed blocks. Send EXACTLY this — it is the reviewer's complete instruction set; add no commentary of your own about the paper:

mcp__codex__codex:
  model: gpt-5.6-sol
  config: {"model_reasoning_effort": "xhigh"}
  sandbox: read-only
  cwd: <absolute PAPER_DIR from Step 0>
  prompt: |
    You are an integrity-forensics reviewer auditing ONE bibliographic citation key
    of a research paper. You judge three things and NOTHING ELSE: does the cited
    paper EXIST, is its METADATA correct, and does it actually SUPPORT the claim each
    citing sentence uses it for. You do NOT judge whether the citing paper's own
    result is true, and you NEVER accuse anyone of misconduct. You audit integrity,
    not authorship.

    Judge existence + metadata from the RESOLUTION SNAPSHOT below — canonical FACTS
    (DBLP / arXiv / DOI) gathered by the executor. It is evidence, NOT a verdict:
    cross-check the .bib's self-report against it. If the snapshot is "unavailable",
    or does not settle a key either way, say so and set verdict_local
    "needs_external_check"; do NOT guess existence, and NEVER fabricate the cited
    paper's contents. Judge CONTEXT only from the snapshot's abstract/title (the
    fetched record — never from memory of the cited paper); if the snapshot lacks
    enough of the cited paper's content to decide, set "needs_external_check" and tell
    the human which section of the cited paper to read.

    ## THE ENTRY UNDER AUDIT (from the evidence ledger; claims.json is in your cwd —
    ## you MAY re-open it to confirm a span is real, but you may NOT introduce a
    ## claim_id that is not listed here):
    [DOSSIER RECORD FOR THIS KEY — dossier.json["entries"][i]:
     {key, n_cites, bib_entry (claimed metadata; null if no .bib), citing:[{claim_id, span, location}]}]

    ## CANONICAL RESOLUTION gathered by the executor (FACTS; may be empty/unavailable):
    [RESOLUTION RECORD FOR THIS KEY — resolution.json["<key>"]]