Curate Evidence and References

Use this workflow whenever evidence or its cited source changes.

Non-negotiable rules

• Quote an exact substring of the cited source. Do not paraphrase or fabricate a snippet.
• Confirm that the quote substantively supports the precise claim. A matching string from the wrong paper, or an unrelated sentence from the right paper, is not evidence.
• Prefer a result sentence from the abstract or authoritative source record. A paper title usually establishes only that a topic was studied.
• Never create or hand-edit references_cache/*.md. Generate or regenerate a cache entry with just fetch-reference <ID>.
• Never use fuzzy auto-repair to rewrite snippets. Read the source and copy the exact passage, choose another source, or remove the evidence.
• Treat deep-research output as leads, not ground truth.

Evidence shape

evidence:
  - reference: PMID:12345678
    supports: SUPPORT
    evidence_source: HUMAN_CLINICAL
    snippet: "Exact text copied from the cited source."
    explanation: "How this passage supports the specific KB claim."

Use supports: SUPPORT, REFUTE, or the value allowed by the schema. Make the explanation connect the quote to the claim without adding conclusions the quote does not establish.

Classify evidence_source by the cited study, not by the curator or the claim:

• HUMAN_CLINICAL: patients, cohorts, clinical observations, or trials
• MODEL_ORGANISM: in vivo non-human animal or organism work
• IN_VITRO: cells, organoids, explants, or biochemical assays
• COMPUTATIONAL: modeling, simulation, or in-silico analysis
• OTHER: evidence that does not fit the categories above

Inspect the schema and nearby current entries if a field or enum is uncertain.

Workflow

1. Screen deep-research sources

If evidence came from research/, first read the report's reference_validation, unresolved_references, needs_review, and off_topic_references results. Do not curate an unresolved identifier. An off-topic flag is a reason to inspect the paper, not an automatic rejection.

For an older report without validation output, run:

just validate-research-reference research/My_Disease-deep-research-falcon.md

The same recipes also check the report's ontology terms (term_validation in the frontmatter, ## Term Validation at the end of the body). That is a different check from the citations one and catches a different error: a report can have every citation verified and still name the wrong MONDO term for the disease. Never bind a CURIE listed under unresolved_terms. For an older report, just validate-research-terms <report> adds the section; see docs/deep-research-term-validation.md.

Before using any report content, check that the report describes the intended disease:

just preflight-dr research/My_Disease-deep-research-falcon.md MONDO:XXXXXXX

Interpret the result as follows:

• PASS: proceed to normal source, snippet, and term verification.
• WARN: resolve the reported conflict or degraded lookup, then manually check the causal gene, OMIM xref, and synonyms.
• FAIL: discard the report. Do not cherry-pick from it.
• SKIP: the automated check cannot discriminate; manually check disease identity before proceeding.

For a WARN or SKIP, inspect the intended MONDO record directly:

uv run runoak -i sqlite:obo:mondo info MONDO:XXXXXXX -O obo

Compare its causal-gene relationship (RO:0004003), OMIM xref, and synonyms with the report. Look specifically for synonym aliasing, eponymic collision, abbreviation ambiguity, or conflation with a closely related disease. On any identity mismatch, discard the report rather than cherry-picking from it.

See docs/deep-research-reference-validation.md and research/nec_risk_disease_classes.md for uncommon cases.

2. Fetch each new reference

just fetch-reference PMID:12345678

Use the actual identifier for other supported reference types. Read the fetched record and confirm its identity, topic, and quoted passage. A successful fetch does not prove that the source supports the claim.

3. Run the fast edit loop

After each disorder-file edit, run:

just validate kb/disorders/MyDisease.yaml
just count-verified-snippets kb/disorders/MyDisease.yaml
just validate-terms kb/disorders/MyDisease.yaml

All three commands accept the files supported by their recipes; batch files where practical. count-verified-snippets is fast and offline, but advisory. It reports missing cache entries and skipped prefixes rather than resolving them.

4. Run the authoritative pre-PR sweep

Once, after the tranche is complete, name every changed disorder file:

just validate-disorders \
  kb/disorders/FirstDisease.yaml \
  kb/disorders/SecondDisease.yaml

This batched command mirrors CI's schema, term, and reference checks and uses --no-full-text. It is the authoritative evidence gate for disorder files. Use just validate-kb-references <file> only when a non-disorder target or a full-text-permitting diagnostic requires it.

Never report a validation command as passing unless it finished and you read its output.

Resolve failures

When a snippet is not found:

1. Read the fetched source record.
2. Confirm the identifier belongs to the intended paper or record.
3. Copy an exact, substantively relevant passage.
4. If no such passage exists, cite a better source or remove the evidence.

If the claim is useful but no quotable evidence is available, move it to a notes field where appropriate, keep an unevidenced description only where the schema and curation policy permit it, or remove the claim. Never manufacture a quote to preserve an evidence block.

Total checks: 0 in reference-validator output means zero issues were counted; it does not mean no evidence was examined. Use the wrapper's affirmative Snippets checked: N/N verified summary to describe cache-backed coverage.

Reference prefixes in skip_prefixes within conf/reference_validator_config.yaml, including DOI:, are not snippet-checked by default, and the (N skipped by prefix) tail of the summary line says how many were passed over.

Skipped is not the same as uncheckable. Where the body is cached, as a DOI: reference's nearly always is, --unskip-prefix verifies it on demand:

just count-verified-snippets --unskip-prefix DOI kb/disorders/YourFile.yaml

The flag is repeatable. Use it before committing a DOI:-keyed snippet, since the gating validator will not check that quote for you. Treat a reference as genuinely unverified only when it is skipped and nothing cached backs it — a dataset accession, for instance.

Titles and brackets

Run just check-title-snippets when adding or repairing evidence. Quote a title only in the rare case that the title itself states a result; explain why it is probative. If the cached record has no abstract, cite the underlying study or a different source instead of treating a topic-shaped title as a finding. Do not manually regenerate tests/title_snippet_baseline.txt when fixing an existing title snippet.

Snippet matching applies literal_bracket_patterns from conf/reference_validator_config.yaml:

• all-caps abbreviations and spans containing a percent sign remain literal and must be quoted exactly;
• numeric citation markers and curator glosses are stripped before matching.

If a verbatim quote fails near brackets, read the reason printed by count-verified-snippets. Do not change the global patterns to accommodate one snippet without replaying validation across the KB.

Read the title off the cache, never from memory

reference_title (on an EvidenceItem) and title (on a top-level references: entry) name the paper you cited, and until #9138 nothing checked them. The failure mode that exposed is specific: correct PMID, verified snippet, invented title. Each gate reads a different field — linkml-validate confirms the slot is a string, count-verified-snippets and validate-kb-references check the snippet, validate-terms checks ontology terms, and check_title_snippets (despite the name) asks whether a snippet quotes a title. None of them reads the title.

On PR #9111 three of twenty (reference, reference_title) pairs named papers that do not exist. Two were written by an agent that had just verified the adjacent snippets as exact substrings of the cached text, then wrote the titles beside them from memory. Being rigorous about the quote and careless about the citation attached to it is a distinct failure mode, and these values are not inert — they render on the disorder page and flow into the cx2 and SEPIO exports.

The correct title is already on disk, in the reference's cache frontmatter:

head -5 references_cache/PMID_34081534.md
# ---
# reference_id: PMID:34081534
# title: Axonal Growth Abnormalities Underlying Ocular Cranial Nerve Disorders.

Copy it from there. just check-reference-titles gates new mismatches (offline, similarity-based, so punctuation, dashes, diacritics and source-XML markup do not trip it) and prints the cached title in the failure message, so the fix is a copy-paste. just list-reference-title-mismatches is the triage view; scripts/find_missing_reference_titles.py is the complementary check for absent titles.

Frequency claims

A phenotype frequency: value is a separate quantitative claim from the disease-phenotype association. Give it evidence that supports the frequency band or omit it. Follow docs/frequency-evidence-guidelines.md for acceptable quantitative, derived, qualitative, and clinical-estimate evidence.

Finding a cache file

A cache filename is the reference id with :, /, ? and = replaced by _, plus .md. The prefix keeps the identifier's own casing — it is not uppercased. PMID:29167994 caches as PMID_29167994.md, but clinicaltrials:NCT05813288 caches as clinicaltrials_NCT05813288.md, so a glob for CLINICALTRIALS_* finds nothing. Searching pmid_* on a case-sensitive filesystem is the mirror of the same mistake.

Do not hand-derive the path when a tool will do it. resolve_cache_path in src/dismech/reference_snippet_audit.py is the authority, and it also resolves a bare identifier (NCT06087757) back to its prefixed file. Prefer the recipes that take a reference id or a KB file:

just fetch-reference PMID:29167994     # fetch or regenerate the cache entry
just count-verified-snippets kb/disorders/Asthma.yaml

When you do need to glob, match case-insensitively and on the tail rather than guessing a prefix:

ls references_cache/ | grep -i "_29167994"

The prefixes in use, with the count of cached records at the time of writing — mixed case is normal and none of it is a typo:

Prefix	Cached	What it is
PMID	35586	PubMed
DOI	6056	DOI-only literature
clinicaltrials	1082	ClinicalTrials.gov (NCT)
NCIT	796	NCI Thesaurus predicate edges
ICEES	505	ICEES KG comorbidity pairs
CGGV	502	ClinGen gene-disease validity
ORPHA	347	Orphanet
GEO	176	Gene Expression Omnibus
url	116	Web page
MYGENESET	100	MyGeneset
STRCHIVE	73	STRchive
PPR	25	Europe PMC preprint
CIVIC	15	CIViC assertions and evidence
ICTRP	7	WHO ICTRP (non-NCT trial registries)
file	5	Local document
CGDS	3	ClinGen dosage sensitivity
METABOLIGHTS	2	MetaboLights
MGNIFY	1	MGnify

The list is a snapshot for orientation, not a closed set — a new structured source adds a prefix. Regenerate it with ls references_cache/ | sed 's/_.*//' | sort | uniq -c | sort -rn rather than trusting these counts.

Reference-cache integrity

Check the derived cache structure with:

just check-reference-cache-frontmatter

If an entry is malformed or incorrect, regenerate it with just fetch-reference <ID>; never patch its filename, frontmatter, or content.