LLM Wiki — Knowledge Distillation Pattern

You are maintaining a persistent, compounding knowledge base. The wiki is not a chatbot — it is a compiled artifact where knowledge is distilled once and kept current, not re-derived on every query.

Three-Layer Architecture

Layer 1: Raw Sources (immutable)

The user's original documents — articles, papers, notes, PDFs, conversation logs, bookmarks, and images (screenshots, whiteboard photos, diagrams, slide captures). These are never modified by the system. They live wherever the user keeps them (configured via OBSIDIAN_SOURCES_DIR in .env). Images are first-class sources: the ingest skills read them via the Read tool's vision support and treat their interpreted content as inferred unless it's verbatim transcribed text. Image ingestion requires a vision-capable model — models without vision support should skip image sources and report which files were skipped.

Think of raw sources as the "source code" — authoritative but hard to query directly.

Don't confuse this with the in-vault _raw/ staging folder, which is a different thing: a scratch inbox for quick captures and drafts awaiting promotion (see wiki-capture and wiki-ingest). Files there aren't Layer 1 sources, but wiki-ingest still moves rather than deletes them on promotion, since some have no other copy.

Layer 2: The Wiki (LLM-maintained)

A collection of interconnected Obsidian-compatible markdown files organized by category. This is the compiled knowledge — synthesized, cross-referenced, and navigable. Each page has:

• YAML frontmatter (title, category, tags, sources, timestamps)
• Obsidian [[wikilinks]] connecting related concepts
• Clear provenance — every claim traces back to a source

The wiki lives at the path configured via OBSIDIAN_VAULT_PATH in .env.

Layer 3: The Schema (this skill + config)

The rules governing how the wiki is structured — categories, conventions, page templates, and operational workflows. The schema tells the LLM how to maintain the wiki.

Wiki Organization

The vault has two levels of structure: categories (what kind of knowledge) and projects (where the knowledge came from).

Categories

Organize pages into these default categories (customizable in .env):

Category	Purpose	Example
concepts/	Ideas, theories, mental models	concepts/transformer-architecture.md
entities/	People, orgs, tools, projects	entities/andrej-karpathy.md
skills/	How-to knowledge, procedures	skills/fine-tuning-llms.md
references/	Summaries of specific sources; academic papers use the Paper Deep-Dive Template (below)	references/attention-is-all-you-need.md
synthesis/	Cross-cutting analysis across sources	synthesis/scaling-laws-debate.md
journal/	Timestamped observations, session logs	journal/2024-03-15.md

Projects

Knowledge often belongs to a specific project. The projects/ directory mirrors this:

$OBSIDIAN_VAULT_PATH/
├── projects/
│   ├── my-project/
│   │   ├── my-project.md      ← project overview (named after project)
│   │   ├── concepts/          ← project-scoped category pages
│   │   ├── skills/
│   │   └── ...
│   ├── another-project/
│   │   └── ...
│   └── side-project/
│       └── ...
├── concepts/                   ← global (cross-project) knowledge
├── entities/
├── skills/
└── ...

When knowledge is project-specific (a debugging technique that only applies to one codebase, a project-specific architecture decision), put it under projects/<project-name>/<category>/.

When knowledge is general (a concept like "React Server Components", a person like "Andrej Karpathy", a widely applicable skill), put it in the global category directory.

Cross-referencing: Project pages should [[wikilink]] to global pages and vice versa. A project's overview page should link to the key concept, skill, and entity pages relevant to that project — whether they live under the project or globally.

Naming rule: The project overview file must be named <project-name>.md, not _project.md. Obsidian's graph view uses the filename as the node label — _project.md makes every project appear as _project in the graph, making it unreadable. So projects/my-project/my-project.md, projects/another-project/another-project.md, etc.

Each project directory has an overview page structured like this:

---
title: >-
    My Project
category: project
tags: [ai, web, backend]
source_path: ~/.claude/projects/-Users-name-Documents-projects-my-project
created: 2026-03-01T00:00:00Z
updated: 2026-04-06T00:00:00Z
---

# My Project

One-paragraph summary of what this project is.

## Key Concepts
- [[concepts/some-api]] — used for core functionality
- [[projects/my-project/concepts/main-architecture]] — project-specific architecture

## Related
- [[entities/some-service]] — deployment platform

Special Files

Every wiki has these files at its root:

Write them with obsidian-wiki memory, never by hand. index.md, log.md, hot.md, and the _meta/ tables share one advisory lock and are written atomically; hand edits in a parallel run drop whichever write lands second. obsidian-wiki memory sync <VERB> key=value does all three in one call. The full procedure — verbs, the Key Takeaways slot that stays yours, the owner profile and todo index — is in references/MEMORY.md.

index.md

A content-oriented catalog organized by category. Each entry has a one-line summary and tags. Rebuild this after every ingest operation. Format:

# Wiki Index

## Concepts
- [[transformer-architecture]] — The dominant architecture for sequence modeling ( #ml #architecture)
- [[attention-mechanism]] — Core building block of transformers ( #ml #fundamentals)

## Entities
- [[andrej-karpathy]] — AI researcher, educator, former Tesla AI director ( #person #ml)

Format rule: Add a space after the opening ( and tags. ❌ Don't: description (#tag) — breaks tag parsing ✅ Do: description ( #tag) — proper spacing and tag parsing

log.md

Chronological append-only record tracking every operation. Each entry is parseable:

## Log

- [2024-03-15T10:30:00Z] INGEST source="papers/attention.pdf" pages_updated=12 pages_created=3
- [2024-03-15T11:00:00Z] QUERY query="How do transformers handle long sequences?" result_pages=4
- [2024-03-16T09:00:00Z] LINT issues_found=2 orphans=1 contradictions=1
- [2024-03-17T10:00:00Z] ARCHIVE reason="rebuild" pages=87 destination="_archives/..."
- [2024-03-17T10:05:00Z] REBUILD archived_to="_archives/..." previous_pages=87

.manifest.json

Tracks every source file that has been ingested — path, timestamps, what wiki pages it produced. This is the backbone of the delta system. See the wiki-status skill for the full schema.

The manifest enables:

• Delta computation — what's new or modified since last ingest
• Append mode — only process the delta, not everything
• Audit — which source produced which wiki page
• Staleness detection — source changed but wiki page hasn't been updated

Source key contract (v2). Source keys — the sources keys in .manifest.json, the sources: frontmatter values on pages, and a project's source_repo — MUST be machine-portable. A vault is synced across machines, so a bare absolute path (/Users/..., /home/...) is never a valid stored key. This is the single canonical definition; other skills reference it rather than restating it.

Where the source lives	Canonical key form	Example
Inside the vault	vault-relative path — POSIX separators, no leading ./, no ..	Raw/database/postgres.pdf, Clippings/article.md
Under $HOME	home-relative path — starts with ~	~/.claude/projects/-Users-name-my-app/abc.jsonl
Not a file at all	pseudo-key — any scheme:/:// identifier, treated as opaque	url:https://example.com/article, agent:claude/<session-id>

Rules:

1. Never store a bare absolute path. Convert before writing, not after.
2. Normalize before comparing. Expand ~ and environment variables, resolve vault-relative keys against the vault root, and treat scheme:/:// pseudo-keys as opaque identifiers. Never compare raw strings without normalizing first.
3. Identity survives path changes. The same logical source keeps the same key across machines.
4. Pseudo-keys are an open namespace. What makes a key a pseudo-key is its shape (scheme: or ://, so it can never be mistaken for a file path), not a fixed list of names. Recommended names: repo:<host/owner/name> for a git project, url:<canonical-url> for a web page, agent:<agent>/<id> for an agent session. A source that is neither in the vault nor under $HOME still needs one — do not let it fall back to an absolute path.
5. Project identity is a repository, not a checkout. In the projects block, identify a project by source_repo (host/owner/name) rather than a machine path. A machine-specific checkout location, if useful at all, belongs in an optional source_cwd_hint (~-relative), never in the identity.

Reading is backward compatible: an existing manifest full of absolute keys keeps working, and scripts/manifest.py migrate <vault> --dry-run converts it to contract v2 (merging collisions, keeping the newest ingested_at). If the vault has moved between machines, its absolute keys are rooted at the old vault path, which matches neither the new vault nor $HOME — pass that old root explicitly with migrate <vault> --from-root <old-vault-root> (repeat the flag if the vault lived at more than one location). The command then reports nothing portable to write — N key(s) kept non-portable rather than claiming success. New writes go through the same normalization, so a skill may pass an absolute path to obsidian-wiki cache-update and still have a portable key land in the manifest.

Recording provenance. When you write a manifest entry, populate pages_created and pages_updated with the vault-relative page paths that source contributed to. This is what makes re-ingestion (when a source changes) able to find the pages to revisit, instead of guessing.

Page Template

When creating a new wiki page, use this structure:

---
title: >-
    Page Title
category: concepts
tags: [ml, architecture]
aliases: [alternate name]
relationships:
  - target: "[[concepts/related-concept]]"
    type: extends
sources: [papers/attention.pdf]
summary: >-
    One or two sentences, ≤200 chars, so a reader (or another skill) can preview this page without opening it.
provenance:
  extracted: 0.72
  inferred: 0.25
  ambiguous: 0.03
base_confidence: 0.65
lifecycle: draft
lifecycle_changed: 2024-03-15
tier: supporting
created: 2024-03-15T10:30:00Z
updated: 2024-03-15T10:30:00Z
# Optional. Written only by `obsidian-wiki snapshots set` / `apply` — do not hand-author.
# Quoted wikilinks with |title: Obsidian Properties does not treat Markdown [text](path) as links.
snapshots:
  - "[[_raw/_archived/example-clip|example-clip]]"
---

# Page Title

One-paragraph summary of what this page covers.

## Key Ideas

- The source's central claim, paraphrased directly.
- A generalization the source implies but doesn't state outright. ^[inferred]
- A figure two sources disagree on. ^[ambiguous]

Use [[wikilinks]] to connect to related pages.

## Open Questions

Things that are unresolved or need more sources.

## Sources

- [[_raw/_archived/example-clip.md]] — snapshot this page was distilled from

Sources section (required, last body section). Every wiki page ends with ## Sources. Entries must be clickable in Obsidian:

• Local snapshot (raw ingest, dropped PDFs/images, Web Clipper files, anything that landed in _raw/ and was archived): [[_raw/_archived/<filename>]] in the body Sources section (body wikilinks may include .md). YAML sources: stays origin keys (url:, agent:, repo paths, …), not the archive path. YAML snapshots: is separate: after moving a file to _raw/_archived/, run obsidian-wiki snapshots set <page> --archive _raw/_archived/<filename> then obsidian-wiki cache-update on that archived path. The CLI writes a List of quoted wikilinks with display text, e.g. "[[_raw/_archived/clip|clip]]" (no .md in the target; |clip is what Properties shows). Do not put Markdown [title](path) in snapshots: — Properties leaves those as unclickable text. The snapshots CLI does not touch the body. Do not link the webpage recorded in clipping frontmatter — that URL is mutable origin metadata.
• Fetched URL (/ingest-url with no saved snapshot): a markdown link to the canonical URL, and YAML sources: as url:<canonical-url>.
• Do not mix those up. A clip of a page is not an ingest-from-URL.

Related wiki pages stay in Related / relationships:, not in Sources.

Parser-safe scalars. Write free-text frontmatter values — at minimum title and summary — with folded scalar syntax (>-) as shown above: a bare scalar containing : (colon + space), #, or quotes breaks YAML parsing, and Obsidian then reports "Invalid properties" and hides the frontmatter. Keep the value indented on the line(s) following title: >- / summary: >-.

Paper Deep-Dive Template

The generic template suits most sources. Academic papers are the exception. For ML/AI/LLM/VLM (and similar) papers landing in references/, the substance lives in the architecture, the equations, and the results table — exactly what a terse "Key Ideas" list flattens away. For these, use the richer template below. This is the one place where "compile, don't retrieve" yields to a thorough, self-contained walkthrough a reader could study instead of the paper.

Obsidian renders the needed primitives natively, so no extra tooling is required: Mermaid fenced diagrams, $$…$$ LaTeX (MathJax), markdown tables, and ![[image]] / ![[paper.pdf#page=N]] embeds.

Use this template only when the source is an academic paper (arXiv/conference) with load-bearing figures or equations. Everything else uses the generic Page Template above. Frontmatter, provenance markers, confidence, lifecycle, and relationships: are unchanged — only the body sections differ.

---
# ...required frontmatter, same as the generic template; category: references...
---

# Paper Title

> [!tldr] One sentence: what's new, plus the headline result.

## Problem & Motivation

What's broken or missing that this paper addresses.

## Method / Architecture

Prose walkthrough. Embed the paper's real architecture figure as the primary
visual (see *Academic papers* in `wiki-ingest` for the PyMuPDF extraction recipe).
Fall back to a Mermaid flowchart only when no figure can be extracted.

![[attachments/<slug>-fig1.png]]
*Figure N (Author Year): one-line caption.*

## Key Equations

The 1–3 core equations as display math, not backtick code:

$$ \mathcal{L} = \mathbb{E}_{x}\!\left[-\log p_\theta(y \mid z)\right] $$

## Results

Headline numbers as a table, not a comma-separated blob — and embed a key
results/motivating figure (scaling plot, benchmark chart, capability collage)
when the paper has one:

| Method | Benchmark | Metric | Cost |
|---|---|---|---|
| Baseline | … | … | … |
| **This paper** | … | … | … |

![[attachments/<slug>-resultsN.png]]
*Figure N (Author Year): one-line caption.*

## Limitations

What the paper concedes or sidesteps. Mark reading-between-the-lines as ^[inferred].

## Related

Typed `[[wikilinks]]` to neighbouring work.

## Sources

- [[_raw/_archived/paper.pdf]] — snapshot distilled (if a local PDF/clip was ingested)
- <https://arxiv.org/abs/XXXX.XXXXX> — only if this ingest fetched the URL and there is no local snapshot

A Mermaid diagram reconstructed from the paper's prose is a synthesis, not a transcription — treat it as ^[inferred] when the interpretation is non-trivial.

Provenance Markers

Every claim on a wiki page has one of three provenance states. Mark them inline so the reader (and future ingest passes) can tell signal from synthesis.

These are framework defaults. A vault's AGENTS.md may add markers or workflow flags. Preserve owner extensions and treat orthogonal workflow flags separately from the extracted/inferred/ambiguous truth-state axis.

State	Marker	Meaning
Extracted	(no marker — default)	A paraphrase of something a source actually says.
Inferred	^[inferred] suffix	An LLM-synthesized claim — a connection, generalization, or implication the source doesn't state directly.
Ambiguous	^[ambiguous] suffix	Sources disagree, or the source is unclear.

Example:

- Transformers parallelize across positions, unlike RNNs.
- This is why they scale better on modern hardware. ^[inferred]
- GPT-4 was trained on roughly 13T tokens. ^[ambiguous]

Why this syntax:

• ^[...] is footnote-adjacent in Obsidian — renders cleanly and never collides with [[wikilinks]].
• Inline (suffix) so a single bullet stays a single bullet.
• Default = extracted means existing pages without markers stay valid.

Frontmatter summary: Optionally surface the rough mix at the page level so the user can scan for speculation-heavy pages without reading them:

provenance:
  extracted: 0.72   # rough fraction of sentences/bullets with no marker
  inferred: 0.25
  ambiguous: 0.03

These are best-effort numbers written by the ingest skill at create/update time. wiki-lint recomputes them and flags drift. The block is optional — pages without it are treated as fully extracted by convention.

Typed Relationships

Plain [[wikilinks]] in page bodies carry no semantic weight — they indicate "related to" but not how. The optional relationships: frontmatter block adds typed, directional edges to the knowledge graph.

The relationships: block

relationships:
  - target: "[[Transformer Architecture]]"
    type: extends
  - target: "[[LSTM]]"
    type: contradicts
  - target: "[[Attention Mechanism]]"
    type: implements

Each entry has two required fields:

• target — a wikilink (using the same format as OBSIDIAN_LINK_FORMAT) to the related page
• type — one of the allowed semantic types below

Allowed relationship types

The table below is the framework default allowlist. A vault's AGENTS.md may extend it; consumers must use the effective allowlist and preserve owner semantics without coercion.

Type	Meaning	Example
extends	This page builds on or generalises the target	GPT extends Transformer Architecture
implements	This page is a concrete realisation of the target concept	BERT implements Masked Language Modelling
contradicts	This page's claims conflict with or refute the target	Evidence A contradicts Evidence B
derived_from	This page is based on or adapted from the target	Fine-tuning is derived from Transfer Learning
uses	This page depends on or relies on the target	RAG uses Vector Databases
replaces	This page supersedes or deprecates the target	GPT-4 replaces GPT-3
related_to	Catch-all: related but no stronger directional type applies	Concept A is related to Concept B

Rules

• Optional field — omit the block entirely if no typed relationships are known. Untagged wikilinks remain valid and are treated as related_to by wiki-export.
• Don't duplicate — if [[foo]] already appears as an inline wikilink, the relationships: entry just enriches it with a type; it is not a second link.
• Direction matters — the page declaring the entry is the source; target is the destination. Only declare relationships from this page's perspective.
• Don't fabricate — only add a typed entry when the source material makes the relationship direction and type clear. When in doubt, use related_to or omit.

Skills that read relationships:: wiki-export (emits typed edges), cross-linker (writes typed entries when inferring links), wiki-query (surfaces type in answers and walks the typed-edge graph for multi-hop "how is X connected to Y" path queries — bounded BFS over the relationships: adjacency, frontmatter-only).

Confidence and Lifecycle

Every page carries two orthogonal trust signals plus an optional supersession link.

The requiredness and lifecycle values below are framework defaults. A vault's AGENTS.md may extend lifecycle values or make trust fields optional. Validators must apply that effective owner schema while still validating any trust value that is present.

The deterministic lint/trust consumer accepts owner schema through OBSIDIAN_ALLOWED_LIFECYCLES, OBSIDIAN_ALLOWED_RELATIONSHIP_TYPES, OBSIDIAN_REQUIRED_TRUST_FIELDS, and OBSIDIAN_SCHEMA_SOURCE. Resolution precedence is CLI > environment/config > these framework defaults (with lifecycle and relationship extensions additive). Explicit blank or whitespace-only values fail closed; omit the variable to select defaults. wiki-lint/SKILL.md owns the operational invocation contract.

Required fields

base_confidence: 0.65          # [0.0, 1.0] — time-independent quality estimate. Stored once, recomputed on content change.
lifecycle: draft               # draft | reviewed | verified | disputed | archived
lifecycle_changed: 2024-03-15  # ISO date of last state transition
# lifecycle_reason: "..."      # optional free-text — why the state changed; surfaced by wiki-query
# superseded_by: "[[new-page]]" # wikilink; only when lifecycle=archived

lifecycle_reason and superseded_by are optional. Never fabricate them.

Confidence formula

The formula is a manual base score, not a deterministic URL classifier:

base_confidence = lineage_count_score * 0.5 + source_quality_score * 0.5

lineage_count_score  = min(independent_evidence_lineages / 3, 1.0)
source_quality_score = avg(reviewed quality score per independent lineage)

After calculating the raw score, assess whether the evidence covers the page's material claims. Partial coverage may justify keeping or lowering the score; unsupported material claims require source/claim repair before any confidence change. Avoid small score churn without meaningful epistemic change.

Source-quality scores (use the highest-matching bucket):

Bucket	Score	Examples
paper	1.0	arXiv, conference proceedings
official	0.9	*.gov, vendor docs
documentation	0.85	well-maintained third-party docs
book	0.8	books, technical references
repository	0.75	content-addressed repository/code evidence
blog	0.55	personal blogs
session_transcript	0.5	conversation history or completed operation
forum	0.4	Stack Overflow, HN, Reddit, issue-grade reports
unknown	0.4	catch-all/current config
llm_generated	0.3	LLM synthesis or unvalidated memory seed

An independent evidence lineage is an origin that can corroborate a claim independently. Canonical source IDs remain useful for identity, but identity alone does not prove independence. Collapse dependent evidence before counting:

• files, releases, and commits from one repository → one repository lineage;
• retry/review/fix tasks in one workstream → one task lineage;
• parent/child Kanban records → one task lineage;
• byte-identical memories across profiles → one memory lineage;
• a snapshot plus the mutable source it captures → one lineage;
• aliases or metadata references resolving to one origin → one lineage.

The deterministic wiki-lint path validates _meta/trust-ledger.json; it does not recompute confidence from source strings. New or materially changed pages are marked for manual review. Refresh the ledger only after explicit human approval.