/create-retrospective — comprehensive lane retrospective

Maps the search space already explored on a (model, lane) pair, identifies where coverage is incomplete, and ranks unexplored directions. Run this before declaring a lane exhausted — the 2026-05-21 case (-jax declared exhausted; tokamax-CE kernel +4.18 pp landed a week later) is exactly the failure this skill exists to prevent.

When to invoke

• The agent says "I'm out of ideas" / "all options exhausted" / "recommend pausing" — call this skill FIRST. The retrospective often surfaces a direction the agent forgot or hasn't tried.
• The frontier hasn't moved in 10+ experiments — search strategy may need to change, not just the next probe.
• N single-axis probes refuted on the same row with no kernel-level work tried — the imbalance pattern.
• Manual invocation for human review — /create-retrospective <model> jax produces the file; user reads + decides.
• NOT for routine iteration — full-read of every experiment page is expensive (~250K–500K tokens for a large lane). Call this when stuck, not on every cycle.

Cardinal rules

• Reads ALL experiments in the lane, not a sample. Coverage is the whole point.
• Single-lane primary, cross-lane brief default ON — the brief is a 5–10 line summary of sibling lanes' frontiers + Knobs-matrix universal levers untransferred to this lane. Full cross-lane experiment-by-experiment read requires explicit --cross-lane-deep.
• Does NOT act on its recommendations. The skill produces an analysis page; the loop / /formulate-hypothesis decides what to dispatch.
• Does NOT update the model page or hypothesis list beyond linking to itself.
• Does NOT auto-invoke /formulate-hypothesis — the agent reads the retrospective and decides.
• Idempotent + dated — each invocation writes a NEW dated file; old retrospectives stay as historical record.

Arguments

Required:

• <model> <lane> — positional. Example: /create-retrospective <model> jax.

Optional:

• --include-sibling-lanes (DEFAULT ON) — adds cross-lane brief at the bottom.
• --no-cross-lane — disables the cross-lane brief if the lane is genuinely isolated.
• --cross-lane-deep — adds full cross-lane experiment reads (expensive; rare).
• --cross-model — adds cross-model patterns (very rare; for newly-bootstrapped model architectures only).
• --variant <size>/<hardware> — restrict the retrospective to one variant row of the model page's matrix. Default: aggregate across all variants of the (model, lane).
• --since <YYYY-MM-DD> — restrict to experiments dated on/after the given date. Default: all-time on the lane.
• --full-redo — force FULL mode (re-read every experiment) even when a prior retrospective exists. Use when you suspect a prior categorization was wrong, when the taxonomy in model-optimization-index.md shifted in non-trivial ways, or after a SCHEMA update. Default: INCREMENTAL when a prior exists (see Step 1a).

Step 0 — KERNEL FAMILY? Apply these substitutions to every step below

Check first. <lane> == pallas, or <model> resolves to a wiki/kernels/<slug>.md family page ⇒ this is a kernel family, not a model lane. The steps below are written for model lanes; run them with these substitutions throughout. Everything else (the FULL/INCREMENTAL logic, the coverage analysis, the ranking) applies unchanged.

Model lane (as written below)	Kernel family (substitute)
Output wiki/analyses/<date>-<model>-<lane>-retrospective.md	wiki/kernel_experiments/<slug>/pallas/<YYYY-MM-DD>-retrospective[-N].md — co-located with the family's program.md / RESULTS.tsv / log.md (SCHEMA's placement exception; still type: analysis). Suffix -2, -3, … for repeats; the family's Stop rule counts these.
Per-lane log wiki/experiments/<model>_.../<lane>/log.md	the family's own log.md + RESULTS.tsv (one row per experiment)
Model page wiki/models/<model>-<lane>.md	family page wiki/kernels/<slug>.md (same type: model machinery: variant matrix keyed <op-point>/<hardware>, Current best, open hypotheses)
Topic taxonomy model-optimization-index.md	wiki/kernel-optimization-index.md signal→lever map + the routed class page's ## Levers (verified) list — the class page's lever list, not the topic buckets, is what coverage is computed against
Pallas kernel catalog	not applicable (this is a kernel)
Experiment pages wiki/experiments/<model>_.../<lane>/	wiki/kernel_experiments/<slug>/pallas/*.md
Prior retrospective: search wiki/analyses/	search the family dir for *-retrospective*.md
Cross-lane brief (default ON)	OFF — a kernel family has no sibling lanes
Bump wiki/index.md Analyses count + global wiki/log.md	append to the family's log.md; do NOT bump the global Analyses catalog (the page lives in the family dir, not wiki/analyses/)
Frontmatter model:/lane:	model: <slug>, lane: pallas

Two kernel-only outputs the model lane has no analogue for. wiki/kernel_experiments/program.md's Stop rule reads this file as its gate, so a kernel retrospective MUST carry:

1. The LEVER-COVERAGE TABLE — mandatory in a closing retrospective (Stop-rule condition 1's artifact) and worth producing in any, since the set-diff below IS the untried-direction computation this skill exists to do: one row per lever, the lever name verbatim from the routed class page's ## Levers (verified) bullets, each row exactly tried: v<NNN> + <receipt path> or ruled out: <why the lever's MECHANISM does not exist at this op-point>. It is audited as a set-diff against the class page — a missing lever row, a route-only reason (a lowering error or compile failure rules out that route, not the lever), or a tried-row whose receipt doesn't resolve is an automatic block. Levers the class page itself marks N/A for this op-point still get a one-word row.
2. If and only if this retrospective is the CLOSING one (an at-ceiling claim): the 4-row EVIDENCE TABLE, one row per Stop-rule condition, each citing the artifact that proves it — the lever-coverage table (1), the two retrospective paths (2 + 3), and the kgate verify --mode full receipt path (4). A missing row, or a row without a resolving artifact, means NOT at ceiling and the close is void on audit. Do not fabricate rows to complete the table: if an artifact does not exist, the honest output is a non-closing retrospective that names what is missing.

Step 1 — Resolve scope + read priors

Resolve <model> and <lane>. If either is missing, infer from CWD (per /start-experiment Step 1 conventions) or ask via AskUserQuestion. If this is a kernel family, apply Step 0's substitutions to everything that follows.

Then read in this order (cheapest first; deep reads come in Step 2):

1. Per-lane log — wiki/experiments/<model>_autoresearch_optimization/<lane>/log.md (one-line summaries; gives a chronological bird's-eye view of every iteration).
2. Per-model refuted-patterns.md — wiki/experiments/<model>_autoresearch_optimization/refuted-patterns.md (the cross-variant refuted-pattern record; cheap, dense).
3. Model page — wiki/models/<model>-<lane>.md. Extract: Variant matrix (Current best per variant + Open hyps + Retired hyps), Knobs translation matrix, Iteration ladder.
4. Topic taxonomy — wiki/model-optimization-index.md. Need the 16-ish topic buckets (Sharding, CE/softmax loss, MoE dispatch, Splash attention, Tensor parallelism, FSDP, VMEM, AC, Batch/seq, Pallas kernels, torch.compile/scan, HBM/IO, Hardware envelope, libtpu/XLA flags, etc.) and what each topic's catalog options are.
5. Pallas catalog — wiki/analyses/2026-04-23-pallas-kernel-directory.md. The list of every Pallas kernel + the function each implements. Will be cross-referenced against this lane's kernel ports.
6. Most recent existing retrospective for this (model, lane) — search wiki/analyses/ for *<model>*<lane>*retrospective*.md and read just the newest. Extract:
   • The retrospective's updated: date (or filename date)
   • The ## Experiment ledger table at the bottom (machine-readable; see Step 8)
   • The frontier MFU + variant + experiment v-ID recorded at the top
   • The recommendations the prior surfaced (so the new retrospective can note which were acted on, which remain open)

Step 1a — Decide FULL vs INCREMENTAL mode

The retrospective produces the same superseding output regardless of mode; only the read cost differs.

• No prior retrospective → FULL mode (read every experiment in Step 2a).
• Prior retrospective exists → default to INCREMENTAL (Step 2b: deep-read only NEW + UPDATED experiments; inherit categorizations for the rest from the prior's ledger).

Force FULL mode (even with a prior) if ANY of the following hold:

• Topic taxonomy shifted — wiki/model-optimization-index.md mtime is later than the prior retrospective's date. New topics may exist or old ones may have been renamed; categorizations need recomputing across all experiments.
• Pallas catalog updated — wiki/analyses/2026-04-23-pallas-kernel-directory.md mtime is later than the prior date. The 🔵 unexplored-kernel marks need recomputing across all experiments' kernel topic.
• Model page variant matrix changed — variant rows in wiki/models/<model>-<lane>.md's Variant matrix differ from the rows the prior retrospective covered (e.g., a new hardware row was added). Per-variant sections need expansion against fresh categorization.
• --full-redo flag passed — user explicit override.

Report the chosen mode in your status update before doing real work:

Mode: FULL (no prior retrospective)
Mode: INCREMENTAL (prior: 2026-05-25-<model>-jax-retrospective.md; +N new experiments since)
Mode: FULL (force-redo: taxonomy shifted — model-optimization-index.md updated 2026-06-01)
Mode: FULL (force-redo: --full-redo flag)

This lets the caller know what cost regime to expect (FULL ≈ 5-8 min on a 400-experiment lane; INCREMENTAL ≈ <2 min).

Step 2 — Read experiment pages

Step 2a — FULL mode

Walk wiki/experiments/<model>_autoresearch_optimization/<lane>/*.md. For each experiment page:

• Frontmatter: variant, hypothesis, commit, status, verdict, backfilled (if any).
• ## Hypothesis under test section: extract Mechanism + Predicted signal (if present, per the stub-first lifecycle).
• ## Results section: extract headline metric delta vs frontier.
• ## Verdict section: reasoning + Phase 3 hypothesis-firing audit result (CONFIRMED / SILENT NO-OP / PARTIAL).

This is the expensive step. For a lane with N experiments at ~5 KB/page, the read cost is N × 5 KB. Budget: 100 KB to 2 MB total depending on lane size. The skill explicitly accepts this — coverage is non-negotiable.

If --variant <size>/<hardware> was passed, filter to experiments whose frontmatter variant: field matches. If --since <YYYY-MM-DD> was passed, filter to experiments dated on/after.

The output of Step 2a is a fresh ledger: one row per experiment with (v-ID, date, variant, topic, verdict, frontier-shift). This ledger feeds Steps 3-7.

Step 2b — INCREMENTAL mode

Parse the prior retrospective's ## Experiment ledger table into rows (v-ID + date + variant + topic + verdict + frontier-shift). This is the inherited ledger.

List the current experiment dir. For each experiment page:

• NEW — v-ID is absent from the prior ledger. Must be deep-read.
• UPDATED — v-ID is present in the prior ledger BUT the experiment page's updated: frontmatter date is later than the prior retrospective's date. Must be deep-read. This catches verdict-assignment-after-the-fact (stub was filed before prior retrospective; verdict assigned after; or status: in_progress → filed flip happened post-prior).
• STABLE — present in the prior ledger, not updated since. INHERIT the prior row verbatim (no read).

Deep-read NEW + UPDATED using the per-page extraction logic from Step 2a (frontmatter + ## Hypothesis under test + ## Results + ## Verdict). Merge into the prior ledger:

• NEW rows append (any position; Step 7 sorts).
• UPDATED rows REPLACE the corresponding prior row by v-ID.
• STABLE rows unchanged.

If --variant / --since filters were passed, apply them to the MERGED ledger (so the filter is consistent regardless of mode).

The output of Step 2b is the same shape as Step 2a — a complete ledger, just produced with N_new + N_updated deep reads instead of N_total. Steps 3-7 are identical from here.

Step 3 — Categorize each experiment

Map each experiment to ONE primary topic from the model-optimization-index taxonomy. Method runs in waves; first match wins. Each wave is more expensive than the last; later waves only fire when earlier ones don't catch the experiment.

Wave 1 — Tag match (cheap)

If the experiment page's frontmatter tags: list contains a topic anchor (e.g., pallas-kernel, splash, fsdp, tokamax, scan, ac, vmem, bf16, gate-up-fusion, data-formatting, topology, etc.), use it.

Wave 2 — Slug keyword match (cheap)

Match slug substrings against a topic dictionary. This dictionary is intentionally broad and additive — if you encounter slugs that don't match an entry, ADD the entry to your in-memory dictionary as you go and surface it for the user (see Wave 5 + dictionary-growth note below). The starting set covers ~16 topics; expect to grow it 2–5 entries per retrospective on a mature lane.

Topic dictionary (extend freely):

• Pallas kernels — pallas|kernel|tokamax|mosaic|tpu-recipes|fused-rope|fused-rmsnorm|fused-qkv|splash-kernel|custom-call|hand-rolled
• Splash attention — splash|sa-block|sa_use|bkv|bq|sliding-window (attention kernel knobs)
• Op fusion / kernel fusion (general — NOT MoE-specific) — fusion|fuse-|coalesc|kernel-merge|epilogue|prologue|loop-fusion|input-fusion|output-fusion|convolution-fusion|elementwise-fusion
• Data transformation overhead — moveaxis|transpose|layout|reshape|spmd-copy|data-format|data_formatting|bitcast|copy-elimination|layout-canonicaliz|permute|view
• Sharding strategy — shard|sharding|gspmd|shardy|partir|mesh|named-sharding|partial-replication|p-spec
• FSDP / collective optimization — fsdp|all-gather|reduce-scatter|all-reduce|overlap|bucketing|async-collective|collective-matmul|host-offload|sparsecore-offload
• Tensor parallelism — tp|tensor-parallel|tensor-paralle|sequence-parallel
• MoE expert dispatch — moe|expert|dispatch|combine|gate|router|top-k|ragged-dot|segment-matmul
• Gate+up fusion (MoE-specific subtype) — gate-up|gate_up|up-gate|fused-gate|weight-fused|fused-weight (often together with MoE)
• torch.compile + scan — scan|compile|torch-compile|torch.compile|graph-trainer|aot|graph-break|dynamo|fullgraph
• Activation checkpointing — ac|checkpoint|remat|rematerialization|selective|ptd-checkpoint|memory-budget
• VMEM / scratch memory — vmem|scoped-vmem|scratch|smem|tile-size|block-size|vmem-budget
• Dimension alignment + dtype — bf16|fp8|fp16|dtype|precision|mxu-align|mixed-precision|reduce-dtype|param-dtype
• Batch-size + sequence amortization — lbs|batch|seq-len|seq_len|seq8k|seq4k|amortization|gradient-accumulation|microbatch|lbs8|lbs16
• HBM transient memory + I/O — hbm|oom|memory-pressure|defrag|peak-memory|cpu-offload|host-offload|spill|persistent-cache
• libtpu / XLA flags — libtpu|xla-flag|libtpu_init|enhanced-launch|barrier|collective-matmul|spmd-threshold|vmem-limit|tpu_use_|tpu_enable_
• Topology / hardware envelope — topology|v5p|v6e|v7x|multi-slice|single-slice|num-slices|cross-gen|cross-arch|chip-count
• Profile-driven attribution — xprof|profile-drilldown|op-profile|bucket-attribution|roofline|measurement (rare; usually adjunct to another topic)
• Checkpointing / persistence — orbax|checkpoint-save|checkpoint-load|async-save|persistence

Wave 3 — Hypothesis statement + Mechanism keyword match (medium cost)

If Waves 1–2 didn't catch it, scan the experiment page's ## Hypothesis under test section (the Hypothesis + Mechanism + Predicted signal labels) with the same dictionary. The hypothesis statement often names the mechanism even when the slug abbreviates it ("v294 splash-1024" slug + "tile sizing reduces kernel-boundary count" mechanism would both hit Splash attention, but slugs like "v167" or "v282" might only describe the change in the body, not the slug).

Wave 4 — Topic-inference from Predicted signal (medium cost)

If still uncategorized, look at the Predicted signal field. Many topics have signature predicted signals:

• "<topic> bucket drops from X to Y" → Profile-driven attribution + the named topic (categorize as the named topic if specific; fallback Profile-driven)
• "custom_call(target=) appears in HLO" → Pallas kernels
• "HLO module count drops" → torch.compile + scan
• "all-gather time drops" → FSDP / collective optimization
• "peak HBM drops" → HBM transient memory + I/O (or Activation checkpointing if AC-driven)
• "MXU utilization rises" → Dimension alignment + dtype (alignment) or Pallas kernels (custom compute)

Wave 5 — LLM inference + propose new topic (the expensive fallback)

If all keyword waves miss, the experiment is genuinely outside the current dictionary. Read the experiment page's Hypothesis + Mechanism + Predicted signal + Verdict reasoning, and infer a topic from the model-optimization-index's 16-topic taxonomy. If NONE of the 16 topics fits even loosely, propose a NEW topic name and flag in the retrospective:

### Uncategorized experiments + proposed topic additions

The following experiments don't map cleanly to any existing topic in model-optimization-index.md:

- v418 (<model>-tpu): tested torch_tpu.scan + register_autograd integration. Proposed new topic: **"Autograd-integrated structural rewrites"** — distinct from torch.compile + scan because the lever is operating BELOW Dynamo (C++/MLIR autograd hooks rather than compile-time graph capture).

- v391y (<model>-jax): tested chunked-XLA backward kernel. Proposed new topic: **"Compiler-side loss kernel rewriting"** — adjacent to Pallas kernels but the lever is XLA-pass-level rather than custom kernel.

→ Recommend updating wiki/model-optimization-index.md to include these topics with their own Mechanism subsections + Generic refuted-pattern principles.

These flagged experiments still get categorized into the closest existing topic for the tree (don't leave them dangling), but the retrospective surfaces the gap so the human curator can extend the index.

Wave 6 — Last-resort Uncategorized

Reserved for experiments where even Wave 5 inference can't find a topic (genuinely unintelligible hypothesis, or the page is too sparse). Categorize as Uncategorized; flag the v-IDs in the retrospective with a "data-quality" note.

Dictionary-growth discipline

Every retrospective run that hits Wave 2 misses (Waves 3–5 firing) should:

1. Note the misses — list which slugs/hypothesis-statements didn't catch on Wave 2
2. Propose dictionary additions — for each repeated miss (e.g., 3+ experiments using the same vocabulary), add the keyword to Wave 2's dictionary in your in-memory copy and mention the addition in the retrospective's "Methodology" subsection
3. Don't auto-edit this SKILL.md — surface the proposed additions; the human curator decides when to update the dictionary persistently

Per-experiment outcome: track per topic [supported_v_ids, refuted_v_ids, inconclusive_v_ids, invalid_v_ids, total_count, categorization_method ∈ {tag, slug, hypothesis, predicted-signal, llm-inference, uncategorized}]. The categorization_method field lets you report the dictionary coverage rate at the bottom of the retrospective ("78% of experiments caught at Wave 1-2; 18% at Wave 3-4; 4% required LLM inference"). Falling Wave 1-2 coverage is a signal the dictionary needs maintenance.

Step 4 — Build the mechanism tree

Render a markdown nested-list tree, one top-level bullet per topic (sorted by total_count desc), one nested bullet per concrete attempt or sub-topic. Use status icons:

• 🏆 frontier-shifting supported experiment
• ✅ supported (non-frontier)
• ❌ refuted
• ⚠️ inconclusive
• 💥 invalid
• 🔵 unexplored catalog option (cross-ref against model-optimization-index's catalog + Pallas catalog)

For each catalog option that has zero experiments in this lane, add a 🔵 bullet citing the catalog source. This is how "what's NOT been explored" surfaces inline in the tree.

Example shape:

- **Pallas kernels** (12 experiments)
  - tokamax CE — 🏆 v391f +4.18pp; 🏆 v391m +1.61pp cross-variant
  - tokamax segment_matmul — ❌ v391y -200× regression (silent-noop refuted)
  - 🔵 fused RMSNorm+RoPE+QKV — UNEXPLORED (catalog: tpu-recipes/maxtext-deepseek)
  - 🔵 NSA (Native Sparse Attention) — UNEXPLORED (catalog: paper 2025-02)
  - 🔵 ring-attention — UNEXPLORED (catalog: jax-ring-attention)
- **FSDP / sharding** (17 experiments)
  - ...

Step 5 — Detect premature-exhaustion signals

Compute and surface these signals prominently at the top of the retrospective (before the tree):

1. Single-axis-probe vs kernel-port ratio. Count topic-buckets:

   • "Single-axis probes" = libtpu/XLA flags + Dimension alignment + dtype + Batch/seq + VMEM (low-risk, single-knob)
   • "Kernel work" = Pallas kernels + torch.compile/scan (structural code changes)
   • Compute ratio. If single-axis : kernel ≥ 5:1 AND frontier-stuck-for-N ≥ 10 → flag.

2. Frontier-stuck duration. Sort supported experiments by date. Find the most recent verdict that moved the lane's Current best (per the model page's variant matrix). Count experiments since: if ≥ 10 → flag. The signal: search strategy needs to change, not the next probe.

3. Topic refutation density. For each topic, count refuted experiments. If any topic has ≥ 5 refuted AND no supported on the same variant → "the lane has comprehensively closed topic X; further experiments in this topic are anti-recommendations".

4. Coverage incompleteness. For each topic, compute (tried catalog options / total catalog options). Topics with < 40% coverage AND no supported result → "under-explored; not yet enough evidence to call closed".

These signals get a dedicated section at the top of the retrospective:

## Search-strategy assessment

⚠️ Single-axis-heavy: 17 flag/precision probes vs 2 kernel ports. Ratio 8.5:1 well above the 5:1 imbalance threshold.
⚠️ Frontier stuck for 14 experiments since v294 (2026-05-21 splash 1024 win).
✅ Sharding topic: well-explored (9 attempts, 1 supported, 8 refuted with diverse mechanisms) — calling this closed is justified.
🔵 Pallas kernels: under-explored (2 of 12 catalog options tried; 1 supported); coverage 17%.

**Inference**: lane likely has +3-5 pp upside in kernel work that single-axis probing won't surface.

Step 6 — Generate cross-lane brief (default ON)

If --no-cross-lane was NOT passed, add a brief that scans sibling lanes for transferable wins. Two-tier read (cheap first, opportunistic second):

Step 6a — Sibling model-page scan (always done if brief enabled)

For each other lane of the same model (e.g., if current is jax, scan tpu / torchax / maxtext):

1. Read wiki/models/<model>-<sibling>.md.
2. Extract:
   • Current best MFU per variant (one line summary)
   • Knobs translation matrix rows marked universal (win in every column)
3. Diff against current lane's Knobs matrix: which universal levers haven't been tried on this lane?

Step 6b — Sibling retrospective scan (opportunistic — only when sibling has one)

For each sibling lane, also search for an existing retrospective:

wiki/analyses/*<model>*<sibling-lane>*retrospective*.md

If found, read just the most recent. Extract three things: