industry-score — keep the competitive map complete and honest, and prove it

What this does. A buyer, a reviewer, or a skeptic asks one question the inward scorecards never answer: where does fak actually stand against the industry? The trap is to answer only with the comparisons fak happened to run — a highlight reel that silently omits the 90% of the field it has no number on. This pass refuses that. It starts from a researched taxonomy of the dimensions the LLM-serving / agent-infra field competes on (continuous batching, FP8/FP4 quant, speculative decoding, disaggregated prefill, multi-LoRA, tokens-per-watt, …), pins the current SOTA bar on each with a dated source, and positions fak honestly on every one — and for most, the honest answer is a named gap, not a win. It makes "our competitive map is complete and honest" a repeatable, provable pass instead of a prose claim that drifts, inflates, and goes stale.

The shape: run the scorecard → close the worst coverage gap or retire the worst parity-debt → keep the SOTA bars current (--stale) → verify sources → regenerate the doc folder → commit only the scorecard lane.

Two numbers are driven:

• coverage — of the industry dimensions that matter, how many the scorecard has considered and positioned fak against. Toward 100%. A dimension with no fak position row at all is coverage_debt.
• parity-debt — of the comparisons that exist, how many are dishonest, incomplete, or unsourced. Kept at 0.

It runs tools/industry_scorecard.py over the modular data directory tools/industry_scorecard.data/ (one concern per file) and folds into the unified tools/scorecard_control_pane.py via corpus.parity_debt.

---

The data directory (modular — one concern per file)

tools/industry_scorecard.data/
  _meta.json          meta (as_of, fak_version, freshness windows) + the category
                      vocabulary (id → doc group). A row/dim with an undeclared
                      category is MALFORMED — this is the data-defined closed vocab.
  _taxonomy.json      the INDUSTRY-FIRST dimension catalog: each dimension that
                      matters, its why-it-matters, the current SOTA bar, the leading
                      systems, and a source_url + source_date (the freshness anchor).
  _competitors.json   the SOTA-system registry, each with a last_reviewed date.
  rows-<group>.json   fak's position rows, grouped by doc group (serving, memory,
                      decoding, numerics, distributed, models, agent, cost,
                      operability, security). Each row's dim_id links to a taxonomy
                      dimension; most are honest no-claim gaps.

Edit a file, then re-run the tool. The generated doc folder docs/industry-scorecard/ (index + one page per group + the dimension catalog + the update process) is rebuilt by --markdown-dir; never hand-edit it.

---

The one rule that overrides everything: never invent a number or a win

This pass records evidence; it never manufactures it.

• No new fak number. Every fak figure must already exist in BENCHMARK-AUTHORITY.md (a commit + a committed artifact) or a results doc. If a benchmark hasn't been run, the honest row is a no-claim gap — that is a valid, complete position, and most dimensions are exactly this. Coverage rewards positioning a dimension honestly; it never rewards a guess.
• No new competitor number. Every SOTA bar must cite a real source_url + source_date (vendor docs, a paper, a leaderboard, an on-box bench).
• No win against a naive baseline. State the gain vs the tuned/SOTA system (competitor_class sota/tuned), never a re-prefill strawman (naive is refused).
• No regime is all-wins. fak's committed artifacts contain the losses (the single-stream CPU decode/prefill trails, the fak-gateway 0.75× of raw SGLang on concurrent throughput, the 7B model-size ceiling). They are first-class trails rows, not buried fences.
• Every verdict matches its ratio. lead needs fak clearly ahead; parity ~±5% or a published band; trails clearly behind; an oracle can only be parity; a ceiling can never be lead; a no-claim only on a genuinely unbuilt (stub/projected/in-flight) axis. A capability lead/loss with no clean ratio is marked qualitative: true and must carry a comparison_note.

If a fix would require changing a measured number or inventing a win, stop — that is out of scope. Re-measure through the benchmark-governance process first.

---

Step 1 — Run the scorecard (it builds both work-lists)

From the repo root:

python tools/industry_scorecard.py                 # the scorecard + standing + both work-lists
python tools/industry_scorecard.py --json          # machine payload (the loop / control-pane)
python tools/industry_scorecard.py --gaps          # the COVERAGE backlog (what to position / measure)
python tools/industry_scorecard.py --stale         # the INDUSTRY-DRIFT backlog (SOTA bars due a re-check)
python tools/industry_scorecard.py --verify-sources # fak numbers still match their committed artifacts

It grades the honesty KPIs (structure · completeness · honesty · traceability) into parity-debt, computes coverage of the taxonomy, blends them into a composite score (0–100, A–F), and prints the data-derived standing (▲ lead · ≈ parity · ▼ trails · ○ honest gap). It is read-only over the data; the only writes are the doc folder under --markdown-dir.

Step 2 — Score both the default and the reasonable coverage envelope

A single aggregate winner is not sufficient evidence of product coverage. For each concern, maintain two views:

• Default frontier: the strongest net-true industry default for the common supported case.
• Coverage frontier: named user/job/constraint cohorts for which another credible system offers a materially useful path that fak does not yet support.

Classify each coverage row with the /field-borrow bounded-superset dispositions: DEFAULT, OPTIONAL-MODULE, RECIPE, WATCH, or EXCLUDE. Count an optional capability as coverage only when its modular seam, opt-in behavior, witness, support cost, and owner/review trigger are explicit. Do not award points for a feature pile, stale compatibility, or an approach that is obviously superseded for every supported cohort. Equally, do not call fak complete merely because its default beats a competitor's default while that competitor serves a legitimate hardware, provider, privacy, topology, cost, compatibility, or operator-preference cohort that fak omits.

The scorecard is a bounded-superset map, not a boil-the-ocean mandate. Record out-of-scope, unsafe, license-incompatible, moment-in-time, and support-uneconomic capabilities as dated WATCH or EXCLUDE rows with a reconsideration condition rather than silently dropping them.

Step 3 — Pick the worst-first move

The scorecard names it. There are two failure modes and the verdict tells you which:

• coverage_debt > 0 → the map is incomplete. Run --gaps; for each unpositioned industry dimension add an honest fak position row to the right rows-<group>.json (linking dim_id to the taxonomy). Most will be no-claim gaps — that is the honest, complete answer, not a cop-out. Reuse an existing committed number only where a head-to-head genuinely exists.
• parity_debt > 0 → a row that exists is dishonest. Retire it worst-group-first using the table below.

Defect	KPI / group	The honest fix
Win vs a NAIVE baseline	baseline_sota / honesty	Restate the gain vs the tuned/SOTA alternative, or drop the row.
Verdict ≠ ratio (the overclaim)	verdict_consistency / honesty	Set the verdict the numbers imply. Oracle → parity; ceiling → never lead; no-claim only on an unbuilt axis.
Contracted regime missing / a loss hidden	axis_coverage / completeness	Add the regime's row — especially the trails losses the contract requires (single-stream, real head-to-heads where fak trails).
Blank competitor	competitor_named / completeness	Name the concrete SOTA / next-best system.
Non-comparable, undisclosed	apples_disclosed / honesty	Set apples_to_apples:false AND add a comparison_note stating what differs.
Measured verdict, no number	well_formed / structure	Add the fak/competitor numbers; or, for a real capability lead/loss with no ratio, set qualitative:true + a comparison_note.
Shipped but untraced	fak_traced / traceability	Add a fak_commit, fak_artifact, or fak_doc.
Competitor number unsourced	competitor_sourced / traceability	Add competitor_source (paper / vendor doc / on-box bench).

After a batch, re-run the scorecard and watch the number fall; that loop is the method.

Step 4 — Keep it current (the two update cadences)

This is what stops the scorecard rotting — in both directions.

As the industry moves. New techniques appear and published SOTA bars age.

• A new dimension the field starts competing on → add it to _taxonomy.json. It is immediately uncovered, so coverage drops and it shows in --gaps until fak is positioned on it. (This is the mechanism that surfaces "we're missing this.")
• A stale SOTA bar → --stale lists every dimension whose last_reviewed (or source_date for backward compatibility) is past the industry_review_window_days window. Re-check it on the web and update accordingly:
  • If the bar changed (new SOTA number or system): update sota_bar, sota_systems, notes, and set both source_date AND last_reviewed to the re-check date.
  • If the bar is still accurate (no new SOTA found): update ONLY last_reviewed to the re-check date (keep source_date unchanged). This is the "re-confirmed" case that prevents silent regrowth — the bar stays stale-checked-out without fabricating a new source date. Advisory, never parity-debt — a number doesn't become false the day it crosses the window, it wants a look. For a larger sweep, fan out a workflow of web-grounded researchers (one per group) to refresh the SOTA bars + sources on a recurring cadence.

As fak moves. A benchmark lands → turn the relevant no-claim into a measured lead/parity/trails, citing the commit/artifact, and bump measured_on. The freshness KPI flags fak numbers older than fresh_window_days (advisory) to re-confirm when a bench node is free.

Step 5 — Verify sources

Run python tools/industry_scorecard.py --verify-sources — no present artifact may MISMATCH (an absent one, on a bench node, skips; fine off-box). For any number you hand-transcribed, spot-check it against the cited doc (or have a subagent re-read it). A wrong number in an honesty instrument is self-defeating.

Step 6 — Prove the drop, regenerate the doc folder

python tools/industry_scorecard.py --json > /tmp/after.json
python tools/industry_scorecard.py --compare /tmp/before.json    # parity-debt + coverage delta
python tools/industry_scorecard.py --markdown-dir docs/industry-scorecard --stamp $(date +%F)

State the before/after (e.g. "coverage 21% → 100%; parity-debt 0; standing 3 lead · 11 parity · 8 trails · 65 honest gap"). Regenerate the committed doc folder from the data — never hand-edit a page. Optionally re-run python tools/scorecard_control_pane.py to confirm the portfolio still folds.

Step 7 — Commit only the scorecard lane, by explicit path

This is a shared trunk; commit your lane, never a peer's work:

fak sync reconcile --apply
fak commit \
  --path tools/industry_scorecard.py \
  --path tools/industry_scorecard_test.py \
  --path tools/industry_scorecard.data \
  --path docs/industry-scorecard \
  -F <msgfile> [--push]
fak sync push

• Stage by explicit path via fak commit --path, never git add -A. Stage and commit in one shell call.
• Data + generated-doc diff → a docs(benchmark): … or chore(scorecard): … subject, lead with a verb, end with the (fak tools) trailer. A code-effect prefix on a data/doc diff overclaims.
• On Windows, pass the message via a file (-F), not a here-string.
• If a peer's MERGE_HEAD is set, wait for it to clear; commit by explicit path.
• Stay on the trunk (main); push promptly via fak sync push (or --push).

---

Scope: what this pass touches, and what it must not

• In scope: the modular data dir tools/industry_scorecard.data/ (taxonomy, competitor registry, position rows) and the generated docs/industry-scorecard/ folder. Also tools/industry_scorecard.py + its test when a KPI needs tuning.
• Out of scope: the fak NUMBERS. Those are produced + committed through the benchmark-governance process (BENCHMARK-GOVERNANCE.md) → recorded in BENCHMARK-AUTHORITY.md. This pass cites them; it does not measure or edit them. CLAIMS.md is the capability ledger, not a scorecard surface.

When to run this

• After a new benchmark lands — turn a no-claim into a measured row.
• When a competitor ships a number that moves a SOTA bar (--stale finds the aging ones).
• When the field starts competing on a new dimension — add it to the taxonomy.
• On a /loop cadence to keep the front-door competitive map complete, honest, and current.

The scorecard is the competitive story's checking-layer, the way refresh-readme is the README's freshness layer and quality-score is the code's. Same discipline: numbers you can move, and prove you moved.