Insights (Lineage & Metrics backbone)

Insights is the connective tissue that makes workflow runs legible. It turns the structured reports/manifests other tools already emit into a queryable lineage graph + common metrics store — the foundation for dashboards and an orchestrating agent. It does not replace any tool; it aggregates them.

Three-access pattern

Source of truth is the FastAPI service (npa/src/npa/workbench/insights/service.py). The CLI (npa/src/npa/cli/workbench/insights.py) and SDK (npa/src/npa/sdk/workbench/insights.py) are thin clients. Do not duplicate logic across layers.

Store layout

The store is an append-only index on S3 under a configurable prefix (--output-path), with a JSONL fallback so it works without any database:

• records.jsonl — metric records (npa.insights.metric_record.v1).
• edges.jsonl — lineage edges (npa.insights.lineage_edge.v1).
• records.d/ and edges.d/ — immutable append shards, one object per write.

Every append writes a new shard object; readers concatenate the base object (legacy stores) plus all shards. Never rewrite a whole JSONL object to append: object storage has no native append, so read-modify-write silently drops rows when two writers overlap (both read N, both write N + their own).

Readers expose a logically idempotent view: metric rows are deduplicated by run, source artifact URI, metric, stage/tool, canonical labels (including curve step), and lineage; lineage edges use their endpoint/version/relation/run identity. This also repairs legacy stores that already contain duplicate shards. Explicit emissions without an artifact URI retain their timestamp/value identity so distinct observations with the same metric name are not collapsed.

Do NOT introduce a database service or hardcode a metadata backend. Reuse the LanceDB tool as the optional query index (HTTP seam in integrations.py), exactly as dataset does; absence degrades to the JSONL scan.

Interfaces

CLI:

npa workbench insights record --input-path <records.json> --output-path <store>
npa workbench insights ingest-run --input-path <run-prefix> --output-path <store>
npa workbench insights query --input-path <store> --tool <t> --metric-name <m>
npa workbench insights lineage --input-path <store> --uri <artifact>
npa workbench insights compare --input-path <store> --base-run <a> --candidate-run <b>
npa workbench insights dashboard --input-path <store> --output-path <s3>
npa workbench insights status --input-path <store>
npa workbench insights system-info
npa workbench insights list

Endpoints: /health, /status, /system-info, /list, POST /record, POST /ingest-run, GET /lineage, GET /query, GET /compare, GET /dashboard.

API contract

• POST /record: append one or more metric emissions (+ lineage edges) keyed by run id + lineage refs. Rows validate against npa.insights.metric_record.v1.
• POST /ingest-run: non-invasive ingestion — scan an S3 run prefix for known schemas (npa.dataset.manifest.v1, npa.dataset.validation_report.v1, npa.scenario_gen.adversarial_set.v1, and gate/threshold decision JSONs), extract their metrics + provenance, and write them into the store. This does NOT require modifying the emitting tools.
• GET /lineage: traverse the provenance graph (ancestors + descendants) for an artifact/version, reconstructed from recorded lineage_edge records.
• GET /query: query metric records by facet (workflow, run id, tool, stage, dataset/model version, metric name, metric kind, cost basis, time range, threshold predicate). Cost records always label cost_basis as estimated or billed; only billing artifacts are authoritative for billed dollars.
• GET /compare: cross-run/cross-stack comparison; emits npa.insights.comparison.v1 (per-metric delta + regressed/improved flags). Metrics whose name looks failure-like (corruption/latency/loss/…) are treated as lower-is-better; override with --lower-is-better.
• GET /dashboard: return npa.insights.dashboard.v1 (grouped metrics + latest-run rollup) and optionally write a self-contained static HTML report to --output-path. Keep viz thin — JSON + a single-file HTML, no web UI.

Known evaluation reports use an explicit numeric taxonomy: score/quality fields become eval_score, step/sample/epoch values become counter, and latency or *_ms values become duration. Arbitrary numeric metadata is not promoted to a score. When root placeholders conflict with nested metrics or success_summary, the authoritative nested value wins deterministically.

Lineage

Thread and preserve lineage that already exists in upstream manifests (input URIs, dataset/checkpoint versions, parent versions, produced_from/derived_from/ evaluated_on relations). The whole point is cross-tool traceability — do not drop it.

GPU routing

CPU-only. Aggregation, query, comparison, lineage, and dashboard need no GPU and no rendering path (headless).

Workflows

SkyPilot stays the execution engine, but the shown catalog is npa.workflow-only (no raw SkyPilot task YAMLs). All insights pipelines are declarative npa.workflow/v0.0.1 specs, CPU-only and cloud: kubernetes:

• Aggregate a run (ingest-run -> dashboard): workflows/testing/insights-aggregate.yaml
• Hardening with insights (hardening stages -> ingest-run -> dashboard): workflows/testing/hardening-with-insights.yaml
• CPU-only smoke (ingest fixture -> compare -> dashboard): workflows/testing/insights-smoke.yaml

toolRefs: workbench.insights.record, workbench.insights.ingest_run, workbench.insights.compare, workbench.insights.dashboard.

Known issues

• Object storage has no native append, so each write lands in its own immutable shard under records.d/ / edges.d/ and reads concatenate base + shards. This is what makes concurrent ingests safe; a store is never rewritten in place.
• Re-ingesting a source prefix may append nothing (recorded_count: 0) when all logical observations already exist. Concurrent writers may still create duplicate immutable shards, but readers deduplicate them without losing distinct source observations or skewing dashboard means.
• Reader version skew: a reader older than sharding sees only the base object and silently reports a truncated store (e.g. an agent VM answering "no runs found" for runs that did ingest). Re-bootstrap deployed agents (npa agent bootstrap --project <alias> --name agent) after upgrading the store writers.
• compare needs both run ids present in the store; comparing a run to itself reports every metric as unchanged (useful as a smoke self-check). A compare that fails with no metrics recorded for base run right after a successful ingest means the base run's rows are missing from the store, not that the run never ingested — check for rows dropped by a writer that rewrote the object.
• A gpus metric only exists when an ingested npa.workflow.run.v1 manifest has a step whose resources_profile.accelerators parses to >= 1, and manifests with status planned are skipped. CPU-only and never-executed runs therefore report no GPU count at all rather than a fabricated zero.
• The service defaults to token authentication and an empty request storage scope. Set INSIGHTS_TOKEN, plus the narrow INSIGHTS_ALLOWED_S3_ROOTS and/or INSIGHTS_ALLOWED_LOCAL_ROOTS boundary. These allowlists apply to deployed FastAPI requests, not default embedded CLI, SDK, or workflow toolRef execution. INSIGHTS_AUTH_MODE=none is an explicit local/test service opt-in only; see docs/security/workbench-service-boundaries.md.