Update Dataset (PR → snapshot → steps → grapher)

Use this skill to run a complete dataset update with Claude Code subagents, keep a live progress checklist, and pause for user approval only when something needs attention.

Paired skill — keep in sync. /review-data-pr is the reviewer-side counterpart of this skill: it verifies the outcomes of the author-side steps defined here. Whenever you add, remove, or change a workflow step in this file, check whether review-data-pr/SKILL.md needs a matching reviewer-side check (and add it in the same commit if so). The reverse also holds — see the mirror note there. The creation-side skills /create-dataset and /create-snapshot belong to the same family: they point into this file's canonical sections (§5b-bis sanity bounds, §5c harmonization audit, §6b metadata quality, §6c metadata checklist + link verification, §6d scheduled issues, and the snapshot-related Guardrails), so when one of those sections changes, check whether the create skills need a matching edit in the same commit too.

The two announcements are owned elsewhere. /data-updates-comms owns the internal Slack form (step 9) and /data-update-announcement owns the public /latest post (step 9b). Both steps here are delegation stubs on purpose — put wording, format, and style guidance in those skills, never back into this file.

Inputs

• <namespace>/<old_version>/<name>
• Get <new_version> as today's date by running date -u +"%Y-%m-%d"
• A bare <short_name> (no namespace/version) is also valid — it's what owid-issues reminder bodies use. Resolve it to <namespace>/<old_version>/<short_name> via the DAG: rg "/<short_name>:?$" dag/ -g "*.yml" | grep -v "^dag/archive" — the :? matters because active entries are YAML keys ending in : (a $-anchored pattern without it only matches dependency lines), and archived entries must never be resolution targets. Take the latest active version; ask the user if the short name is ambiguous across namespaces. Several space-separated short names (/update-dataset <short_name1> <short_name2>) mean a grouped update of related datasets: run the full workflow for each, on one shared branch/PR.

Read the dataset's scheduled-issue body before running etl update (step 6d locates the update-*.yml workflow — peek at it now, not only at 6d): it often carries grouped-update instructions naming companion datasets that must bump together, e.g. a deflator ("US PPI New Office Construction also needs to be updated alongside to adjust for inflation"). Also check the target's garden DAG entry for same-vintage dependency datasets (a data://garden/... dep sharing the old version date is usually a companion). Seed etl update with every companion snapshot so the chains move in one run — updating only the main dataset leaves the derived series (e.g. inflation-adjusted) with a silent NaN tail for the new months.

Optional trailing args:

• branch: The working branch name (defaults to current branch)

Assumptions:

• All artifacts are written to workbench/<short_name>/.
• Persist progress to workbench/<short_name>/progress.md and update it after each step.
• Persist reusable update facts to workbench/<short_name>/update-context.yml as they are discovered. This is the canonical context artifact for the PR description, review handoff, and data-updates-comms.

Progress checklist (maintain, tick live, and persist to progress.md)

• Parse inputs and resolve: channel, namespace, version, short_name, old_version, branch
• Clean workbench directory: delete workbench/<short_name> unless continuing existing update
• Run ETL update workflow via etl-update subagent (help → dry run → approval → real run)
• Add yourself to dataset.owners in the new garden .meta.yml (don't reorder; preserve existing names and markers)
• Catalog # NOTE: / # TODO: comments carried over from the old step files into notes_to_check.md
• Detect any sanity_checks functions and their log-control flags; append to notes_to_check.md
• Create or reuse draft PR and work branch
• Update snapshot and compare to previous version; capture summary
• Meadow step: run + fix + diff + summarize
• Garden step: run + fix + diff + summarize
• Surface new indicators: check meadow + garden diffs (and snapshot columns when meadow hardcodes a column subset) for new-version columns absent from the old; warn user + PR body, flag new-but-dropped columns, route rename pairs to the step-7 mapping
• Review sanity_checks output (enable log flag, re-run, scan log, revert flag) — if none found and the garden step does non-trivial logic, recommend adding them; if present but missing value bounds (positive / [0,1] / [0,100] per indicator type), suggest those too (see 5b-bis)
• Country harmonization audit: validate .countries.json against canonical regions (flag provider regions not yet in the regions dataset → /add-provider-regions), audit .excluded_countries.json, scan garden log for missing/unused/unknown warnings
• Region-provider drift: if this dataset's aggregates are in regions.yml (defined_by: <provider>), check whether the new version changed the provider's region set or country membership; if so, update regions.yml and re-propagate via /add-provider-regions
• Grapher step: run + verify (skip diffs), or explicitly mark N/A
• Re-evaluate each catalogued # NOTE: / # TODO: against fresh data; delete resolved workarounds + comments together, or record status in PR body
• Check metadata: typos, Jinja spacing, style guide compliance
• Verify indicator-metadata coverage, dataset.update_period_days, snapshot DVC date_published and citation_full year (etl update copies both verbatim — bump to the producer's real release date / year, or to date_accessed / current year if the source doesn't publish one), and that all URLs resolve (HEAD-check) and every #fragment matches a real anchor in the target page (anchor pass, see 6c)
• Always suggest the optional adversarial data & metadata review (/adversarial-data-review) — verify metadata claims against the producer's fetched documentation and cross-check values against independent sources. Surfacing this offer to the user is mandatory every run (even when you recommend skipping it); only the run is opt-in — it's heavy (~25–45 web calls), so skip by default and run on user opt-in or visible red flags (see 6c-bis)
• Scheduled-issue workflow check (owid-issues): locate the dataset's update-*.yml (exact / fuzzy / group match), verify cron vs the observed release cadence + update_period_days, filename convention, and that the issue body says to run /update-dataset <short_name>; auto-fix body/title, ask before cron changes or new workflows — commits go straight to owid-issues main (see 6d)
• Commit, push, and update PR description
• Run indicator upgrade on staging and persist report
• Run the hardcoded-time-bounds audit (check-hardcoded-years) after all remaps — numeric minTime/maxTime/timelineMin/MaxTime/map.time pins on every surface carrying the new indicators (charts, MDim/explorer views, narrative charts, article time= links), graded against the new data's latest time; a pin below it means the update is invisible on that surface — propose "latest" fixes with user sign-off (see step 7)
• Always suggest the optional empty-entity audit (check-empty-entities) after all remaps — it sweeps every surface (charts, map columnSlug, MDim views, explorer views, narrative charts, gdoc country= references). Surfacing this offer to the user is mandatory every run (even when you recommend skipping it); only the run is opt-in — it can consume many tokens on widely-charted datasets, so run on user opt-in or when the remap touched many views; regressions vs production = fix, pre-existing = document (see step 7)
• Run the referencing-prose audit (standard) — read the prose of every surface the sweep found (articles, data insights, key-chart blurbs) for quantitative claims this update invalidates; an unbounded claim ("has grown 1,300-fold", "now accounts for X%") goes stale; a time-bounded one ("by late 2025 it had reached $62B") does not on an append-only update, but does if this update revised the period it names. Read the sweep's coverage gaps too — a "nothing stale" verdict covers only the surfaces swept. Hand the recomputed numbers to content — never edit published prose yourself (see step 7)
• Update update-context.yml with published chart count and 1–3 chart views for the public announcement
• Render Slack announcement via data-updates-comms, save to workbench, post @codex review as a separate PR comment, and notify user to post it to #data-updates-comms
• Draft the public-facing "Data update" post for OWID /latest via data-update-announcement (Mode A) — two versions, user picks, then the Google Doc in /Data updates, and hand the user the link (not added to the PR). The skill declines when we posted about this data less than six months ago; a declined post is a completed item — record the eligible date, don't override
• Address Codex review comments (fix valid ones + resolve all threads)
• Run downstream-dependency check (rg "<namespace>/<old_version>/<short_name>" dag/ -g "*.yml" | grep -v "^dag/archive"); for each consumer outside the dataset's own chain, decide with the user whether to bump in this PR or document under "Downstream dependencies" for a follow-up PR (see "Downstream dependency check" section below for details)
• Run the silent-breakage check whenever downstream consumers were repointed in this PR: confirm the buildkite/etl-automated-staging-environment PR check is green (red = a consumer crashed on staging, and the report under-reports until it's fixed; .venv/bin/etlr --modified --continue-on-failure is the optional local equivalent for small fan-outs), then triage the data-diff report — every red "− lost N data point(s)" entry in its Top-changes list and every 🔴-tier dataset (see "Silent-breakage check" section) and run the full-report audit probes (structural / World / raw-country / >30% / wipe-vs-edge per loss)
• Ask the user whether to remove the old version; if yes, remove+archive its DAG entries now and relocate the new entries into the old slot, but KEEP the old step files until review sign-off — the consecutive-version review diffs them from disk; deleting the files is the final commit before merge (see "Removing the old version & reordering the DAG") — don't forget this step
• Hand off the QA links to the user (Anomalist + Chart Diff on the staging branch, plus the data-diff report) — this is the final step

Persistence:

• After ticking each item, update workbench/<short_name>/progress.md with the current checklist state and a timestamp.
• Append every generalizable lesson to workbench/<short_name>/lessons.md the moment you hit it, not at the end. One entry per lesson: what you expected, what actually happened, and the general rule — plus which skill file should own it. Reconstructing these from memory after a long session loses the specifics that make a lesson usable (the exact column name, the threshold that separated signal from noise, the check that would have caught it earlier), and the ones worth keeping are exactly the ones that cost time mid-run. Fold them into the skills when the user asks, following feedback_skill_lessons_writeback conventions: phrase each as a general pattern with the incident as a trailing one-liner, and re-read the target skill first — a long-running branch can be several skill revisions behind master, so the section you remember may already have been rewritten by someone else's session.

Checkpoints — when to pause

Default: keep going. Run through the full workflow without stopping unless one of the conditions below is met.

Stop and ask the user when:

• A step fails and the fix is ambiguous (multiple reasonable approaches, or you're unsure of the correct one)
• Data structure changed significantly (columns removed/renamed, large row count drops, schema changes that may affect charts)
• Country harmonization has new unmatched countries that need manual decisions
• The snapshot requires a manual download or credentials you don't have
• Indicator upgrade had imperfect matches (< 100% similarity) that need human review
• Anything that could silently break charts or lose data

Don't stop for:

• Routine assertion count updates (just update them and note in the summary)
• Clean step runs with only row increases
• Expected warnings (SettingWithCopyWarning, known unmapped territories)
• Straightforward filename/version reference updates

When you do stop, present a concise summary of the issue and what options exist.

When the update isn't a drop-in version bump

Some updates carry structural changes that make the standard rename-only flow the wrong tool. Recognise them up front and adjust the workflow.

Triggers — any of these means you're in restructure territory, not a version bump:

• short_name changes (producer rebranded the dataset).
• File format/schema changes (wide → long, different file extension with a different column set, new dimensions).
• Policy/indicator set changes substantially (splits, dropped composites, newly added areas).
• Score semantics change (e.g. binary → continuous with subnational coverage).

Workflow adjustments:

1. Skip etl update. The rename-only flow copies the old step files into a new folder — useless when the schema is different. Author the new step chain by hand, using the old version as inspiration but not as a starting copy.
2. Add the new chain to the DAG before removing the old. Leave both chains active while you build and validate v2; remove the v1 entries only once v2 is on staging and the chart remap is queued or done.
3. Decide on naming convention upfront. Ask the user whether to preserve v1 short_names where they map cleanly, or to adopt the source's fresh naming scheme. Fresh naming is cleaner but means the auto-Indicator-Upgrader can't help.
4. Hand-curate the v1 → v2 indicator mapping. When short_names change entirely, the auto-upgrader has nothing to match on, but the Indicator Upgrader also matches on title — so if v2 titles are descriptive (full sentences rather than the bare short_name), you can hand the user a table of v1 title → v2 title pairs and they can drive the chart remap from there. Generate this table from the v1 meta.yml + the v2 grapher catalog.
5. Before remapping a chart onto a successor indicator, diff the chart's config against the new indicator's shape. Three things silently break on a v1 → v2 remap and none of them fails a build: (a) the chart's pinned selectedEntityNames may not exist in the successor (v1 steps often computed regional aggregates the producer doesn't publish — check the retired step's aggregation method via git show <archive-marker-sha>:<path> and rebuild the aggregates in garden the same way, e.g. population-weighted means); (b) pinned yAxis min/max sized to the old index's range can clip the new one; (c) the subtitle/FAUST may describe the old construction ("combines 18 policies") and become factually wrong. When the producer publishes several related successor indices, their codebook usually says which is the primary measure and which is a transparency baseline — cite that framing when picking the chart's indicator. (LGBTI: the v1 rights-index chart pinned World + six continents, an axis max of 13 against a successor peaking at 18.7, and a v1-specific subtitle — all three needed changes.)
6. Defer the Slack and /latest announcements until charts have been remapped. Both posts depend on charts.published_count and charts.selected_views from the v2 chain. Drafting them before the remap gives the wrong count (zero) and no representative views. Tell the user to ping you when the chart remap is done, then run steps 8 / 9 / 9b.

For the long-format with dimensions sub-case specifically (e.g. one row per (country, year, <dim1>, <dim2>)), use the modern OWID pattern:

• Meadow + garden: tb.format(["country", "year", <dim1>, <dim2>, ...], sort_columns=True).
• Aggregations: paths.regions.add_aggregates(tb, index_columns=[...full key...], regions=REGIONS, aggregations={...}).
• Grapher: pass long tables through unchanged; the framework auto-expands them into per-cell variables.
• Metadata: variables are keyed by the long-column name, with <% if <dim> == "X" and <dim2> == "Y" %>...<% endif %> Jinja blocks inside title, description_short, display.name. Grep this repo for tb.format(["country", "year" with more than two index entries to find current reference examples.
• Jinja coverage: after building the grapher dataset, verify every active (dim1, dim2) cell renders a non-empty value — read every column from the built grapher dataset and assert metadata.title is non-empty. A dimension combination with no matching <% if %> branch ships an untitled indicator.

Workflow orchestration

0. Initial setup

   • Check if workbench/<short_name>/progress.md exists to determine if continuing existing update
   • If starting fresh: delete workbench/<short_name> directory if it exists
   • Create fresh workbench/<short_name> directory for artifacts

1. Run ETL update command (etl-update subagent)

   • Inputs: <namespace>/<old_version>/<short_name> plus any required flags
   • Pick the URI that matches what's actually changing:
     • If the source data is changing — new source files, modified extractor, anything that affects the snapshot output — run from the snapshot URI with --include-usages. This bumps the whole chain (snapshot → meadow → garden → grapher) to the new version together:

       etl update snapshot://<ns>/<old_v>/<short>.<ext> --include-usages
       

     • Foundational / widely-used datasets (e.g. wb/*/income_groups, regions, population): add --direct-only. Plain --include-usages follows usages transitively and would try to version-bump every downstream consumer (income_groups has ~85 across 15 dag files). --direct-only restricts the bump to steps sharing the dataset's own namespace/version/short_name, i.e. just its chain. Caveat: --direct-only excludes sibling steps with a different short_name that belong to the same chain (e.g. income_groups_aggregations, which the grapher step also depends on) — pass those as extra seed steps so the grapher doesn't end up mixing a new-version garden with an old-version sibling. Dry-run and confirm the proposed set is exactly the chain before executing:

       etl update snapshot://<ns>/<old_v>/<short>.<ext> data://garden/<ns>/<old_v>/<sibling> --include-usages --direct-only --dry-run
       

     • If only garden logic / metadata is changing and the source data is unchanged, run from the garden URI. This bumps garden and grapher only; snapshot and meadow stay on the old version.
   • Either way, run etl update once. Don't call it separately per channel — that leaves stale version references in the DAG (e.g., new garden pointing to old meadow).
   • Perform help check, dry run, approval, then real execution; capture summary for later PR notes
   • After running, always verify the dag file: grep for the old version and confirm all internal references between the new steps point to the new version (e.g., garden depends on new meadow, not old meadow).
   • etl update writes the new entries in the flat DAG form — convert them to the nested (compact) form now, while you're in the file, rather than leaving it until archiving (otherwise the flat block tends to survive the whole update unnoticed). See the example and load_dag() parse-check under "DAG archiving & reordering" step 4.

1a-bis) Add yourself to dataset.owners in the new garden .meta.yml

You've just become a contributor to this dataset, so add your canonical OWID name to its owners: list. Don't reorder — keep the existing primary first; append yourself at the end. Skip if you're already there.

Your canonical name must match an entry in the schema enum (schemas/dataset-schema.json). Resolve it from git config user.name via etl.owners.resolve_owner; if that returns None, add a mapping in etl/owners.py and a row in the schema enum before continuing.

Edit the YAML in place, preserving comments and the existing # review / # backport / # fasttrack markers on other entries.

1b) Check for outdated practices (check-outdated-practices skill)

• After etl update creates new step files, run the /check-outdated-practices skill on every new step file — including helper modules that etl update doesn't generate but you copied by hand (e.g. *_omms.py), since those carry legacy patterns too
• The skill reads the extension as the source of truth for the full pattern set (the geo.add_* aggregation/population helpers are flagged, not just geo.harmonize_countries) — don't rely on a remembered subset
• Fix any findings before proceeding — this avoids propagating legacy patterns into new versions
• geo.harmonize_countries → paths.regions.harmonize_names is mechanical and safe. geo.add_regions_to_table → paths.regions.add_aggregates changes the aggregation core — prove equivalence with a controlled A/B test, not a diff against the old feather. Build the new garden both ways against the same current catalog (swap the call, rebuild, save output; revert, rebuild, save output) and diff the two. Do NOT conclude "the helper shifts aggregates across all years" from a new-vs-old-feather diff — that conflates the helper with upstream-dataset drift (see step 5). In practice the two helpers are equivalent bar tiny historical edge cases (e.g. one region-year's population residual); if so, modernize. add_aggregates also auto-resolves income groups from the DAG, so it's the right tool when you later need WB income-group aggregates.

1c) Catalog # NOTE: / # TODO: comments in the copied step files (don't resolve yet)

• Run rg -n "#\s*(NOTE|TODO|FIXME|HACK|XXX):" snapshots/<namespace>/<new_version>/ etl/steps/data/{meadow,garden,grapher}/<namespace>/<new_version>/.
• Filter out generic boilerplate (e.g. # NOTE: To learn more about the fields, hover over their names. at the top of .meta.yml).
• Save the remaining actionable items to workbench/<short_name>/notes_to_check.md — one entry per annotation, recording file path, line number, which step it lives in (meadow/garden/grapher), and what the workaround does.
• Don't act on them yet. Resolution requires fresh data and happens after each step's run — see step 6a.

1d) Detect sanity-check logic in the copied step files Sanity checks live in two different forms — detect both: