Software and data provenance

Provenance answers the question every published number eventually faces: where did this come from, exactly? The complete answer names the input data (which version), the code (which commit), the parameters, the environment, who or what ran it, and when - for every derivation step from raw data to figure. This pack already records provenance in layers (seeds and configs in rseng-reproducibility, environment lockfiles, AI contributions in rseng-ai-declaration, file-level origin attributes in rseng-scientific-file-formats); this skill makes the capture systematic, standard and packaged.

Capture at run time, not from memory

Provenance reconstructed after the fact is testimony; provenance captured at run time is evidence. Instrument the pipeline so every run writes its own record:

• Per run, record automatically: input paths WITH checksums or dataset versions (rseng-data-management), the code identity (commit hash, dirty-tree flag), the resolved configuration (the actual parameters, not the defaults file), the environment (lockfile hash or container digest), seeds, start/ end times, and the executing agent - human, scheduler or AI (rseng-ai-declaration's runtime counterpart). For the code side, lean on git's own metadata rather than restating it: authorship is the commit author/committer fields and trailers, and a SIGNED release tag is a cryptographic provenance anchor (rseng-version-control-review) - reference the tag, do not copy names and dates into the manifest by hand.
• Write the record NEXT TO the outputs (a run manifest per results directory - JSON or YAML), so results and their origin travel together; a results file without its manifest is an orphan.
• Workflow engines do much of this for free: their run logs and hashes are provenance capture (rseng-workflows); the manifest distills what the engine knows into what a stranger can read.
• Chain the steps: each derived artifact's record names its direct inputs, so lineage from figure back to raw data is a walk, not an investigation - the data-flow diagram (rseng-data-management) is this chain drawn once.

Speak the standard: W3C PROV

PROV is the vocabulary that makes provenance interoperable - three concepts carry almost everything: Entities (data, files, results), Activities (runs, transformations) and Agents (people, software, organizations), linked by relations like wasGeneratedBy, used, wasDerivedFrom and wasAttributedTo. Use the model even in plain-JSON manifests (name fields after it), and emit PROV-O (the RDF/JSON-LD form) when tools or repositories consume it - handwritten triples are rarely needed; the mapping from a good run manifest is mechanical.

Package it: RO-Crate

RO-Crate packages a research object - data, code, workflow, results AND their provenance - as a directory with one ro-crate-metadata.json (schema.org-based JSON-LD, human-editable):

• Use it when results ship as a unit: replication packages (rseng-reproducibility's compendium gains machine-readable structure), workflow deposits (WorkflowHub speaks RO-Crate natively), archive deposits (rseng-archiving - a crate is exactly what a Zenodo deposit wants to be).
• The crate names each file's role (dataset, software, result), its origins (wasDerivedFrom chains), licenses per part (rseng-licensing) and identifiers (DOIs, ORCIDs - rseng-citation-metadata); profiles exist for common shapes (workflow runs, datasets).
• Start minimal - a crate with root metadata beats no crate; add detail where reuse demands it (rseng-fair-software's proportionality).

Provenance hygiene across the pack

• Embedded beats adjacent where formats allow: self-describing files carry their own origin attributes (rseng-scientific-file-formats); the manifest aggregates, not replaces, them.
• Honest gaps: when a step was manual or a record is missing, say so in the record ("digitized by hand from lab notebook, 2026-03") - a stated gap is provenance too; an invented link is corruption (rseng-honesty).
• Verify like everything else: a provenance spot-check - pick a published figure, walk its chain to raw data - belongs in the milestone review (rseng-code-review); a chain that breaks is a finding.
• AI in the loop is provenance: agent contributions to code, data transformations and documents are Agents in the PROV sense and belong in both the run records and aidecl.yaml (rseng-ai-declaration) - one practice, two granularities.

Working with this skill

This skill is source-independent: its authority is the W3C PROV specifications and the RO-Crate community standard linked below. It systematizes what rseng-reproducibility, rseng-data-management and rseng-ai-declaration record layer by layer.

Learn more (verified):

• https://www.w3.org/TR/prov-overview/ - W3C PROV overview
• https://www.w3.org/TR/prov-o/ - PROV-O ontology
• https://www.researchobject.org/ro-crate/ - RO-Crate
• https://workflowhub.eu - WorkflowHub (RO-Crate-native workflow registry)

Related skills

Check whether any of these applies before moving on:

• rseng-ai-declaration - agents are PROV Agents too
• rseng-archiving - RO-Crate is the deposit shape
• rseng-data-management - dataset versions and checksums in records
• rseng-honesty - stated gaps beat invented links
• rseng-reproducibility - same promise at a different layer
• rseng-scientific-file-formats - embedded origin metadata in files