Notebook engineering

Notebooks are where most research computing happens - and their superpower (interactive, stateful, out-of-order execution) is exactly their reproducibility hazard. The discipline is not "stop using notebooks"; it is knowing what notebooks are FOR (exploration, narrative, teaching) and applying engineering at the point where a notebook turns into infrastructure without anyone deciding it should. The CodeRefinery Jupyter lesson teaches the same split.

Hidden state: the core hazard

A notebook's on-screen code and its kernel's actual state can disagree: cells run out of order, deleted cells leave live variables, and a figure can come from code that no longer exists. Working rules:

• Restart-and-run-all IS the test: before trusting, committing or sharing any notebook, restart the kernel and run top to bottom. A notebook that only works in one execution order is broken - fix the order now, while you still remember it.
• Keep cells re-runnable: no cell should break when run twice (guard appends, recreations, cumulative mutations); avoid mutating a variable across distant cells - derive new names instead.
• Definitions up top: imports, parameters and configuration in the first cells; a parameters cell makes intent explicit and enables parameterization later.
• Watch execution counts: out-of-sequence numbers in a "final" notebook are the tell (rseng-version-control-review reviewers can see them too).

Version control that works

Raw .ipynb diffs are JSON noise with embedded outputs:

• Pair notebooks with a script/markdown twin via jupytext: the text twin is the reviewable, diffable, mergeable source; the .ipynb carries outputs locally.
• Decide the output policy explicitly: strip outputs before commit (clean diffs, smaller repos - a pre-commit hook automates it) OR commit executed notebooks deliberately as result records (rseng-reproducibility's compendium may want exactly that); mixing the two policies in one repo confuses everyone.
• Large embedded outputs (images, tables) are the notebook form of committed data - the same instincts apply (rseng-data-management).

Testing and automation

• Execution tests: run notebooks in CI and fail on errors (nbval-style pytest integration checks stored outputs too; plain execution via papermill or nbconvert catches rot). A notebook that no longer runs is the most common form of repository decay (rseng-ci-cd; budget-conscious: run on reduced data - rseng-green-computing).
• Parameterize with papermill: a tagged parameters cell turns a notebook into a callable with arguments - the honest bridge from "notebook" to "batch job" (sweeps via rseng-hpc-computing job arrays or rseng-workflows without abandoning the notebook form).
• Environments: a notebook without a pinned environment is a screenshot; keep the kernel's environment in the repository's lockfile (rseng-reproducible-environments) and Binder-ready when sharing is the goal (rseng-reproducibility).
• Seeds and data paths follow the same rules as any code (rseng-defensive-coding) - notebooks get no exemption from provenance.

The graduation path

Notebooks accumulate load-bearing code; the move is extraction, not rewrite:

• When a function is used twice, or another notebook copies it, extract it to a module in the repository and import it - the notebook keeps the narrative, the module gets tests (rseng-testing) and design (rseng-software-design). Autoreload during development keeps the loop tight.
• When a notebook IS the pipeline (runs nightly, feeds a paper), graduate the computation to scripts/workflow stages (rseng-workflows) and keep a thin notebook as the exploratory window onto results.
• The notebook that remains is the good kind: narrative analysis, figures with their reasoning, teaching material (rseng-trainer) - prose and code interleaved doing what neither does alone (rseng-science-communication for the storytelling layer).

Working with this skill

This skill is source-independent: its authority is the Jupyter ecosystem tooling and the CodeRefinery lesson linked below.

Learn more (verified):

• https://jupyter.org - Project Jupyter
• https://jupytext.readthedocs.io - jupytext notebook/text pairing
• https://github.com/computationalmodelling/nbval - nbval notebook validation
• https://papermill.readthedocs.io - papermill parameterization
• https://coderefinery.github.io/jupyter/ - CodeRefinery Jupyter lesson

Related skills

Check whether any of these applies before moving on:

• rseng-hpc-computing - papermill batch sweeps on clusters
• rseng-reproducibility - executed notebooks as deliberate result records
• rseng-science-communication - narrative layer of analysis notebooks
• rseng-scientific-visualization - figure-producing notebook cells
• rseng-trainer - notebooks as teaching material
• rseng-version-control-review - jupytext twins make diffs reviewable