Computational Experiments

Lifecycle skill for algorithmic research projects where the code IS the scientific contribution.

Modes

Mode	What it does	Phases
Scaffold	Create/audit package structure + algorithm skeleton	1–2
Experiment	Design and run pre-specified sweep campaigns	1, 3–4
Explore	Adaptive experiment loop: modify → run → evaluate → keep/discard	1, 3E–4
Autonomous	Parallel self-correcting sweep with sub-agents	1, 3A–4
Figures	Generate publication output from results	1, 4
Full	Complete pipeline	1–5

Default: Full. Detect mode from user request or ask if ambiguous.

--scaffold Flag

Sets a stage progression template for the experiment campaign. Templates provide structured checklists and exit criteria for each stage.

Scaffold	Stages	Best for
standard	Init → Tune → Creative → Ablate	Algorithm development, ML, simulation
robustness	Main spec → Alternatives → Placebo → Sensitivity	Causal inference, econometrics
replication	Exact → Our data → Extensions → Robustness	Replicate-and-extend papers

Templates live in templates/experiments/. When a scaffold is active:

1. Present the current stage's checklist before starting work
2. Gate progression: don't move to the next stage until exit criteria are met
3. Log stage transitions in the experiment breadcrumb

Default (no flag): user-defined stages (current behavior). Scaffold is a guide, not a cage — users can skip stages or reorder with explicit acknowledgment.

--budget Flag

Sets a campaign-level time budget in minutes for the entire experiment run. When set:

1. Record start time at the beginning of Phase 3 (any variant)
2. Check remaining budget before launching each new config, sweep batch, or explore iteration
3. Soft stop when budget is exhausted: finish the current run, save all results collected so far, skip remaining configs
4. Never hard-kill a running experiment mid-execution — always let the current run complete

Mode	How budget applies
Experiment	Skip remaining sweep configs when time is up
Explore	Exit the explore loop at the next iteration boundary
Autonomous	Do not spawn new agent batches; let running batches finish
Figures	Budget does not apply (figure generation is fast)

Reporting: When a budget triggers early stop, append to the breadcrumb:

- **Budget:** Stopped after <N>/<M> configs (budget: <X> min, elapsed: <Y> min)

Default: No budget (run all configs to completion). Typical values: --budget 30 for quick exploration, --budget 120 for overnight sweeps.

Use Experiment when the design space is known upfront (grid/random sweep). Use Explore when the design space is unknown and the agent should adaptively search (inspired by Karpathy's autoresearch).

When to Use

• "Set up experiments for my algorithm" / "Scaffold the project"
• "Run the benchmark" / "Design the sweep"
• "Explore what auction parameters reduce collusion" / "Try different architectures"
• "Generate convergence plots" / "Make publication figures from results"
• Any project where algorithms, simulations, or optimisation loops are the contribution

When NOT to Use

• Empirical data analysis (observational/survey data) → data-analysis
• Experimental design for human subjects → experiment-design
• Causal inference strategy → causal-design
• LaTeX compilation → latex

Workflow

Phase 1: Detect & Configure

1. Read project context: CLAUDE.md, MEMORY.md, .context/project-recap.md if they exist
2. Detect structure: Scan for src/, experiments/, tests/, pyproject.toml, setup.py
3. Detect language: Check existing files. Default: Python with uv. Ask if ambiguous.
4. Read MEMORY.md: Look for [LEARN:code] tags, notation registry, key decisions — apply established conventions
5. Determine mode: From user request or ask. If existing package exists, skip scaffold.
6. Inventory existing code: Map modules, entry points, config objects, result files

Read references/package-scaffold.md if scaffold mode is active.

Phase 2: Scaffold

Skip if: Package structure already exists and user requested experiment/figures mode.

1. Package structure: Create src/<pkg>/, tests/, experiments/configs/, scripts/
2. Build config: pyproject.toml with hatchling, dev dependencies (pytest, matplotlib, numpy)
3. Algorithm skeleton: Base classes (Algorithm, Experiment, Metric) with # TODO: markers. For multi-agent projects, use BaseAgent, Environment, MultiAgentSimulation instead — see references/multi-agent-patterns.md
4. Test skeleton: Unit test stubs, convergence test template, smoke test
5. Experiment infrastructure: Config dataclass template, runner script template
6. Gitignore: Add results/, *.pkl, wandb/, __pycache__/

Read references/algorithm-templates.md for skeleton code. Read references/package-scaffold.md for directory layout.

Gate: Verify package installs with uv pip install -e ".[dev]" before proceeding.

Phase 3: Experiment Design

Prerequisite: Working package (Phase 2 or pre-existing).

1. Config schema: Python dataclass with validation, defaults, serialization to/from YAML
2. Config hashing: SHA-256 fingerprint for reproducibility tracking (see references/experiment-patterns.md)
3. Sweep definitions: Grid sweep (all combinations), random sweep (budget-limited), manual configs
4. Seed management: np.random.default_rng(seed) everywhere. Seeds passed through config, never global state.
5. Runner script: Config → initialize → loop → collect → save. Reads config, runs n_seeds repetitions, saves per-seed results.
6. Baseline implementations: Same interface as main algorithm, registered in config
7. Result aggregation: Per-seed CSV → aggregated stats (mean ± std). Canonical column naming.
8. Parallelization: concurrent.futures.ProcessPoolExecutor for independent seeds/configs. For large sweeps (10+ configs × 10+ seeds, GPU-bound, or >30-min runs): move to [HPC cluster] HPC — see docs/guides/hpc.md in Task Management and copy templates/slurm/{array,gpu}.sbatch into hpc/ with sync-up.sh / sync-down.sh. Recent reference implementations: Projects/NLP/{example-project-a,benchmark-gaming-llm-safety}/hpc/.
9. Checkpointing: Save intermediate results to allow resume on crash
10. Dual output: Dated archive + "latest" symlink for quick access (see references/experiment-patterns.md)

For multi-agent simulations, also include:

11. Agent composition: Heterogeneous population specs with distribution presets (uniform, weighted, custom)
12. Feature toggles: Opt-in capabilities per agent type (memory, messaging) — enables ablation studies
13. Messaging service: Round-based broadcast channel for inter-agent communication
14. Multi-level metrics: Agent-level (individual performance) + system-level (aggregate outcomes) + emergent-level (herding, convergence)

See references/multi-agent-patterns.md for all multi-agent patterns.

Read references/experiment-patterns.md for config, sweep, and runner patterns.

Gate: Run a smoke test — single config, single seed, verify output files are created.

Phase 3E: Explore Loop (Adaptive Experiments)

Use instead of Phase 3 when: The design space is unknown, the user wants to adaptively search rather than run a pre-specified sweep, or the user says "explore", "try things", "see what works".

Read references/explore-loop.md for the full protocol. Summary:

1. Branch: Create experiments/<tag> branch (e.g. experiments/mar14-collusion-params)
2. Baseline: Run the code as-is, record baseline metric in results.tsv
3. Loop (autonomous until interrupted or budget exhausted):
   • Propose a modification based on results so far (informed by prior keep/discard outcomes)
   • Apply the change to the target file(s)
   • git commit the change
   • Run with time budget: timeout <budget>s uv run python <script> > run.log 2>&1
   • Extract metric from output
   • Log to results.tsv: commit hash, metric, status (keep/discard/crash), description
   • If improved → keep (advance branch)
   • If not improved → discard (git reset --hard HEAD~1)
   • If crashed → attempt fix once, else discard and move on
4. Stop when: user interrupts, --budget time exhausted (campaign-level), per-run timeout exceeded, or N consecutive discards with no progress

Key principles:

• One change per iteration — never stack multiple untested changes
• Log every attempt, including discards and crashes
• The TSV is the ground truth — it captures the full exploration history
• Simplicity criterion: prefer changes that simplify the code for equal or better metrics

Gate: results.tsv exists with at least a baseline entry before entering the loop.

Phase 3A: Autonomous Sweep (Parallel Self-Correcting)

Use instead of Phase 3 when: 10+ configs, known failure-prone experiments, or user says "autonomous", "hands-off", "self-correcting sweep".

Read references/autonomous-sweep.md for the full protocol. Summary:

1. Batch configs into groups of max 4 parallel agents
2. Spawn sub-agents — each runs its batch, detects failures by signature (shape mismatch, NaN, empty array, timeout), diagnoses, fixes, retries (max 3 cycles)
3. Aggregate results — coordinator collects batch results, reports success/fixed/unresolved counts
4. Generate figures for all successful configs

Key constraints:

• Max 4 parallel agents
• Max 3 retry cycles per config
• Agents must not modify shared code simultaneously
• Each fix is committed for traceability

Graceful degradation: If some batches fail while others succeed, collect all successful results and report failures. Only stop entirely if ALL batches fail. See shared/skill-design-patterns.md (Graceful Degradation section).

Gate: At least one config must succeed. If all fail, report the unresolved errors and stop.

Breadcrumb: After any Phase 3 variant completes, append to .planning/state.md (if exists) or .context/current-focus.md:

### [computational-experiments] Experiments complete [YYYY-MM-DD HH:MM]
- **Done:** [N configs run, N seeds, mode: experiment/explore/autonomous]
- **Outputs:** [result files at <path>, N successful / N total]
- **Next:** Publication output (figures/tables)

Phase 4: Publication Output

Prerequisite: Result files exist in results/ or experiments/results/.

1. Convergence plots: Metric vs iteration/budget (mean ± std across seeds, multiple methods)
2. Comparison charts: Bar chart or table comparing methods across problems
3. Ablation plots: Performance with/without each component (for multi-agent: use feature toggle ablation configs)
4. Sensitivity analysis: Metric vs parameter (line plot with shaded confidence band)
5. Tables: Performance tables exported as .tex via \input{} — never hard-code results
6. Figure script: scripts/make_all_figures.py that regenerates all figures from saved results

Read references/figure-recipes.md for matplotlib recipes.

Output routing:

• Figures → paper/figures/ as PDF (per overleaf-separation rule)
• Tables → paper/tables/ as .tex (per no-hardcoded-results rule)
• Figure scripts → scripts/ (never inside paper/)

Phase 5: Audit & Save

1. Reproducibility check:
   • Seed determinism: same config + seed → same output?
   • Config completeness: can results be regenerated from saved configs?
   • Output freshness: are figures newer than their source data?
   • Dependency pinning: are versions locked in pyproject.toml or uv.lock?
2. Code review: Invoke the code-review agent on all generated scripts (via skill-routing mechanism)
3. Record learnings: Write [LEARN:code] tags for project-specific conventions discovered
4. Suggest next steps: compilation with latex, additional experiments, replication-package

Breadcrumb: After Phase 5 completes, append to .planning/state.md (if exists) or .context/current-focus.md:

### [computational-experiments] Phase 5 complete [YYYY-MM-DD HH:MM]
- **Done:** [reproducibility check, code review score, N learn tags recorded]
- **Outputs:** [figures at <path>, tables at <path>]
- **Next:** [suggested next steps]

Evolution Protocol

This skill improves with each invocation on a project:

1. Before work: Read MEMORY.md for existing [LEARN:code] entries and Key Decisions
2. During work: Note any convention, naming pattern, or structural choice that should persist
3. After work: Record [LEARN:code] tags and update Key Decisions table

Examples of learnings to capture:

• [LEARN:code] This project uses ElicitationConfig dataclass, not YAML files
• [LEARN:code] Metrics are in src/utils/metrics.py, not a separate package
• [LEARN:code] Seeds are managed via utils/seeds.py with MASTER_SEED + offset

Cross-References

Resource	When read
references/package-scaffold.md	Phase 2 (project structure)
references/algorithm-templates.md	Phase 2 (skeleton code)
references/experiment-patterns.md	Phase 3 (configs, sweeps, runners, config hashing, dual output)
references/multi-agent-patterns.md	Phase 2–3 (agent composition, messaging, multi-level metrics)
references/multi-agent-infrastructure.md	Phase 2–3 (feature toggles, config hashing, simulation runner, visualization)
references/explore-loop.md	Phase 3E (adaptive explore loop)
references/autonomous-sweep.md	Phase 3A (parallel self-correcting sweep)
references/figure-recipes.md	Phase 4 (matplotlib recipes)
shared/publication-output.md	Phase 4 (table/figure format standards)
shared/multi-language-conventions.md	Phase 1 (if non-Python)
docs/guides/hpc.md (Task Management)	Phase 3 (move to Avon for large/GPU/long sweeps)
templates/slurm/*.sbatch (Task Management)	Phase 3 (drop-in SLURM templates; all log git-SHA to OUT_DIR)
no-hardcoded-results rule	Phase 4 (output routing)
overleaf-separation rule	Phase 4 (file placement)
the code-review agent	Phase 5 (auto-invoked)
data-analysis skill	Redirect if task is empirical, not computational
replication-package skill	Phase 5 (suggested next step)
cross-language-check skill	Phase 5 (suggested next step for verification)
references/multi-analyst-design.md	Phase 3–5 (many-analysts robustness diagnostic)
shared/worker-critic-protocol.md	Phase 3–4 (inline review of generated code/results)
shared/checkpoint-resumability.md	All phases (save/resume on crash)
templates/experiments/standard.md	--scaffold standard (init/tune/creative/ablate)
templates/experiments/robustness.md	--scaffold robustness (econometrics robustness)
templates/experiments/replication.md	--scaffold replication (replicate-and-extend)
figure-feedback skill	Phase 4 (VLM analysis of generated plots)