design-experiment

Purpose

Translate a falsifiable hypothesis into a statistically defensible experiment. Factorial, ablation, comparison, scaling, and robustness are modes.

When to use / not applicable

Use after a hypothesis has an operational construct and discriminating prediction. Not applicable when the hypothesis or measurable outcome is absent.

Input contract

mode_contracts:
  factorial: &experiment_input
    required: [falsifiable_hypothesis, outcome, factors, constraints]
    optional: [baseline, candidate_models, budget, robustness_axes]
    constraints: [analysis_plan_must_be_preregistered]
  ablation: *experiment_input
  comparison: *experiment_input
  scaling: *experiment_input
  robustness: *experiment_input

Execution protocol

Do not perform called SOP operations inline; each loaded SOP owns its contract and thresholds.

1. You MUST load skill identify-variables to operationalize the outcome, factors, controls, and their functional roles. You MUST load skill enumerate-dimension-values to define admissible levels or perturbation values. If the resource envelope or feasibility constraints cannot support an executable design, consider analyze-constraints-readiness before committing to a mode.
2. Choose factorial, ablation, comparison, scaling, or robustness mode.
3. You MUST load skill specify-metrics to preregister metrics, estimands, directionality, and decision thresholds. You MUST load skill estimate-sample-size to derive the sample or repetition requirement. You MUST load skill select-statistical-method to select the inference or estimation method before observing outcomes; specify power and the stopping rule. If the proposed metric or validator may share artifacts, labels, or assumptions with the system under test, consider audit-validator-independence before freezing the analysis plan.
4. You MUST load skill construct-design-matrix to construct the runnable matrix for the selected mode. You MUST load skill design-randomness-protocol to define seeds, repetitions, and propagation rules. You MUST load skill specify-execution-environment to capture interpretation-relevant hardware, software, data, configuration, and versions. You MUST load skill specify-reproducibility-protocol to define and test the intended reproduction level. You MUST load skill optimize-design-under-budget to select a feasible information-efficient design under the declared resource envelope; include resource and failure checks. Once the design has produced observations and the task changes from planning to inference, analyze-experiment-results may be the better next tactic.

Mode branches

For comparison or robustness, You MUST load skill select-experimental-baseline to choose a controlled baseline matched to the claim.

• factorial: vary multiple factors in a structured design so main effects and interactions are estimable within the declared resource envelope.
• ablation: remove or replace components systematically to attribute the outcome to individual parts and suspected interactions. You MUST load skill map-ablation-components to define ablatable units, dependencies, and legal removal or replacement operations.
• comparison: evaluate a target against controlled baselines with matched confounds, compute, tuning effort, and preregistered tests.
• scaling: instantiate geometric or otherwise justified scale points to test how the outcome changes across the declared regime.
• robustness: perturb relevant conditions or inputs and measure whether the claimed effect survives the defined stress space.

Output contract

mode_contracts:
  factorial:
    produces: [factor_level_matrix, factor_level_catalog, estimands_main_effects_interactions, metric_significance_plan, sample_power_plan]
    delta_fields: [findings, decisions, uncertainties, open_questions]
  ablation:
    produces: [ablation_matrix, baseline_anchors_full_minimal, attribution_contrasts, component_interaction_plan]
    delta_fields: [findings, decisions, uncertainties, open_questions]
  comparison:
    produces: [controlled_baseline_comparison, matched_confound_controls, seed_environment_protocol, statistical_comparison_plan, reproducibility_protocol]
    delta_fields: [findings, decisions, uncertainties, open_questions]
  scaling:
    produces: [scaling_axes, geometric_scale_points, scaling_experiment_grid, curve_fit_plan, scale_budget_plan]
    delta_fields: [findings, decisions, uncertainties, open_questions]
  robustness:
    produces: [perturbation_stress_matrix, severity_axes, baseline_comparison, degradation_metrics, survival_criteria]
    delta_fields: [findings, decisions, uncertainties, open_questions]

Thresholds and quality gates

• experiment-design HARD-GATE and Budget Gate remain mandatory; no exit before declared minimum yield.
• Resource and run-scale gates are relative to the declared resource envelope and eligible factor/condition space; record numerator, denominator, batch increment, stopping reason, and source references.
• Scale, ablation, comparison, and robustness coverage must reach a justified relative floor over the declared design space; do not substitute an unreasoned fixed run count.
• Factor levels, comparison baseline, statistical test, significance threshold, sample-size rationale, and stopping rule must all be explicit.
• Significance threshold must be pre-registered, not chosen post-hoc.
• Budget-constrained design must report at least one feasible design under the stated resource envelope.
• Scaling mode retains the source criterion of a geometric progression, typically 4-8 points, while evaluating coverage relative to the declared scale domain. Relative audit: declared universe = eligible scale domain; numerator = scale points instantiated; batch increment = points added per design pass; stopping reason = predeclared coverage or saturation rule; source references = scaling-design ledger and design records; direction/threshold rationale = geometric spacing preserves regime sensitivity, with 4-8 retained as the source band.

Failure and counterexamples

Reject post-hoc factor selection, outcome leakage, missing control, unpowered comparison, or a design whose claimed inference exceeds its measured outcome.

Provenance map

experiment-design, factor-level-design, ablation-design, comparison-design, scaling-design, robustness-design, statistical-method-selection, reproducibility-protocol, budget-constrained-design; mode-specific steps retained, agent dispatch removed.

Legacy context checkpoint / Delta notes

Record hypothesis ID, design mode, factors/levels, analysis plan, budget, preregistration status, and unresolved threats.

Preserved source criteria ledger

source	source line	kind	source criterion
experiment-design	45	textual	## HARD-GATE
experiment-design	47	textual	Before entering this campaign, the following must be satisfied:
experiment-design	64	textual	4. How to ensure reproducibility
experiment-design	79	textual	## Budget Gate
experiment-design	83	numeric	\
experiment-design	84	numeric	\
experiment-design	85	numeric	\
experiment-design	86	numeric	\
experiment-design	95	textual	## Minimum Yield
experiment-design	97	textual	Every campaign invocation must produce at minimum:
ablation-design	43	textual	## Budget Gate
ablation-design	47	numeric	\
ablation-design	48	numeric	\
ablation-design	49	numeric	\
ablation-design	50	numeric	\
ablation-design	51	numeric	\
comparison-design	34	textual	- Fair Comparison Protocol (Bouthillier 2021): Control all confounds, same compute budget, same tuning effort.
comparison-design	47	textual	6. reproducibility-protocol (tactic) -> Ensure all results are reproducible
comparison-design	50	textual	## Budget Gate
comparison-design	54	numeric-table	\
comparison-design	55	numeric	\
comparison-design	56	numeric	\
comparison-design	57	numeric-table	\
scaling-design	43	numeric	2. level-specification -> Define scale points (geometric progression, typically 4-8 points)
scaling-design	49	textual	## Budget Gate
scaling-design	53	numeric	\
scaling-design	54	numeric	\
scaling-design	55	numeric	\
scaling-design	56	numeric	\
robustness-design	50	textual	## Budget Gate
robustness-design	54	numeric	\
robustness-design	55	numeric	\
robustness-design	56	numeric	\
robustness-design	57	numeric	\
robustness-design	58	numeric-table	\
statistical-method-selection	30	numeric-table	\
statistical-method-selection	31	numeric-table	\
statistical-method-selection	32	numeric-table	\
statistical-method-selection	33	numeric-table	\
statistical-method-selection	34	numeric-table	\
statistical-method-selection	36	numeric-table	\
statistical-method-selection	48	textual	- Is the significance threshold pre-registered (not chosen post-hoc)?
reproducibility-protocol	17	textual	# Tactic: Reproducibility Protocol
reproducibility-protocol	21	textual	1. Assess Reproducibility Requirements -> Determine level needed (exact, statistical, conceptual)
reproducibility-protocol	22	textual	2. seed-protocol-design -> Design random seed strategy for all stochastic components
reproducibility-protocol	24	textual	4. Define Verification Plan -> How to confirm reproducibility (re-run subset, cross-machine test)
reproducibility-protocol	25	textual	5. Document Non-Determinism -> Identify and document unavoidable sources of variance
reproducibility-protocol	29	textual	\
reproducibility-protocol	37	textual	- Are all random seeds documented and controllable?
reproducibility-protocol	38	textual	- Is the full software environment captured (versions, dependencies)?
reproducibility-protocol	39	textual	- Are hardware-specific non-determinisms identified (GPU atomics, cuDNN)?
reproducibility-protocol	40	textual	- Is there a verification protocol (re-run N times, check variance)?
reproducibility-protocol	44	textual	- Is there a plan for cross-machine reproducibility testing?
budget-constrained-design	34	numeric	\
budget-constrained-design	35	numeric	\
budget-constrained-design	36	numeric	\
budget-constrained-design	37	numeric	\
budget-constrained-design	38	numeric-table	\
budget-constrained-design	53	textual	- Are early stopping criteria pre-defined (not post-hoc)?

Context checkpoint / Delta notes

Record hypothesis ID, design mode, factors/levels, analysis plan, budget, preregistration status, and unresolved threats.