Strategy: Comparison Design

Question: How much better is our method than the baseline?

Methodology

• Fair Comparison Protocol (Bouthillier 2021): Control all confounds, same compute budget, same tuning effort.
• Multi-Baseline Comparison: Compare against multiple baselines (SOTA, simple, ablated).
• Multi-Dataset Evaluation: Test across diverse datasets to avoid dataset-specific overfitting.
• Bayesian Comparison (Benavoli 2017): Posterior probability of superiority, not just p-values.
• Bootstrap/Permutation Tests: Non-parametric significance without distributional assumptions.

Execution Flow

1. baseline-selection → Select appropriate baselines (SOTA, simple, oracle)
2. metric-specification → Define primary metric and secondary metrics
3. sample-size-estimation → Power analysis for detecting meaningful differences
4. seed-protocol-design → Ensure fair random initialization across methods
5. environment-specification → Lock environment to prevent confounds
6. reproducibility-protocol (tactic) → Ensure all results are reproducible
7. statistical-method-selection (tactic) → Choose Bayesian or frequentist comparison

Budget Gate

Comparison Scope	Baselines	Datasets	Seeds	Min Runs
Minimal	1 SOTA + 1 simple	1	3	6
Standard	2-3 baselines	2-3	5	30-45
Comprehensive	4+ baselines	3-5	5-10	100+
Publication-ready	All relevant	5+	10+	200+

Available Tactics

Optional, no fixed order; the final leaf is always a sop.

Tactic	When to use
reproducibility-protocol	Ensure experiment reproducibility through systematic environment and seed control
statistical-method-selection	Select appropriate statistical methods for experiment analysis

Available SOPs

Optional, no fixed order; the final leaf is always a sop.

SOP	When to use
baseline-selection	Select appropriate baselines for experimental comparison
environment-specification	SOP: define complete experiment environment specification
metric-specification	Define experiment metrics and significance standards
sample-size-estimation	SOP: power analysis and required experiment count estimation
seed-protocol-design	SOP: design random seed strategy for reproducibility