establish-empirical-baseline

Purpose

Establish a fair empirical baseline by inventorying methods, extracting comparable performance, normalizing conditions/compute, checking discrepancies, and estimating progress/headroom.

Input contract

required: [method_records, benchmark_or_task, performance_measure]
optional: [historical_series, compute_metadata, condition_schema]
constraints: [comparability fields and source provenance required]

Execution protocol

Do not perform called SOP operations inline; each loaded SOP owns its contract and thresholds.

1. You MUST load skill inventory-reference-items to inventory methods and comparison records.
2. You MUST load skill extract-evidence-record to extract performance evidence. You MUST load skill audit-reporting-quality to audit missing or ambiguous reporting. You MUST load skill normalize-comparison-scale to normalize units, data, compute, and evaluation protocol.
3. You MUST load skill detect-performance-discrepancy to detect comparison discrepancies. You MUST load skill estimate-performance-headroom to estimate headroom. You MUST load skill analyze-temporal-trajectory to analyze progress and leaderboard dynamics. You MUST load skill check-dominance to identify dominated and incomparable records.
4. Synthesize the baseline with uncertainty and known incomparable records. If the resulting methods or baselines require explicit comparative selection, consider rank-candidates as the next tactic.

Output contract

produces: [method_inventory, normalized_baseline, discrepancy_report, progress_curve, headroom_estimate]
delta_fields: [findings, evidence_updates, uncertainties, decisions, open_questions]

Thresholds and quality gates

• Baseline acquisition gates are relative to a declared eligible universe: record numerator, denominator, batch increment, stopping reason, and source references for each ratio.
• method-inventory: method coverage ratio reaches a justified floor over the eligible method universe.
• performance-extraction: comparable-record coverage ratio reaches a justified floor over eligible records.
• condition-standardization: complete condition-vector ratio reaches a justified floor over comparable records.
• discrepancy-analysis: score-pair coverage ratio reaches a justified floor over eligible comparison pairs.
• progress-quantification: historical-time coverage and independent-source ratio reach justified floors; stop when added periods no longer change the trajectory conclusion.
• Across modes, report marginal information gain and saturation state when added records or periods no longer change the baseline conclusion.
• Normalization must expose condition, compute, metric, and unit transformations.

Failure and counterexamples

Do not call a baseline fair when conditions are missing, metrics are incomparable, or leaderboard values are copied without protocol verification. Mark headroom unknown when the historical series is below its floor.

Provenance map

9 architecture old entries: baseline, inventory, extraction, standardization, discrepancy, progress, leaderboard, normalization, curve construction. Repeated reporting prose compressed.

Legacy context checkpoint / Delta notes

Append method IDs, normalized records, excluded records with reasons, discrepancy pairs, progress model, and headroom uncertainty.

Preserved source criteria ledger

source	source line	kind	source criterion
baseline-establishment	28	textual	\
baseline-establishment	40	textual	\
baseline-establishment	58	textual	\
baseline-establishment	70	numeric-table	\
baseline-establishment	71	numeric-table	\
baseline-establishment	72	numeric-table	\
baseline-establishment	73	numeric-table	\
baseline-establishment	74	numeric-table	\
baseline-establishment	75	numeric-table	\
method-inventory	23	numeric-table	\
method-inventory	24	numeric-table	\
method-inventory	25	numeric-table	\
method-inventory	30	textual
method-inventory	33	numeric-table	\
method-inventory	34	numeric-table	\
method-inventory	35	numeric-table	\
method-inventory	36	numeric-table	\
method-inventory	37	numeric-table	\
method-inventory	38	textual
method-inventory	41	numeric	Cannot exit until methods_discovered >= 40 (80% of target).
performance-extraction	18	textual	Extract structured performance data from papers, leaderboards, and reproducibility studies. Each data point is a (Task, Dataset, Metric, Score, Conditions) tuple with full provenance. Prioritizes primary sources (original papers) but cross-references against leaderboards and third-party reproductions.
performance-extraction	24	numeric-table	\
performance-extraction	25	numeric-table	\
performance-extraction	26	numeric-table	\
performance-extraction	27	numeric-table	\
performance-extraction	32	textual
performance-extraction	35	numeric-table	\
performance-extraction	36	numeric-table	\
performance-extraction	37	numeric-table	\
performance-extraction	38	numeric-table	\
performance-extraction	39	numeric-table	\
performance-extraction	40	numeric-table	\
performance-extraction	41	textual
performance-extraction	44	numeric	Cannot exit until data_points >= 120 (80% of target).
condition-standardization	25	numeric-table	\
condition-standardization	26	numeric-table	\
condition-standardization	27	numeric-table	\
condition-standardization	28	numeric-table	\
condition-standardization	33	textual
condition-standardization	36	numeric-table	\
condition-standardization	37	numeric-table	\
condition-standardization	38	numeric-table	\
condition-standardization	39	numeric-table	\
condition-standardization	40	numeric-table	\
condition-standardization	41	textual
condition-standardization	44	numeric	Cannot exit until data_points_standardized >= 48 (80% of target).
condition-standardization	60	textual	3. Group methods by comparable condition sets
condition-standardization	63	textual	6. Produce fair comparison subsets where conditions are controlled
discrepancy-analysis	24	numeric-table	\
discrepancy-analysis	25	numeric-table	\
discrepancy-analysis	26	numeric-table	\
discrepancy-analysis	27	numeric-table	\
discrepancy-analysis	32	textual
discrepancy-analysis	35	numeric-table	\
discrepancy-analysis	36	numeric-table	\
discrepancy-analysis	37	numeric-table	\
discrepancy-analysis	38	numeric-table	\
discrepancy-analysis	39	numeric-table	\
discrepancy-analysis	40	textual
discrepancy-analysis	43	numeric	Cannot exit until score_pairs_compared >= 36 (80% of target).
discrepancy-analysis	52	textual	- reproducibility-checklist-audit - Assess paper reproducibility completeness
discrepancy-analysis	59	textual	4. Apply reproducibility-checklist-audit to papers with large discrepancies
discrepancy-analysis	85	textual	"reproducibility_checklist_score": 0,
progress-quantification	26	numeric-table	\
progress-quantification	27	numeric-table	\
progress-quantification	28	numeric-table	\
progress-quantification	29	numeric-table	\
progress-quantification	34	textual
progress-quantification	37	numeric-table	\
progress-quantification	38	numeric-table	\
progress-quantification	39	numeric-table	\
progress-quantification	40	numeric-table	\
progress-quantification	41	numeric-table	\
progress-quantification	42	numeric-table	\
progress-quantification	43	textual
progress-quantification	46	numeric	Cannot exit until historical_data_points >= 80 (80% of target).
leaderboard-harvesting	43	numeric	- Flag discrepancies > 1 standard deviation
leaderboard-harvesting	58	textual	## Minimum Yield
leaderboard-harvesting	62	numeric-table	\
leaderboard-harvesting	63	numeric-table	\
leaderboard-harvesting	64	numeric-table	\
leaderboard-harvesting	65	numeric-table	\
condition-normalization	28	textual	- Random seeds: number of runs, seed selection, variance reported
condition-normalization	51	textual	### Stage 4: Fair Comparison Baseline
condition-normalization	53	textual	Apply normalization to produce fair comparison subsets:
condition-normalization	58	textual	Yield: Fair comparison tables with methodology notes.
condition-normalization	60	textual	## Minimum Yield
condition-normalization	64	numeric-table	\
condition-normalization	65	numeric-table	\
condition-normalization	66	numeric-table	\
condition-normalization	67	numeric-table	\
progress-curve-construction	62	textual	## Minimum Yield
progress-curve-construction	66	numeric-table	\
progress-curve-construction	67	numeric-table	\
progress-curve-construction	68	numeric-table	\
progress-curve-construction	69	numeric-table	\
progress-curve-construction	73	numeric	- progress-curve-fitting (for Stages 2-3)

Context checkpoint / Delta notes

Append method IDs, normalized records, excluded records with reasons, discrepancy pairs, progress model, and headroom uncertainty.