Calibration Trainer

Overview

The Calibration Trainer skill provides capabilities for assessing and improving forecaster calibration. It helps decision-makers align their confidence levels with actual accuracy, reducing overconfidence and improving the quality of probabilistic judgments.

Capabilities

• Calibration quiz generation
• Confidence interval elicitation
• Brier score calculation
• Calibration curve plotting
• Overconfidence/underconfidence diagnosis
• Training exercise management
• Progress tracking over time
• Benchmark comparison

Used By Processes

• Cognitive Bias Debiasing Process
• Decision Quality Assessment
• Predictive Analytics Implementation

Usage

Calibration Quiz

# Generate calibration quiz
quiz_config = {
    "type": "general_knowledge",
    "format": "confidence_interval",
    "questions": 20,
    "confidence_levels": [50, 80, 90],  # percentiles to elicit
    "difficulty": "medium",
    "domains": ["business", "economics", "technology", "geography"]
}

# Example question
quiz_question = {
    "id": "Q001",
    "question": "In what year was Amazon founded?",
    "actual_answer": 1994,
    "format": "numeric_interval",
    "required_responses": [
        {"confidence": 50, "prompt": "Give your best estimate"},
        {"confidence": 80, "prompt": "Give a range you're 80% confident contains the answer"},
        {"confidence": 90, "prompt": "Give a range you're 90% confident contains the answer"}
    ]
}

Response Collection

# Collect responses
responses = {
    "participant": "John Smith",
    "date": "2024-01-15",
    "questions": [
        {
            "question_id": "Q001",
            "responses": {
                "point_estimate": 1997,
                "interval_80": [1995, 2000],
                "interval_90": [1992, 2002]
            }
        }
        # ... more questions
    ]
}

Calibration Analysis

# Analyze calibration
calibration_analysis = {
    "participant": "John Smith",
    "n_questions": 20,
    "by_confidence_level": {
        "80%_intervals": {
            "expected_hit_rate": 0.80,
            "actual_hit_rate": 0.55,
            "calibration_gap": -0.25,
            "interpretation": "overconfident"
        },
        "90%_intervals": {
            "expected_hit_rate": 0.90,
            "actual_hit_rate": 0.70,
            "calibration_gap": -0.20,
            "interpretation": "overconfident"
        }
    },
    "brier_score": 0.18,  # lower is better, 0 = perfect
    "overconfidence_index": 0.23,
    "recommendations": [
        "Widen confidence intervals by ~25%",
        "Practice with domain-specific questions",
        "Use reference class thinking"
    ]
}

Training Exercises

# Calibration training program
training_program = {
    "participant": "John Smith",
    "baseline_calibration": 0.55,  # hit rate for 80% intervals
    "target_calibration": 0.75,
    "exercises": [
        {
            "week": 1,
            "focus": "interval_widening",
            "exercise": "Practice giving intervals 50% wider than instinct",
            "quiz_count": 10
        },
        {
            "week": 2,
            "focus": "reference_class",
            "exercise": "For each estimate, identify a reference class first",
            "quiz_count": 10
        },
        {
            "week": 3,
            "focus": "decomposition",
            "exercise": "Break complex estimates into components",
            "quiz_count": 10
        },
        {
            "week": 4,
            "focus": "consolidation",
            "exercise": "Apply all techniques, track improvement",
            "quiz_count": 20
        }
    ]
}

Progress Tracking

# Track progress over time
progress_data = {
    "participant": "John Smith",
    "history": [
        {"date": "2024-01-01", "hit_rate_80": 0.55, "brier_score": 0.22},
        {"date": "2024-01-15", "hit_rate_80": 0.62, "brier_score": 0.19},
        {"date": "2024-02-01", "hit_rate_80": 0.68, "brier_score": 0.16},
        {"date": "2024-02-15", "hit_rate_80": 0.74, "brier_score": 0.13}
    ],
    "trend": "improving",
    "improvement_rate": "4% per session"
}

Input Schema

{
  "operation": "quiz|analyze|train|track",
  "quiz_config": {
    "type": "string",
    "format": "string",
    "questions": "number",
    "confidence_levels": ["number"]
  },
  "responses": {
    "participant": "string",
    "questions": ["object"]
  },
  "training_config": {
    "target_calibration": "number",
    "duration_weeks": "number"
  }
}

Output Schema

{
  "quiz": {
    "questions": ["object"],
    "total_count": "number"
  },
  "calibration_analysis": {
    "by_confidence_level": "object",
    "brier_score": "number",
    "overconfidence_index": "number",
    "calibration_curve": "object"
  },
  "recommendations": ["string"],
  "progress": {
    "history": ["object"],
    "trend": "string",
    "target_achieved": "boolean"
  }
}

Calibration Metrics

Metric	Formula	Interpretation
Hit Rate	% of intervals containing true value	Should match confidence level
Brier Score	Mean squared error of probabilities	Lower is better (0-1)
Calibration Gap	Expected - Actual hit rate	Positive = overconfident
Overconfidence Index	Average calibration gap	Quantifies overall bias

Calibration Curve

A well-calibrated forecaster has:

• 50% intervals capturing truth 50% of the time
• 80% intervals capturing truth 80% of the time
• 90% intervals capturing truth 90% of the time

The calibration curve plots stated confidence vs. observed accuracy.

Best Practices

1. Use feedback immediately after each quiz
2. Track calibration separately by domain
3. Focus on the most common confidence levels (80%, 90%)
4. Practice regularly (weekly is better than monthly)
5. Use domain-relevant questions for business applications
6. Compare to well-calibrated benchmarks (superforecasters)
7. Celebrate improvement, not just accuracy

Techniques to Improve Calibration

Technique	Description
Widen intervals	Start wider, narrow only with strong evidence
Reference classes	Use base rates from similar situations
Decomposition	Break estimates into components
Devil's advocate	Actively seek reasons to be less confident
Pre-mortem	Imagine being wrong, identify why

Integration Points

• Feeds into Decision Quality Assessment
• Connects with Risk Distribution Fitter for expert elicitation
• Supports Debiasing Coach agent
• Integrates with Reference Class Forecaster for base rate thinking