Test Failure Analysis Mindset

Establish a balanced investigative approach for all test failures encountered in this session.

Core Principle

Tests are specifications - they define expected behavior.

Dual Hypothesis Approach

Always consider both possibilities when a test fails:

Hypothesis A	Hypothesis B
Test expectations are incorrect	Implementation has a bug
Test is outdated	Test caught a regression
Test has wrong assumptions	Test found an edge case

Investigation Protocol

For EVERY test failure:

1. Pause and Read

• Understand what the test is trying to verify
• Read its name, comments, and assertions
• Check the test's history (git blame) for context

2. Trace the Implementation

• Follow the code path that leads to the failure
• Understand actual behavior vs. expected behavior
• Check if recent changes affected this code path

3. Consider the Context

• Is this testing a documented requirement?
• Would current behavior surprise a user?
• What would be the impact of each possible fix?

4. Make a Reasoned Decision

Situation	Action
Implementation is wrong	Fix the bug
Test is wrong	Fix test AND document why
Unclear	Seek clarification before changing

5. Learn from the Failure

• What can this teach about the system?
• Should additional tests cover related cases?
• Is there a pattern being missed?

Red Flags (Dangerous Patterns)

• Immediately changing tests to match implementation
• Assuming implementation is always correct
• Bulk-updating tests without individual analysis
• Removing "inconvenient" test cases
• Adding mock/stub workarounds instead of fixing root causes

Good Practices

• Treat each test failure as a potential bug discovery
• Document analysis in comments when fixing tests
• Write clear test names that explain intent
• Consider adding more tests when finding ambiguity

Related Skills

• analyze-test-failures: Detailed analysis of specific test failures
• comprehensive-test-review: Full test suite review