Debugging

Purpose

Find the actual cause of a defect, not a change that makes the symptom disappear. Debugging is a search problem, and the method is binary search over hypotheses.

When to Use

• A test fails and the reason is not obvious from the assertion.
• Production behavior differs from local behavior.
• An intermittent or timing-dependent failure.
• A performance regression with no obvious cause.
• A bug that "came back" after being fixed.

Capabilities

• Deterministic reproduction, including of flaky failures.
• Bisection over code history, input space, and configuration.
• Instrumentation: strategic logging, breakpoints, tracing, core dumps.
• Concurrency debugging: race detectors, deadlock analysis, lock ordering.
• Performance debugging: CPU and allocation profiles, flame graphs.

Inputs

• The failure: error message, stack trace, failing test, or observed behavior.
• The last known-good state, if any.
• Environment differences between where it fails and where it does not.

Outputs

• A reproduction that fails reliably.
• A stated root cause, with the evidence that proves it.
• A fix, plus a regression test that fails without the fix.

Workflow

1. Reproduce — Reduce to the smallest input and shortest path that still fails. If you cannot reproduce it, you cannot verify a fix. For flaky failures, run in a loop until you have a failure rate.
2. Read the evidence — Read the entire stack trace, including the parts you have seen before. Read the actual error, not the one you assume it is.
3. Form one hypothesis — State it as a falsifiable claim: "the cache returns a stale value because the invalidation runs before the write commits."
4. Test the hypothesis — Add instrumentation that would distinguish true from false. Do not change behavior yet.
5. Bisect — If no hypothesis survives, bisect. git bisect over commits; comment out halves of the input; disable half the config.
6. Prove the cause — You have the cause when you can turn the bug on and off at will.
7. Fix and regress — Write the failing test first, then fix, then confirm the test passes and the rest still do.

Best Practices

• Never change two things at once. You will not know which one mattered.
• "It works now" without an explanation means the bug is still there.
• Trust the machine over your memory of what the code does — read the code that is running, on the branch that is deployed.
• If the bug is in a dependency, prove it with a minimal script before reporting or working around it.
• Timing-dependent bugs are nearly always missing synchronization or an assumed ordering. Look for shared mutable state first.
• Delete instrumentation you added, or promote it to permanent, structured logging. Do not leave debug prints behind.

Examples

Hypothesis log for an intermittent failure:

Symptom : /checkout returns 500 approximately 1 in 40 requests under load.
Evidence: Stack trace shows NullPointerException in CartCache.get(); no error at low load.

H1: Cache eviction races with read.
    Test: log cache size + key on every get/put; run 500 concurrent requests.
    Result: FALSE — evictions never coincide with the failures.

H2: Cart is written by request thread but read by an async pricing task
    before the write commits.
    Test: log thread id and transaction id at write and at read.
    Result: TRUE — pricing task reads on a different connection, 3-8ms before commit.

Cause: Async task enqueued inside the transaction, executed outside it.
Fix  : Enqueue on transaction commit (after-commit hook).
Test : Regression test asserting the task is not enqueued until commit.

Notes

• git bisect run <script> automates bisection completely when you have a scripted reproduction. It is the single highest-leverage debugging tool most engineers underuse.
• Heisenbugs that vanish under a debugger are usually timing or optimization-related. Reach for logging and race detectors instead of breakpoints.
• A bug that reappears was never fixed — the original fix addressed a symptom. Reopen the investigation rather than patching again.