root-cause-tracing-process

<tracing_input>$ARGUMENTS</tracing_input>

Root-Cause Tracing Process

Inputs and completion contract

Record the question, affected scope, requested operation, and success criteria from the input and conversation. An investigation-only request does not authorize product or test edits.

Success requires a reproduced failure, a cited mechanism, and an independently established contract before declaring a defect and recommending its correction. When execution is unavailable or prohibited, return a partial evidence assessment; do not invent reproduction or a root cause.

Use this causal statement only as far as the evidence supports it:

Under conditions C, X produces Y through mechanism M, violating contract K.

Keep explanation of current behavior separate from the decision that the behavior is wrong.

Evidence-chain protocol

Give each material claim an evidence entry:

CLAIM: [one observation, interpretation, or causal claim]
EVIDENCE: [command + output/exit status, file:lines, or directly observed state]
KIND: [runtime observation / source observation / requirement authority / hypothesis]
VERIFIED: [yes/no, for this claim rather than its entire source]
DEPENDS ON: [prior claim IDs]
MISSING: [observation or decision needed, when unverified]

Command output, read source, and direct observations establish what was observed. A source read establishes the code present at that revision, not that the path executed. Documents and explicit owner decisions can establish requirement authority; they do not demonstrate runtime compliance. Check which document is authoritative and current rather than treating any prose as a requirement.

Label derived interpretations and hypotheses; retain their supporting observations. Training recall, analogy, absent documentation, and an untested explanation are not verified runtime evidence. An unverified dependency leaves dependent conclusions unverified. Do not promote a coherent interpretation into established intent.

0. Discover relevant capabilities

Identify the available shell, diagnostic tools, connected MCP tools, specialist agents, and sandbox options needed for this incident. Use independent checks in parallel when supported. Use native discovery rather than assuming Unix commands, an installed plugin, or a sub-agent API. Record unavailable capabilities and their effect on verification. Do not install unrelated tools or perform a broad inventory when it cannot change the next decision.

1. Preserve the signal and establish the contract

Before updating dependencies, regenerating files, or editing assertions, preserve the failing revision, actual checkout SHA, workflow/job, command, complete output, exit status, and relevant runtime versions, configuration, test selection, order, seeds, and parallelism. For a PR job, distinguish its tested merge commit from the branch head. Keep logs as reported observations, not as your own reproduction.

Separate assertion failures, collection/setup failures, lint failures, and downstream aggregate failures. Group symptoms by demonstrated mechanism, not filenames or a shared red workflow.

Identify the smallest useful contract: purpose, consumer/entry point, observable guarantee, invariants, authoritative source, and falsifier. Read applicable purpose, architecture, interface, and requirement records. Keep goals stable; refine local detail only where it changes a decision. The current architecture can itself conflict with higher-level intent. Neither an existing test nor the current implementation wins a disagreement automatically.

For unknowns, record the cheapest discriminating observation. Resolve factual uncertainty with available evidence; ask the owner only when choosing among legitimate behaviors would create or alter intent, or when safe execution needs information only the owner can supply.

2. Establish reproduction safety

Classify constraints before executing the failing operation:

• Bound: inputs and side effects can be inspected. Read enough to assess risk; use an isolated directory, disposable data, dry run, or other appropriate containment before risky reproduction.
• Unbound: credentials, external state, side effects, or permissions cannot be evaluated. Complete safe read-only work, then batch the missing safety/input questions. Do not cross that boundary until the necessary information and authorization exist.

Respect the requested oversight and read-only scope. Reassess safety if a new external or destructive boundary appears. A requirement to reproduce never authorizes production writes, messages, deployments, or other irreversible operations.

3. Reproduce, then reduce

Run the same operation end-to-end under the preserved conditions when safe and permitted. Capture all arguments and relevant environment, stdout/stderr, exit status, timing, and side effects, including files, network activity, and child processes. Observe the mechanism as well as the final failure. Inspect source or add bounded instrumentation where the mechanism is hidden.

Then reduce to the smallest reproducer that retains the mechanism. Preserve both the full-system observation and the reduced case. Compare isolated/full-suite, serial/parallel, or clean/reused state only when those contrasts can discriminate a hypothesis. Use a known-good revision or bisection when a stable reproducer makes the comparison meaningful.

If results differ from the report, record the environmental/operational differences rather than assuming the report is wrong. Use available CI commands and logs to recover missing steps before asking the user. If reproduction remains unavailable, continue with explicit evidence limits. A successful rerun is an observation, not a fix.

4. Challenge competing explanations

Consider implementation, producer/consumer contract, test expectation, fixture/mock, observation, and CI/environment explanations. State what evidence would distinguish them before editing. Choose one discriminating intervention at a time; record confounds and hold relevant conditions fixed. Source inspection and controlled observations must support the mechanism, not just a correlation with the most recent change.

Intermittent or non-reproducing failures

Do not require a probabilistic cause to fail on every run. Choose the hypothesis form before examining confirmatory results:

• Deterministic: under stated preconditions, setting X produces Z; specify the counterexample that would contradict this guarantee.
• Probabilistic: setting X changes the probability of Z under stated conditions. State a null of no effect and an alternative with the direction/effect relevant to the correction decision.

Predeclare comparable arms, measurement, reset/isolation procedure, repeat-count rationale, and stopping rule. Use randomization or interleaving where time/order could confound results; record seeds when applicable. Size the experiment for the observed frequency and decision-relevant uncertainty, not an arbitrary universal retry count. Keep exploratory and confirmatory runs distinct.

Capture every run in both arms, including successes. Report failures/trials, effect estimates and uncertainty where estimable, uncontrolled conditions, and the resulting evidence boundary. Distinguish supported, contradicted, and inconclusive hypotheses. Failure to reject a null is not proof of no effect; zero observed failures is not proof of impossibility. A nonzero failure rate in both arms does not by itself falsify an effect hypothesis.

Return to source/mechanism tracing when the experiment isolates a relevant condition. Otherwise record the remaining hypotheses and next discriminating experiment; do not loop until a desired result or label an association a verified root cause.

5. Trace the product and the test

Read the complete relevant code paths, including setup and cleanup. Cite paths and line ranges and quote the lines needed to support each claim. Reuse the capability matrix for runtime instrumentation or specialist analysis when source alone cannot resolve the mechanism.

Trace both paths where a test is involved:

Product: intent -> entry point -> routing -> inputs/state -> execution -> output -> consumer
Test:    claimed contract -> fixture/setup -> invoked boundary -> observation -> assertion -> verdict

Include recovery and terminal behavior where relevant. At each material boundary record producer, consumer, state crossing it, caller assumptions, callee guarantees, and partial-failure/recovery ownership. Compare expected and observed behavior to locate the earliest demonstrated divergence, not merely the assertion or exception that finally reports it.

Do not derive the test's expected answer from the same implementation logic being tested. Distinguish a valid requirement with weak observation from an incorrect requirement. Locally correct components can still compose incorrectly; a product defect and a test defect can coexist.

6. Build the causal chain

Connect symptom -> observed mechanism -> triggering condition -> supported structural contributors. For each link retain evidence and dependencies. Distinguish:

• Trigger: what exposed the incident now.
• Mechanism: how the incorrect behavior occurred.
• Structural contributor: ownership, lifecycle, dependency, or boundary design that permits it.

Do not force several contributing conditions into one unsupported root cause. Trace only as deep as needed to choose a correction and preserve relevant parent invariants. Stop expanding when finer resolution cannot change that decision, evidence is unavailable, or an intent decision is needed.

7. Adjudicate the correction and hand off

Classify independently of which file is easiest to edit:

Evidence-backed finding	Correction
Implementation violates a valid contract	Correct the product and add fault-sensitive regression protection.
Local contracts conflict at a handoff	Correct ownership/interface/composition and test the boundary.
Test expectation is unsupported or superseded	Correct the test and record the authoritative reason.
Fixture, mock, observation, or CI conditions misrepresent the contract	Correct the harness without weakening the product guarantee.
Intent remains contradictory or undefined	State alternatives and obtain the consequential decision.

Several rows may apply. For an architectural proposal, name the causal condition it removes and why a smaller local correction is insufficient. Do not require a redesign merely because a test failed, or dismiss a recurring structural defect with a symptom patch.

When process-siren:improve-processes is available, pass the same contract, evidence chain, counterexample, and uncertainty record into its CHALLENGE/IMPROVE/VALIDATE loop. Do not restart investigation or create a parallel model. When it is unavailable, use this standalone handoff:

1. Declare the targeted claim, expected improvement, preserved invariants, unacceptable regressions, and smallest useful baseline before examining the candidate result.
2. Apply only authorized corrections determined by established intent. Record each affected original behavior as preserved, relocated, automated, proven redundant, uncertain, or lost. Required lost behavior blocks acceptance; behavior-affecting uncertainty remains unresolved.
3. Validate the delta using comparable before/after cases and affected interfaces. For substantial redesign, add independent/held-out cases where practical and record any unavailable evaluation.
4. Feed counterexamples back to the nearest affected claim. Revalidate evidence invalidated by later edits. Report actual execution and remaining uncertainty, not only green CI.

For test changes, use Comprehensive Test Review to assess the oracle, relevant fault detection, refactor tolerance, and boundary fidelity. Keep inventory freshness, generator correctness, and actual skill/consumer execution as separate claims. Never regenerate a stale artifact before its freshness check and call that evidence the original state was current.

8. Present findings

QUESTION:
SCOPE / REQUESTED MODE:
SUCCESS CRITERIA MET: yes | partial | no
CONTRACT: purpose, guarantee, authority, invariants, falsifier
REPRODUCTION: exact revision/command/conditions, result, or why unavailable
EVIDENCE CHAIN: claim IDs with evidence, kind, verification, and dependencies
FIRST DIVERGENCE: product boundary and/or test-observation boundary
ROOT CAUSE: supported statement, or not established
TRIGGER / MECHANISM / STRUCTURAL CONTRIBUTORS:
CORRECTION: product | architecture/interface | test oracle | harness/CI | intent decision
CHANGE CONTRACT / HANDOFF: proposal, or authorized work actually performed
VALIDATION: claim -> test/experiment -> observed result -> evidence boundary
UNVERIFIED ITEMS: missing evidence, residual risk, next observation/decision

Do not call reported logs your reproduction, source review a behavioral run, or an unexecuted scenario a passing test. Do not suppress warnings, skip inconvenient assertions, or loosen gates to manufacture success. State unsupported hypotheses as hypotheses, never as diagnoses.