/ci-repair — Triage, Isolate & Repair Red CI/CD Workflows on Trunk

Deterministic, evidence-backed protocol for rapidly converting red CI/CD workflows and broken trunk gates into verified green runs across both public fak and private fak-private. Decomposes compound CI failures into atomic root causes, arbitrates disjoint write sets, dispatches parallel subagents for implementation, verifies on-device test witnesses, and lands explicit-path commits without peer collision.

---

When to Invoke

1. GitHub Actions Red Alert: Any core CI workflow (spine-invariance, ci-fast, ci, garden, build-matrix, security-audit) fails on main or in an active PR.
2. Push & Synchronization Blockers:
   • fak commit --push or fak sync push fails with COMMITTED_RED, CI_BASE_RED, TRUNK_CLOSED, or pre-push hook rejection.
   • fak sync drain becomes stuck in exponential backoff because the upstream trunk has a broken commit.
3. Dual-Repository Wedges (fak-private): fak-sync repo fails fast-forward verification because public fak does not compile, has broken tests, or fails boundary checks (fak-boundary check).
4. Automated Loop Escalation: Headless super-loops (super-loop, run-it-all-night, fleet-wave) encounter an unrecoverable red trunk and need to heal the mainline before continuing unattended waves.

---

Non-Negotiable Invariants

1. One Root Cause, One Issue, One Commit: Never batch unrelated CI fixes into a kitchen-sink commit. Decompose compound failures into distinct concerns (e.g., an architest upward import vs. an adjudicator structural policy vs. a stale scorecard README) and commit each under its own issue and leaf stamp (fak <leaf>).
2. Witness-First Verification: Never assume a fix works because "the code looks clean." Reproduce the failure first using the exact command CI executes, implement the minimal remedy, verify on-device exit code 0, and cross-validate with an independent subagent before committing.
3. Tree-Disjoint Worker Boundaries: When dispatching concurrent repair workers, assign each worker a mutually disjoint file set. No two workers may concurrently edit the same package or files on the shared trunk.
4. Preserve Architectural Invariants:
   • Never loosen the architest tier table to resolve an upward import; invert the dependency via a registration seam (Register...) or push the shared type down a layer.
   • Never suppress policy rules without structural AST deciders; ensure remedy text does not self-refute by matching its own deny regex.
   • Never silently delete non-secret env reads without updating admittedPostFreeze or moving them to the config surface.
5. Safe Synchronization: Always land commits using fak commit --path or fak sweep --apply. Always push using fak sync push (which safely retries transient non-fast-forward races and refuses destructive merges).

---

Systematic Failure Taxonomy & Remedies

When diagnosing failed runs from gh run list and gh run view <id> --log-failed, classify errors into one of six canonical categories:

Category A: Architest & Layered-DAG Violations (internal/architest)

• Upward Import (TestNoUpwardImports):
  • Symptom: layered-DAG import rule violated: <pkgA> -> <pkgB>. Rule: tier(A) >= tier(B).
  • Remedy: Invert the dependency. Define an interface or function type in the lower tier (e.g. type ContentScreener func(body []byte) bool in internal/mcpbroker), provide a Register... hook, and wire it from a higher-tier package (e.g. internal/headroom/admit.go).
• Unregistered Engine ID (TestSingleEngineDriverPerID):
  • Symptom: engine id "X" is registered by [P] in non-test code but is not a declared id in engineDriverRole.
  • Remedy: Add "X": {"P": "role description"} to engineDriverRole in internal/architest/architest_test.go.
• Off-List Self-Registration (TestRequestPathLeavesRegistered):
  • Symptom: leaf P self-registers (init() -> abi.Register*) but is neither blank-imported in internal/registrations nor on regOffList.
  • Remedy: Add "P": true to regOffList in internal/architest/architest_test.go.
• Non-Literal Subprocess Exec (TestRequestPathInterpreterFree):
  • Symptom: request-path package P calls exec with a non-literal program argument....
  • Remedy: Add "P": "justification" to interpreterExecAllow in internal/architest/architest_test.go.
• Missing Third-Party Tool Inventory (TestThirdPartyEffectsRegisterCoversTree):
  • Symptom: the tree pulls in subprocess_tool "X" and docs/sbom/third-party-effects.json has neither a row nor an exclusion.
  • Remedy: Add "X" under class: "subprocess_tool" in alphabetical order in docs/sbom/third-party-effects.json with "pin": "host_provided".
• Declared Reason Missing Code Emitter (TestEveryDeclaredReasonHasAnEmitter):
  • Symptom: dos.toml declares [reasons.X] but no non-test source contains the token.
  • Remedy: Define const ReasonX = "X" in the declaring package and wire it into the producer's refusal/status path.
• Brittle Test Sleeps (TestNoBrittleSleepsInAuditedPackages):
  • Symptom: brittle time.Sleep synchronization in test; use channels, sync.WaitGroup, or bounded retry/polling loops.
  • Remedy: Replace time.Sleep(...) with channel synchronization or select { case <-time.After(...): }.

Category B: Policy & Structural Surface Rules (internal/adjudicator, internal/policy)

• Unrecognised Structural Rule (TestEveryShippedStructuralRuleIsRecognised):
  • Symptom: tool "X" arg "command" ships a deny_regex with no structural decider and no inventory entry.
  • Remedy: Implement a structural AST/command parser in internal/adjudicator/<rule>.go (e.g. isGitPushArgRule, commandExecutesGitPush), wire it in decide_argpredicates.go, and register it in policy_structural_surface_test.go under families.
• Self-Refuting Remedy Text (TestEveryShippedDenyRuleNamesARemedy):
  • Symptom: the fix text for regex MATCHES its own deny_regex, and this rule has NO structural decider.
  • Remedy: Add the structural decider to the decided boolean in policy_structural_surface_test.go so inert mentions in commit messages, greps, or quoted text are admitted without re-tripping the rule.

Category C: Generated Artifact & Scorecard Freshness (concept freshness, docs/)

• Disambiguation Scorecard Drift:
  • Symptom: concept generated-artifact freshness fails with {"verdict":"stale","stale_paths":["docs/concept-disambiguation-scorecard/README.md"]}.
  • Remedy: Regenerate via go run ./cmd/fak concept generate, then verify with go run ./cmd/fak concept freshness --check --json. Commit under lane (fak docs).

Category D: Uncommitted Companion WIP / Compiler Breakages (COMMITTED_RED)

• Missing Struct Field / Method:
  • Symptom: X undefined (type T has no field or method X).
  • Remedy: Check the working tree for uncommitted companion edits from peer sessions, or implement the missing exported field/method with appropriate doc comments and tests.

Category E: Ratchet & Debt Ledgers (internal/envconfiglint, CONFIG_NOT_ENV)

• Unrecorded Non-Secret Env Read:
  • Symptom: env read X is not a declared secret; move it to the config surface (CONFIG_NOT_ENV).
  • Remedy: If the read landed during an unmonitored window, record "X" in admittedPostFreeze in internal/envconfiglint/admitted.go and verify with go test ./internal/envconfiglint.

Category F: Formatting & Line Hygiene (gofmt)

• Unformatted Go Source:
  • Symptom: gofmt: N file(s) not formatted (run 'gofmt -w .' from fak/).
  • Remedy: Format the specific affected files with gofmt -w <paths>, verify with gofmt -l <paths>, and commit via fak commit --path.

---

Step-by-Step Execution Workflow

Phase 1: Triage & Failure Capture

1. List recent workflow runs:

   gh run list --limit 15
   

2. Inspect failed workflow jobs:

   gh run view <run_id> --log-failed
   

   Or examine specific failed job logs:

   gh run view --job=<job_id> --log
   

3. Partition failures: Group failures into distinct, non-overlapping concerns (Category A through F).

Phase 2: Advance Ticketing & Arbitration

1. Create dedicated GitHub issues: For each distinct failure category:

   gh issue create --title "fix(<subsystem>): <concise summary>" --body "Problem: ... Intended outcome: ... Witness: ..."
   

2. Arbitrate lane leases: Verify lane availability with dos arbitrate:

   dos arbitrate --lane <lane> --mode exclusive
   

Phase 3: Parallel Subagent Delegation

1. Assign disjoint write sets: Launch concurrent worker subagents for independent failure components.
   • Worker 1: e.g. internal/mcpbroker/..., internal/headroom/admit.go
   • Worker 2: e.g. internal/adjudicator/git_push.go, internal/adjudicator/...
   • Worker 3: e.g. docs/concept-disambiguation-scorecard/README.md
2. Execute on-device witness commands: Each worker must execute and return clean witness command output (e.g. go test -v ./internal/architest -run TestNoUpwardImports).

Phase 4: Independent Cross-Validation

1. Dispatch a cross-validator or tester subagent to execute the full affected test matrix independently before landing.
2. Confirm zero regressions on related packages.

Phase 5: Atomic Staging, Commit & Push

1. Lint subject before commit:

   fak commit --preview --path <p1> --path <p2> -m "fix(<lane>): <msg> #<issue> (fak <lane>)"
   

2. Commit explicit paths:

   fak commit --path <p1> --path <p2> -m "fix(<lane>): <msg> #<issue> (fak <lane>)"
   

3. Safe sync and push:

   fak sync push
   

   If racing against peer commits, fak sync push automatically retries fast-forward rebases up to the budget.

Phase 6: Post-Push CI Verification

1. Monitor new workflow runs triggered by the push:

   gh run list --limit 5
   

2. Watch until spine-invariance, ci-fast, and ci complete green.

---

Integration with Automated Processes & Loops

To keep unattended fleets moving when CI breaks, other processes interact with /ci-repair as follows:

1. The fak sync drain Recovery Hook

fak sync drain queues commits stranded when the upstream trunk is red. Instead of passive exponential backoff that deadlocks on a permanent trunk break:

• On attempt 2 (attempts >= 2), fak sync drain inspects gh run list --workflow ci-fast.
• If the failure is a known deterministic break (e.g., stale concept scorecard or unformatted files), it automatically triggers the deterministic healing pass (fak concept generate / gofmt -w), commits the fix, and flushes the drain queue.

2. fak recover COMMITTED_RED

When fak commit or preflight encounters COMMITTED_RED:

• fak recover COMMITTED_RED provides executable recovery:

  fak recover COMMITTED_RED --execute
  

  which invokes the /ci-repair diagnostics and suggests the exact remediation command.

3. Dual-Repo Sync (fak-sync repo in fak-private)

When fak-sync repo is wedged by public trunk compilation errors:

• fak-sync repo automatically runs boundary and compilation probes (fak validate --mine, go work sync).
• If public fak is red, it suggests or triggers /ci-repair on the public checkout before attempting to fast-forward the private mirror.

4. Headless Super-Loops (run-it-all-night, fleet-wave, super-loop)

When a headless coordinator catches an EXIT_TRUNK_RED error:

• It checks dos arbitrate --lane ci.
• If free, it acquires the ci lane lease, executes the bounded /ci-repair packet, verifies trunk health, and resumes the overnight issue wave without terminating.