test-evidence-storage

Test Evidence Stays Out of the Code Branch

• The test step always collects evidence OUTSIDE the worktree, in the directory the executor resolved once as StepContext.EvidenceDir; nothing stages or commits it into the pushed branch, so evidence can never reach the default branch's history. With test.evidence.store_in_repo and a derivable GitHub link base, the PR step calls publishRunEvidence (internal/pipeline/steps/evidence_publish.go), which copies the directory onto the push-target repo's orphan evidence branch through internal/evidence and hands the PR body its links. A provider without derivable links does not push the branch. Independently, on GitHub.com/GHEC, test.evidence.attach_media (default true, also implied by store_in_repo) uploads image/video artifacts that pass gh's attach rules to GitHub user-attachments at PR render time (internal/scm/github/attachments.go, attachRunEvidenceMedia in evidence_attach.go). Fail closed: upload errors keep today's rendering. Text artifacts stay inlined.
• internal/evidence owns the orphan-branch mechanism and its fail-closed rules: plumbing only (scratch GIT_INDEX_FILE + hash-object/write-tree/commit-tree), so HEAD, the index, and the worktree are untouched and a detached or shallow clone works; the parent is the just-fetched remote tip so the push is a plain fast-forward and never a force; an existing branch without the .no-mistakes-evidence marker at its tip is refused, which is what makes a wrong branch name (main) harmless. Every failure returns an error and the PR body omits links that would not resolve; an uploaded media attachment may still render, otherwise the artifact falls back to its local-path reference.
• PR links are pinned to the evidence COMMIT, not the branch, so a later run overwriting the same paths cannot change what an old PR shows. Link bases come from Repo.UpstreamURL/ForkURL, never the push URL, which can carry a credential.
• test.evidence.branch is trusted-only in EffectiveRepoConfig (it names a ref the daemon pushes to); local_root/retention/max_runs are global-only (applyEvidenceStorageOverrides is called from Merge with GlobalConfig alone); the rest of test.evidence (store_in_repo, attach_media, dir) stays pushed-readable. Invalid branch names, relative local_root, unparseable retention, and negative max_runs all fail the config at parse time (validateTestRaw).
• Regressions: internal/evidence/publish_test.go, internal/evidence/branch_test.go, internal/pipeline/steps/evidence_publish_test.go, internal/pipeline/steps/pr_attach_media_test.go, internal/scm/github/attachments_test.go, TestPushStep_DoesNotPublishTestEvidenceIntoThePushedBranch, TestEffectiveRepoConfig_EvidenceBranchTrustedOnly, TestLoadGlobalConfig_InvalidEvidenceBranchFailsClosed, internal/config/evidence_storage_test.go.

no-mistakes Owns Its Own Scratch (never the shared system temp dir)

• Evidence lives at <NM_HOME>/evidence/<runID> (paths.EvidenceDir/EvidenceRoot/RunEvidenceDir), never os.TempDir(). The daemon's service unit exports only HOME, PATH, and proxy vars, so TMPDIR is unset and os.TempDir() resolved to the shared /tmp - a systemd tmpfs on Ubuntu 24.10+, so evidence consumed RAM. The app root is disk-backed on all three platforms, so there is deliberately NO runtime.GOOS branch; do not add one.
• One owner for the path: the executor resolves it (Executor.runEvidenceDir) into StepContext.EvidenceDir, and agent.WithSteering(a, evidenceRoot) takes it as an argument. Steps and the steering preamble must never rebuild it - two independent os.TempDir() copies is exactly the drift this replaced.
• Cleanup is ours, in three layers: RunManager.cleanupRunEvidence removes a finished run's dir when empty (os.Remove, never RemoveAll - the test step creates the dir before the agent decides it has anything to write, and that litter was 94% of observed accumulation), reapEvidence bounds the directory by age and count oldest-first, and reapLegacyEvidence drains the pre-relocation temp directory under the same policy. All three reuse skipWorktreeCleanup's pending/running guard and are best effort. No OS temp timer is load-bearing.
• HELD SCOPE: internal/eval/replay.go sandboxes stay in the system temp directory. They are the largest scratch this program creates, but a replay materializes its own nested NM_HOME and worktree while Store.Prune, the case records, and the object pools all live under <NM_HOME>/eval - so relocating the sandbox inside the app root nests it in the state it is replaying, which e2e TestEvalJourney refuses on purpose. Moving it needs a disk-backed root outside NM_HOME, which does not exist yet; do not "fix" it by weakening that assertion. Every remaining os.MkdirTemp("", ...) caller is auto-named and self-cleaning with defer; keep it that way.
• Regressions: internal/paths/evidence_test.go, internal/config/evidence_storage_test.go, internal/daemon/evidence_reap_test.go, TestSteeringNamesTheConfiguredEvidenceRoot, TestTestEvidenceDir_DefaultResolutionStaysUnderTheAppRoot, e2e TestTestEvidenceLivesUnderAppRootNotSharedTemp / TestRunCleanupLeavesNoEmptyEvidenceDirectory.