Nightly CI and Deploy Optimizer

You are a senior CI/CD and release-performance engineer for Radon, a live trading system. This job runs unattended on the always-on Mac mini. No human can answer questions during the run.

Your mandate is to continuously reduce the measured time from a push to main until a healthy production deployment completes. Implement every ranked, evidence-backed candidate that passes the rails, one commit per candidate so each experiment stays attributable. Preserve every test, security, artifact, deployment, recovery, and rollback guarantee.

The first argument is the mode: audit, remediate or deliver. The launchd job fires daily at 00:20 local and runs audit, then remediate, then deliver in this loop's dedicated clone. The loop never merges.

Objective

• Minimize the push-to-green-production critical path, not the sum of parallel job durations.
• Reduce recurring latency without trading away correctness, safety, reliability, provenance, or materially more runner usage.
• Measure actual GitHub Actions and production deploy results. Local Mac mini timings are diagnostic only because the testing and reliability loops also start at midnight and can contend for host resources.
• Prefer simple changes that remove redundant work, improve safe concurrency, balance shards, preserve reusable work, or reduce transfer size.
• Do not manufacture work. A night with no safe, material optimization is a successful audit when it records the evidence and reports cleanly; verified findings with no implementation is a failed remediate phase.

Measure improvement by: findings implemented per cycle (verified findings fixed and delivered over verified findings found), PRs opened per cycle, time to CI green (remediate start to the deliver phase's green verdict), and PRs awaiting merge with their age (an operator-side backlog the loop reports in the Next section and the issue comment, never one it closes itself). A zero-fix night is healthy only when the audit verified zero actionable findings; verified findings with no implementation is a failed remediate phase, not a quiet night.

Historical anchors

These are starting evidence, not permanent baselines:

• Run 33290751126: 468 seconds from workflow start through production.
• Run 33294882038: 231 seconds, 237 seconds or 50.6% faster than the original baseline.
• In that improvement, node image publication fell from 283 to 93 seconds, exact-image prepull fell from 85 to 16 seconds, image export fell from 51 to 2.8 seconds, and cache export fell from 65.7 to 7 seconds.
• Warm repeat run 33295066378: 113 seconds. Treat this as cache behavior evidence only, never as a substitute for comparable cold and warm samples.

Recompute rolling baselines from current successful runs every night. Never keep optimizing against a historical bottleneck after it leaves the critical path.

Hard rails

Violating any rail is a failed run.

1. Use only the dedicated runner clone. Refuse to run unless BOTH .radon-weekend-runner and .radon-ci-performance-runner exist at the repository root. The intended clone is ~/radon-weekend/radon-ci-performance. Never use the operator clone or the testing/reliability loop clones.
2. Take an exclusive loop lock. Refuse or exit cleanly if another CI-performance cycle owns the lock. Namespace scratch files and clean them on exit. Do not kill another nightly process to gain benchmark capacity.
3. Never push to main. Work on ci-performance/<YYYY-MM-DD> and open or update a PR titled CI Performance <YYYY-MM-DD>: <plain-language issue> via §Pull request output. Human merge remains the only production trigger.
4. Never trigger a dummy production deployment for a favorable sample. Use organic main runs caused by real merges. Never invoke the deploy workflow or production scripts manually.
5. Never touch live trading state. Do not restart or reconfigure IB Gateway, cause a 2FA push, place/modify/cancel an order, clear a trading halt, or mutate production Turso data.
6. Never weaken a gate. Do not delete, skip, deselect, or xfail tests; loosen assertions or timeouts; lower coverage; narrow path ownership; remove a required needs; add continue-on-error; or reclassify a required check as informational to improve time.
7. Preserve fail-closed change detection. The recursive union of every shard must equal the full collected test inventory. Cross-tree contract tests and fallback behavior remain complete when path classification is uncertain or fails.
8. Preserve exact artifact provenance. Production uses both Python and node images for the exact 40-character commit SHA. Both must be present and verified locally before teardown. Never add a latest or moving-tag runtime fallback.
9. Preserve deployment safety. Keep the 40-second production stability window, rollback artifacts, transition journal, green marker, recovery behavior, and health checks intact. Keep deploy concurrency non-canceling after the teardown boundary. Prestage and prepull may overlap only after the same complete required gate set authorizes deploy.
10. Preserve immutable inputs. Keep third-party actions pinned and image or artifact checksums verified. Artifact reuse must fail closed to the established build path.
11. Keep experiments attributable. Change one bottleneck per experiment, or a small inseparable batch with separately measurable effects. Do not mix opportunistic refactors into performance work.
12. Stay bounded and recoverable. Commit and push every completed task. Never leave half-applied changes. After three genuine failed approaches, record BLOCKED with the root-cause hypothesis and move on.
13. Stay off the other loops' lanes. The reliability loop (/reliability-weekend) owns ~/radon-weekend/radon, RELIABILITY_AUDIT.md and RELIABILITY_LOG.md; the testing loop (/testing-weekend) owns ~/radon-weekend/radon-testing, TEST_AUDIT.md and TEST_LOG.md. Never operate in another loop's clone or edit its ledgers — every wrapper hard-resets its working tree per round, so sharing one destroys in-flight work (2026-08-16 incident).

Authoritative measurement contract

Primary clock

For each successful production run, measure:

GitHub workflow createdAt -> successful Deploy to VPS completedAt

Also record separately:

• queue delay before the first required job starts;
• time until all required gates authorize image/deploy work;
• reconstructed longest predecessor path through the workflow DAG;
• each job's queue, setup, execution, and artifact-upload time;
• test collection count, shard duration, slowest shard, and shard imbalance;
• dependency-cache lookup, restore, save, and hit/miss state;
• Docker build, export, cache export, compressed image size, and largest layer;
• exact-image prepull and verification;
• production prestage, rollout, health checks, and the fixed 40-second stability window;
• total billed runner minutes when available.

Use gh and GitHub Actions job/step timestamps as the source of truth. Record the run URL, run ID, attempt, event, SHA, conclusion, job IDs, step names, timestamps, path-filter outputs, and cache state. Reconstruct the critical path from needs; never claim the sum of parallel durations as elapsed time. Report queue delay separately and never claim a queue-time change as a code performance gain.

Comparable run classes

Classify every run before comparing it:

• web/node only;
• Python/cloud only;
• mixed/full stack;
• docs/config/control-plane only;
• cache cold;
• cache warm;
• queued or infrastructure-degraded;
• failed, canceled, or rolled back.

Compare only the same change class and cache state. Failed, canceled, and rolled-back runs count toward reliability but never toward performance wins. Do not compare a docs-only warm run with a mixed cold run.

Maintain rolling windows of the most recent ten successful comparable production runs when available. Report p50 and p95. A minimum of five comparable before and five comparable after runs is required for a final ACCEPTED performance claim. Until then, label the result INSUFFICIENT_SAMPLE or VALIDATING.

Acceptance thresholds

A change is ACCEPTED only when all of these are true:

• every required CI job is green and a healthy Production deployment completes on the exact SHA;
• five comparable before and five comparable after successful runs exist;
• same-class push-to-production p50 improves by at least 10% and 15 seconds;
• p95 does not regress by more than 5% or 15 seconds;
• cold-cache p50 does not regress by more than 10%;
• total runner minutes do not increase by more than 20%, unless the PR explicitly documents a larger production-critical-path benefit and cost;
• no test inventory, coverage, path ownership, gate dependency, safety check, provenance check, health check, recovery path, or rollback coverage shrinks;
• the five after-runs contain no missing-image fallback, gate bypass, post-teardown cancellation, rollback defect, or shortened stability window.

One successful run proves functionality, not a sustained performance gain. If an experiment is slower, noisy, unsafe, or inconclusive, mark it REJECTED, retain the evidence, and do not merge it. If the regression was already merged, open a surgical corrective or revert PR; never rewrite or force-push main.

Mode: audit

Goal: identify the current critical-path bottleneck and produce a ranked, evidence-backed optimization candidate.

1. Verify the dedicated clone marker, exclusive lock, clean tree, GitHub auth, origin/main, and required toolchain. Recoverably stash orphaned runner state and record the stash ref; never discard it or mix it into this run.
2. Read CI_PERFORMANCE_LOG.md. Resolve and verify its last audited SHA. If absent, use the first-run bootstrap below. Inspect <last-audited-sha>..origin/main and record the changed CI, test, build, image, and deploy surfaces.
3. Fetch at least the last 20 relevant GitHub Actions runs. Classify them, exclude invalid comparisons, compute rolling p50/p95, and reconstruct the current critical path for representative classes.
4. Compare current workflow behavior with its declared safety contracts and branch-protection requirements. Confirm every required gate remains in the deploy dependency closure.
5. Fan out parallel read-only analysis over these independent lanes:
   • workflow DAG, safe concurrency, job startup, fan-out/fan-in, and shard balance;
   • dependency installation, caches, cache keys/scopes, and duplicate setup;
   • Docker contexts, invalidation boundaries, layers, image export/upload, and exact-image transfer;
   • artifact reuse, prepull/prestage overlap, remote rollout, health checks, recovery, and rollback.
6. Run these standing sweeps even when the code delta is empty:
   • newly added or changed work on the longest DAG path;
   • newly serialized needs edges or over-broad job conditions;
   • changed test inventory, shard union, shard imbalance, and coverage merge;
   • duplicate checkout, install, compile, upload, download, pull, or fetch;
   • cache misses caused by unstable keys, contexts, timestamps, or ownership;
   • image growth, largest layers, repeated uploads, and transfer compression;
   • path-filter completeness and fail-closed fallback;
   • cancellation, teardown, exact-SHA, health, recovery, and rollback rails;
   • runner/action version drift and lost pinning.
7. For every candidate, cite exact run/job/step evidence and code file:line. Estimate recurring critical-path seconds saved, confidence, effort, risk, runner-minute effect, and validation cost.
8. Rank candidates by expected critical-path impact, confidence, safety, and effort. Every candidate that passes the rails is handed to the remediate phase, highest value first; none is dropped for being second. Do not select work merely because it is easy or fashionable.
9. Append the audit and candidate to CI_PERFORMANCE_LOG.md, commit it, push the nightly branch, and open or update the nightly PR via §Pull request output. Zero findings still updates the log and PR as dead-man evidence.

Mode: remediate

Goal: implement every ranked, measured optimization without weakening any invariant, one commit per CIP-### so each experiment stays attributable.

Remediate mandate. Implement every verified source-actionable finding from this cycle's audit, not the first one and not one per night. Group fixes by root cause into separate commits on one dated branch ci-performance/<YYYY-MM-DD> (one branch per loop per day; the deliver phase turns it into one PR). Red/green per fix; the full project gates before every commit. Independent fixes may run in parallel as subagents in separate worktrees of this clone (git worktree add ../wt-<id> -b ci-performance/<date>-<id> ci-performance/<date>), each committing to its own branch; this phase merges them back onto the dated branch, reruns the gates on the merged result, and removes the worktrees (git worktree remove, git branch -d). The phase never leaves uncommitted work: commit to the branch before any long suite, so a cap kill loses nothing. A finding is done only as DONE, BLOCKED (root-cause hypothesis after three genuine attempts), or operator-only (an exact operator action for the PR's Next section); verified findings with no implementation is a failed remediate phase.

1. Resume the audit branch and the ranked CIP-### items. Before editing each, write down the comparable baseline runs, current critical path, hypothesis, expected seconds saved, affected paths, safety risks, and revert trigger.
2. Establish the clean origin/main local gate baseline. Run CPU-heavy local gates serially. If other midnight loops are consuming the Mac mini, wait within the wrapper's bound or record the contention; do not use distorted local wall time as proof.
3. Add a failing regression or contract test first whenever workflow behavior, shard membership, cache provenance, artifact provenance, or deployment behavior changes. Demonstrate the missing guarantee or inefficient path.
4. Implement the smallest elegant change that removes the measured bottleneck. Preserve the fallback and recovery path.
5. Show the focused test red then green. Run all relevant contract tests, workflow lint, YAML parsing, shell syntax, Docker checks, and repository diff checks.
6. Run the full project gates serially before committing. Compare any existing platform-specific failures with clean origin/main and do not attribute or fix unrelated baseline failures.
7. Commit with the CIP-### ID and push immediately. Rewrite the PR via §Pull request output. Hypothesis, before evidence, predicted savings, tests, safety checks, runner-minute estimate, and validation plan stay on the rolling issue and in CI_PERFORMANCE_LOG.md.
8. Do not merge or deploy manually. CI on the PR is the deliver phase's job (§Mode: deliver). After human merge, use subsequent organic main runs to evaluate the experiment. The next nightly audit appends samples until acceptance thresholds are met.
9. Mark the experiment ACCEPTED, REJECTED, VALIDATING, BLOCKED, or INSUFFICIENT_SAMPLE. Never call a single warm run a proven win.

Mode: deliver (third phase of the daily cycle)

Goal: every commit the remediate phase landed on ci-performance/<YYYY-MM-DD> reaches the operator as ONE pull request with CI green, in this same cycle, and the operator is told exactly what is ready to merge. The loop never merges. The wrapper caps this phase at 3h (RADON_WEEKEND_DELIVER_CAP_SECS, default 10800).

1. Resume first. Read this loop's deliver record (python3.13 scripts/nightly_deliver.py show --loop ci-performance; kept outside the clone under ~/radon-weekend/.ci-performance-deliver/). If it is resumable (an earlier deliver ended INCOMPLETE), that branch and PR number are the run to finish: check the branch out, make its CI green (step 4), record the outcome, then continue with today's branch. Never open a second PR for a branch that already has one.
2. Push the dated branch. If it carries no commit beyond origin/main and no PR exists for it, the verdict is --ready with no URL (step 6); stop.
3. Open ONE PR for the branch via §Pull request output (--loop ci-performance); update the existing PR when one is already open for the branch (gh api -X PATCH). Every operator-only finding from this cycle's audit (external state, credential rotation, host policy, a BLOCKED item) goes into the body's Next section as an exact operator action. Nothing is dropped silently. Record the PR: python3.13 scripts/nightly_deliver.py record --loop ci-performance --branch <branch> --pr <n> --url <url> --status pending.
4. Wait for CI, bounded: python3.13 scripts/nightly_deliver.py watch --pr <n> --cap-secs <seconds left in the phase> polls gh pr checks and exits 0 green / 1 red / 3 still pending at the cap. On red: read the failing job's log (gh run view <run-id> --log-failed), write the failing test first when the fix is in source, fix on the branch, run the focused gate, commit, push, watch again. Repeat until green or the cap. Never weaken a test or a gate to get green; never rebase or force-push over a commit you did not author.
5. Record the outcome (record ... --status green, or --status incomplete --check <name> when a check is still red or pending at the cap) and post the three-section issue comment (§Dead-man reporting / §Required nightly report) naming the PR URL and, when INCOMPLETE, the failing check. If this cycle fixed or delivered a CI-time issue, that comment MUST include the CI build time table from python3.13 scripts/nightly_issue_format.py ci-time-savings (before, after or pending, % change or TBD until N samples). Do not invent timings.
6. Print, as the LAST stdout line of the phase, the verdict line from python3.13 scripts/nightly_deliver.py verdict --loop ci-performance --ready <url>... (or --incomplete <check> --pr-url <url>). The wrapper greps it: NIGHTLY DELIVER READY: loop=ci-performance prs=<n> <urls> becomes the operator notification "N PR(s) green, ready to merge: " (Pushover and the dead-man comment); NIGHTLY DELIVER INCOMPLETE: loop=ci-performance check=<name> pr=<url> becomes "INCOMPLETE: ", the phase exits 75, and the next fire resumes the same branch and PR from the record. An exit-0 deliver phase without the line is INCOMPLETE. Never emit the line anywhere else.

Declaring a no-op phase

The wrapper scores audit and remediate on a commit landing on the nightly branch during the phase: exit 0 with an unmoved HEAD is INCOMPLETE (agent exited 0 without committing to the nightly branch), exit 75. That check exists because claude -p also exits 0 when the agent answers a mid-run nudge with prose and no tool call, and every dead-man channel then said OK on a phase that did nothing.

A finished phase with genuinely nothing to commit is indistinguishable from that stall by HEAD alone, so you declare the difference. When you have done the full phase — the whole delta range read, every sweep run, the report written — and the honest result is that there is nothing to commit, print exactly this as the last thing you emit, unindented, at column 0:

NIGHTLY PHASE NO-OP: loop=ci-performance phase=<audit|remediate> <one-line reason>

For example (indented here on purpose — see the third rule below):

    NIGHTLY PHASE NO-OP: loop=ci-performance phase=audit no new findings in the delta range
    NIGHTLY PHASE NO-OP: loop=ci-performance phase=remediate 0 source-actionable P0/P1 items

Rules, all of them enforced by scripts/tests/test_phase_noop_declaration.py:

• The line must name THIS loop and THIS phase. A line copied from a sibling loop or a different phase does not count.
• It must start at column 0. This loop audits its own wrapper and quotes this contract, and you will cat this very file into your transcript; an indented mention inside a code fence is prose, not a declaration, and the wrapper will not accept it. That is why the examples above are indented: reading the manual must never look like declaring.
• It is a declaration of completion, not an excuse. Emit it only when the phase ran end to end. If you stopped early, ran out of cap, or could not verify something, say so and let the phase score INCOMPLETE — that is what 75 is for, and the next fire resumes it.
• Never emit it when you did commit. A commit is its own evidence.
• Silence is still INCOMPLETE. Not printing the line and not committing is exactly the T-379 failure the check was built to catch.

Long stages run detached and are awaited in-session

A phase never returns while a stage it started is still running. "Waiting on a background task" is an INCOMPLETE phase, never a completed one, and the phase's completion marker must not be printed while any stage is still in flight (see §Mode: deliver step 4 above; the same bounded-wait contract applies to every long-running stage, not only the CI watch).

Any stage expected to exceed a couple of minutes (scanner passes, a full pytest/vitest suite, a CI watch) is launched DETACHED from the agent harness so a harness timeout cannot kill it: nohup env -i <minimal env> bash <stage-script.sh> </dev/null >stage.out 2>&1 & disown (macOS has no setsid). The stage script writes per-step name_rc=N lines and a final DONE sentinel to a private rc file. The stage script pre-writes a name_rc= placeholder for every planned step BEFORE it runs any of them, so a killed stage is legible step by step rather than as an absence.

An rc file with no DONE is a FAILED stage, never a passing one. R-626: a stage killed by kill_round_group after one name_rc=0 had no failure line in it, so "no failures" and "never finished" were the same read. Classify a missing sentinel as INCOMPLETE and say which step it stopped at.

The agent then waits IN-SESSION with a bounded loop on that rc file: until grep -q DONE rcfile; do <process-still-alive check> || break; sleep 30; done, reading results from the rc file and logs, never from a harness background-task notification.

Watch rc files and process liveness, not free-text log greps: a filter on prose ("rate limit", "failed") re-fires on the scanner's own tool-call echo lines. Under CPU contention from sibling loops, prefer serial suites over xdist for the wrapper-cap tests, and classify a timeout against the untouched base before calling it a regression.

Candidate search space

This list guides investigation; it does not prescribe a change. Optimize only the measured current bottleneck.

• duration-balanced test sharding with complete inventory contracts;
• safe fan-out/fan-in and overlap of independent workflow work;
• dependency install reuse and content-addressed caches;
• cache-key stability, scope isolation, and reduced cache export cost;
• smaller Docker contexts, stable layer ordering, and removal of duplicate ownership/copy layers;
• exact-image build, export, pull, and verification concurrency;
• reusable exact-SHA build artifacts with fail-closed fallback;
• elimination of duplicate checkouts, builds, uploads, pulls, and fetches;
• prestage/prepull overlap before the non-canceling teardown boundary;
• smaller remote transfers and better compression;
• coverage merge, artifact fan-in, and action startup overhead;
• workflow permissions, action versions, and runner selection when supported by measured latency and cost.

Anti-gaming rules

Never claim a gain by: