Auto Review Loop: Autonomous Research Improvement

🔒 Do not wrap this skill in /loop, /schedule, or CronCreate. It already loops internally (review → fix → re-review) and the reviewer carries round-to-round memory in one threadId (codex-reply). An external timer re-enters from the top each tick — fresh threadId, reviewer memory reset — firing the verdict on wall-clock time instead of on artifact change: zero new signal, full token cost. If you want to schedule something, schedule the external wait that precedes it (experiments done → then run this once). See shared-references/external-cadence.md.

Autonomously iterate: review → implement fixes → re-review, until an independent reviewer gives a policy-approved positive assessment or MAX_ROUNDS is reached.

Context: $ARGUMENTS

Constants

• MAX_ROUNDS = 4
• POSITIVE_THRESHOLD: score >= 6/10 AND verdict ∈ {"ready", "almost"} — both must hold. This matches the operative Phase-E STOP CONDITION exactly; the verdict vocabulary is {"ready", "almost", "not ready"} (a high score with a "not ready" verdict does NOT stop the loop). Earlier wording here used or and a stale verdict set ("accept"/"sufficient"/"ready for submission") — that was an internal inconsistency; the AND form is authoritative.
• REVIEW_DOC: review-stage/AUTO_REVIEW.md (cumulative log) (fall back to ./AUTO_REVIEW.md for legacy projects)
• REVIEWER_MODEL = gpt-6-astra — Default model for the Codex backend. Must be an OpenAI model (e.g., gpt-6-astra, o3, gpt-4o). Manual backend uses a model the user chooses — it must be a recognized model from a different family (OpenAI, Anthropic, Google, DeepSeek, Moonshot/Kimi, Qwen).
• REVIEWER_BACKEND — With no reviewer directive, start as auto; Step -1 runs exactly one two-call native marker/challenge probe for the first review. A bound Copilot CLI root session uses copilot-native (built-in complementary rubber-duck subagent); an unbound/non-Copilot host keeps the existing codex default. Explicit — reviewer: codex, oracle-pro, agy, or manual bypasses the probe and selects that external backend. Explicit — reviewer: copilot retains the compatibility copilot --agent drive mode and its later Codex/manual finalizer. The native path gets both actual model IDs from host session events; it never needs COPILOT_CLI or caller-provided --executor-model. See shared-references/reviewer-routing.md.
• OUTPUT_DIR = review-stage/ — All review-stage outputs go here. Create the directory if it doesn't exist.
• HUMAN_CHECKPOINT = false — When true, pause after each round's review (Phase B) and present the score + weaknesses to the user. Wait for user input before proceeding to Phase C. The user can: approve the suggested fixes, provide custom modification instructions, skip specific fixes, or stop the loop early. When false (default), the loop runs fully autonomously.
• COMPACT = false — When true, (1) read EXPERIMENT_LOG.md and findings.md instead of parsing full logs on session recovery, (2) append key findings to findings.md after each round.
• REVIEWER_DIFFICULTY = medium — Controls how adversarial the reviewer is. Three levels:
  • medium (default): Current behavior — MCP-based review, the executor controls what context the reviewer sees.
  • hard: Adds Reviewer Memory (the reviewer tracks its own suspicions across rounds) + Debate Protocol (the executor can rebut, the reviewer rules).
  • nightmare: Everything in hard + Codex exec reviewer reads the repo directly via codex exec (the executor cannot filter what the reviewer sees) + Adversarial Verification (the reviewer independently checks if code matches claims).
• RENDER_HTML = true — When true (default), auto-render review-stage/AUTO_REVIEW.md to HTML on loop termination via /render-html. Uses --no-review (the loop itself IS the cross-model review; the HTML is a structural conversion). Set false to skip, or pass — render html: false.

⚠️ Nightmare + Manual incompatibility: If REVIEWER_BACKEND = manual and REVIEWER_DIFFICULTY = nightmare, STOP with: "difficulty: nightmare requires Codex CLI / codex exec and is not compatible with --reviewer: manual. Use difficulty: hard, or switch reviewer to codex."

💡 Override: /auto-review-loop "topic" — compact: true, human checkpoint: true, difficulty: hard

Reviewer Calling Convention

When calling the reviewer, branch on REVIEWER_BACKEND:

If no --reviewer: directive was supplied: Set REVIEWER_BACKEND to auto. At Step -1 of the first round, resolve copilot_native_evidence.py using the canonical four-layer helper chain. Generate a fresh binding <run_id>_r<round>_review_<8-random-hex> and invoke marker, wait, then invoke challenge as two distinct root Bash calls. Put the literal binding and concrete resolved helper path in both calls; Copilot Bash calls do not share variables. If the challenge binds, set REVIEWER_BACKEND to copilot-native and use that same challenge for the first review. Do not issue a second activation challenge in Phase A. If it exits 3 because no current Copilot root session is bound, use codex. Explicit reviewer directives bypass this probe. If the helper is missing, native acceptance is unavailable; use Codex only if that external backend is positively available, otherwise emit REVIEW_UNAVAILABLE.

If REVIEWER_BACKEND = copilot-native: Read the challenge nonce and host-reported executor model. Invoke the host's native task tool with agent_type: rubber-duck; do not start a subprocess and do not specify a reviewer model. The prompt contains the exact standalone ARIS_REVIEW_NONCE=<nonce> line, artifact/diff paths, the output contract, and (round 2+) review-stage/REVIEWER_MEMORY.md. It contains no executor summary or fix narrative. After the task completes, invoke copilot_native_evidence.py verify to create the evidence and raw-response artifacts. The verifier must observe one successful linked rubber-duck lifecycle and known, different host-reported model families.

Pass the evidence to both review_gate.py --native-evidence and save_trace.sh --backend copilot-native --native-evidence. A qualifying native positive may stop directly; no external finalizer is needed. A native negative continues with a fresh marker/challenge/subagent next round. Every verdict-bearing native call—including a hard-mode rebuttal ruling—gets one unique <run_id, round, purpose> artifact set and exactly one challenge. Missing, same/unknown-family, malformed, stale, or mismatched evidence is never a verdict. If native complementary dispatch is unavailable, fall back only to a positively available opposite-family backend: Anthropic/Google executor → Codex; OpenAI executor → manual with a reported non-OpenAI model. Otherwise emit REVIEW_UNAVAILABLE. Full protocol: shared-references/reviewer-routing.md.

If REVIEWER_BACKEND = copilot: Require --executor-model: if not provided → emit REVIEW_UNAVAILABLE. Determine executor family from --executor-model (see reviewer-routing.md). Router picks opposite-family profile:

• executor_family=openai → profile="aris-reviewer-claude" (anthropic)
• executor_family=anthropic → profile="aris-reviewer-openai" (openai)
• executor_family=google → profile="aris-reviewer-openai" (openai, default cross)
• executor_family=unknown → REVIEW_UNAVAILABLE (fail closed). Verify the profile file exists at .github/agents/<profile>.agent.md. If missing → REVIEW_UNAVAILABLE. Read its model: field into REVIEWER_MODEL, derive reviewer_family from that model string, and verify it differs from executor_family. Pass the same value through subprocess --model; never trust a caller-supplied family label or profile-only pinning under an Auto session. Identity assurance: --executor-model is caller-declared routing input, not runtime attestation. Record executor_model_source: caller-declared, the derived family_relation, and independence_verified: unverified. A pair of different model strings must never be promoted to independently verified. Capability gate: copilot --help must advertise --model, --effort, and --allow-tool; otherwise emit REVIEW_UNAVAILABLE. Use the copilot --agent subprocess (documented Copilot CLI form) with the selected profile, --model "$REVIEWER_MODEL", --effort xhigh, and --allow-tool=read for each review call. Multi-round: each round is a fresh copilot --agent call with the same profile; reviewer memory is carried via review-stage/REVIEWER_MEMORY.md artifact. If copilot CLI is unavailable → REVIEW_UNAVAILABLE for that drive round; do not silently substitute another transport. A later positive Copilot verdict still requires the separately documented Codex/manual finalizer. See shared-references/reviewer-routing.md for the full copilot contract.

If REVIEWER_BACKEND = codex: Use mcp__codex__codex for new review threads. Use mcp__codex__codex-reply for follow-up rounds (reuse threadId).

If REVIEWER_BACKEND = manual: Use mcp__manual_review__review for new review threads with: prompt: [exact same prompt that would go to Codex] config: {"model_reasoning_effort": "xhigh", "executor_model": "", "require_reviewer_model": true} Save the returned threadId. Use mcp__manual_review__review_reply for follow-up rounds with: threadId: [saved manual-review threadId] prompt: [follow-up prompt] config: {"model_reasoning_effort": "xhigh", "executor_model": "", "require_reviewer_model": true} A verdict-bearing manual response MUST begin with Reviewer-Model: <exact-model-id>. Derive reviewer_family from that model identity. Missing, unknown, or same-family identity cannot acquit; for a mandatory escalation, emit REVIEW_UNAVAILABLE rather than guessing.

Prompt fidelity: the manual review task must be exactly the same text that Codex would receive; the transport may add only the required Reviewer-Model: response-format instruction. Review tracing applies to every backend. Native traces are populated from the revalidated host-event artifact rather than caller model declarations.

State Persistence (Compact Recovery)

Long-running loops may hit the context window limit, triggering automatic compaction. To survive this, persist state to review-stage/REVIEW_STATE.json after each round:

{
  "run_id": "run_20260713_a1b2c3d4",
  "round": 2,
  "threadId": null,
  "reviewer_profile": "rubber-duck",
  "reviewer_backend": "copilot-native",
  "executor_model": "claude-sonnet-4.6",
  "executor_model_source": "host-session-event",
  "executor_family": "anthropic",
  "requested_reviewer_model": null,
  "reported_reviewer_model": "gpt-5.5",
  "reviewer_model_source": "host-session-event",
  "reviewer_family": "openai",
  "family_relation": "different",
  "identity_assurance": "host_event_verified",
  "independence_verified": true,
  "native_evidence_id": "cne_0123456789abcdef0123456789abcdef",
  "native_evidence_path": "review-stage/COPILOT_NATIVE_run_20260713_a1b2c3d4_ROUND_2_REVIEW.evidence.json",
  "requires_external_acquittal": false,
  "status": "in_progress",
  "difficulty": "medium",
  "last_score": 5.0,
  "last_verdict": "not ready",
  "pending_experiments": ["screen_name_1"],
  "timestamp": "2026-03-13T21:00:00"
}

• run_id — Globally unique per invocation. Generated on fresh start as run_<YYYYMMDD>_<8-char-hex> (e.g., run_20260713_a1b2c3d4). Preserved across round writes. On resume, read from state file unchanged. This binds all round state, reviewer-memory appends, and acquittal receipts to one run so a stale completed state from a previous invocation cannot leak into the current run's acquittal check.

When REVIEWER_BACKEND = copilot-native, save the evidence ID/path and the host-event executor/reviewer models, derived families, and sources. Each round is a fresh rubber-duck subagent and therefore gets a fresh evidence artifact; there is no persistent child handle. When REVIEWER_BACKEND = compatibility copilot, retain reviewer_profile, requested model, caller-declared executor model, independence_verified: "unverified", and the external-finalizer obligation. For codex save its MCP threadId; for manual save threadId and the reported reviewer identity. On resume, use reviewer_backend to select the continuation mechanism and preserve requires_external_acquittal.

Write this file at the end of every Phase E (after documenting the round). Overwrite each time — only the latest round's state matters. The run_id field MUST persist unchanged across overwrites within the same run.

On completion (positive assessment or max rounds), set "status": "completed" so future invocations don't accidentally resume a finished loop.

Append-Only External-Finalizer Receipt

Whenever a Copilot path hands the verdict to an external backend—after a positive compatibility-drive review or after a pre-verdict native dispatch failure—maintain an append-only finalizer log at review-stage/ACQUITTAL_LOG.jsonl. Each line records the Codex/manual reviewer that completed that run. A successful native rubber-duck round never needs or writes this receipt; its evidence sidecar is the acceptance record. The historical filename is retained for compatibility:

{"run_id":"run_20260713_a1b2c3d4","round":3,"backend":"codex","effort":"xhigh","verdict":"ready","score":7.5,"executor_model":"claude-sonnet-4-5","executor_model_source":"caller-declared","executor_family":"anthropic","reviewer_model":"gpt-6-astra","reviewer_model_source":"requested","reviewer_family":"openai","family_relation":"different","identity_assurance":"caller_declared","independence_verified":"unverified","trace_id":"auto-review-loop/2026-07-13_run03","timestamp":"2026-07-13T14:22:00Z"}

Rules (non-negotiable):

Rule	Detail
Append-only	Never delete, never truncate, never overwrite lines. Only >>.
Who writes	Only a codex or manual round at xhigh effort when round_requires_external_acquittal was true. A Copilot review/dispatch never writes a finalizer line itself.
When to write	At the end of Phase E, after the policy-approved finalizer returns score >= 6 AND verdict ∈ {"ready", "almost"}. A normal default-Codex run does not need this sidecar.
run_id binding	Every line carries the current run_id and round so the Copilot → finalizer transition is auditable.
Trace linkage	trace_id MUST reference the real trace artifact in .aris/traces/; source and family fields in the receipt must exactly match that trace.
Identity honesty	Re-derive family_relation from the model strings, but preserve their sources. With the current caller-declared executor identity, write identity_assurance: "caller_declared" and independence_verified: "unverified"; never promote different strings to independent attestation.
No overwrite	REVIEW_STATE.json is overwritten each round (only latest state). ACQUITTAL_LOG.jsonl is NEVER overwritten — it is the permanent, cumulative record.

Why this exists: REVIEW_STATE.json is overwritten each round. The log preserves evidence that a compatibility drive verdict or failed native attempt did not terminate by itself. A successful copilot-native verdict instead uses its host-event evidence sidecar.

Output Protocols

Follow these shared protocols for all output files:

• Output Versioning Protocol — write timestamped file first, then copy to fixed name
• Output Manifest Protocol — log every output to MANIFEST.md
• Output Language Protocol — respect the project's language setting

Workflow

Initialization

1. Check for review-stage/REVIEW_STATE.json (fall back to ./REVIEW_STATE.json if not found — legacy path):
   • If neither path exists: fresh start (normal case, identical to behavior before this feature existed)
     • Generate run_id: run_<YYYYMMDD>_<8-char-hex> (e.g., run_20260713_a1b2c3d4). Use date +%Y%m%d and 8 random hex characters. This run_id persists across all round writes and binds acquittal receipts to this invocation.
   • If it exists AND status is "completed": fresh start (previous loop finished normally — but its ACQUITTAL_LOG.jsonl entries are retained as an audit trail with their own run_id, and are NOT valid for the current run's stop gate)
     • Generate a new run_id for this invocation.
   • If it exists AND status is "in_progress" AND timestamp is older than 24 hours: fresh start (stale state from a killed/abandoned run — delete the file and start over)
     • Generate a new run_id for this invocation.
   • If it exists AND status is "in_progress" AND timestamp is within 24 hours: resume
     • Read the state file to recover run_id, round, threadId (or evidence/profile fields for Copilot backends), reviewer_backend, last_score, pending_experiments
     • Legacy backward compat: if reviewer_backend is absent from the state file, default to codex (pre-copilot-era states did not record this field). If requires_external_acquittal is absent, default it to false; a legacy default-Codex run must not inherit the stricter Copilot-finalizer state. If run_id is absent from the state file (pre-run_id era), generate a new run_id and log: "No run_id in legacy state file; assigned run_<...> for this resume."
     • Read review-stage/AUTO_REVIEW.md to restore full context of prior rounds (fall back to ./AUTO_REVIEW.md)
     • If pending_experiments is non-empty, check if they have completed (e.g., check screen sessions)
     • Resume from the next round (round = saved round + 1)
     • Use reviewer_backend to determine continuation: codex-reply for codex; a fresh marker/challenge/rubber-duck/evidence cycle for copilot-native; a fresh copilot --agent subprocess with the saved profile/model for compatibility copilot; manual_review_reply for manual
     • Log: "Recovered from context compaction. Resuming at Round N."
2. Read project narrative documents, memory files, and any prior review documents. When COMPACT = true and compact files exist: read findings.md + EXPERIMENT_LOG.md instead of full review-stage/AUTO_REVIEW.md and raw logs — saves context window.
3. Read recent experiment results (check output directories, logs)
4. Identify current weaknesses and open TODOs from prior reviews
5. Initialize round counter = 1 (unless recovered from state file)
6. Create/update review-stage/AUTO_REVIEW.md with header and timestamp
7. If this is a fresh run with no explicit reviewer directive, initialize REVIEWER_BACKEND to auto. Step -1 of Round 1 performs activation and uses that same challenge for the review. Explicit reviewer directives initialize their selected backend and bypass activation. Do not use environment heuristics.

Loop (repeat up to MAX_ROUNDS)

Step -1 — Resolve the automatic backend and prepare one native challenge:

• If REVIEWER_BACKEND is auto, resolve the native helper and run one root marker call followed by one root challenge call. Use binding <run_id>_r<round>_review_<8-random-hex> and output review-stage/COPILOT_NATIVE_<run_id>_ROUND_<round>_REVIEW.challenge.json. A bound challenge sets REVIEWER_BACKEND to copilot-native and NATIVE_CHALLENGE to that path. Exit 3/unbound sets REVIEWER_BACKEND to codex. Any other failure follows the fail-closed capability rules.
• If REVIEWER_BACKEND is already copilot-native (a later round or a resumed run), create one fresh marker/challenge pair with the same run-scoped naming pattern and set NATIVE_CHALLENGE. An unbound or invalid challenge cannot be treated as a verdict or silently relabeled.
• Explicit external or compatibility backends do nothing in this step.

The challenge created here is the challenge consumed by Phase A. Do not run another marker/challenge for the same review. Run-scoped filenames are append-only audit identities; never pass --replace to reuse evidence from an older invocation.

Step 0 — Snapshot current-round state: After Step -1 resolves auto, set round_backend = <current REVIEWER_BACKEND> and round_requires_external_acquittal = <current requires_external_acquittal, default false>. These variables label the backend and obligation that actually governed the CURRENT round. If compatibility-drive escalation occurs later in Phase B.5.1 (copilot → codex/manual), the snapshots retain their pre-escalation values while the forward-looking state is updated for the NEXT round. A native dispatch failure is different because no review occurred: replace both snapshots with the external fallback values before that reviewer call, as specified in Phase A. A successful native call never sets the finalizer obligation. Phase E uses only the resulting snapshots when documenting or writing a finalizer receipt.

Phase A: Review

Route by REVIEWER_BACKEND and REVIEWER_DIFFICULTY.

If REVIEWER_BACKEND = copilot-native, execute one fresh native cycle: