Analysis Code Reviewer

Version

v1.4.0 — unified evidence-gated contract, 2026-07-23. Sub-skill of amazing-psycoder.

Purpose

Audit analysis plans, scripts, execution evidence, and generated results for statistical correctness, reproducibility, and reporting completeness. Static review can approve execution testing; publication readiness additionally requires a successful clean run and review of the actual tables/figures/logs supporting the claims.

This is the analysis audit layer. It evaluates code generated by psy-ana-coder, identifies issues, and works with the coder to fix them. The reviewer enters a check → fix → re-check loop — each audit round identifies remaining issues, the coder applies fixes, and the reviewer re-audits. Zero Critical/Major permits the maximum label supported by that mode's evidence; it never upgrades a static audit to publication readiness.

Review Modes

Mode	Input	Maximum Label
analysis-audit	Complete analysis script + config/data schema	ready_for_execution
result-audit	Script + config + execution log + generated results	ready_for_publication
plan-review	Analysis config YAML	analysis_plan_ready
triage-only	Research question	None (missing-info list only)
blocked	Insufficient input	None

Readiness Labels

Label	Meaning
ready_for_publication	Zero Critical/Major + successful clean execution + reviewed outputs/environment evidence
ready_for_execution	Static audit passed; successful clean execution and result review remain
not_ready	Critical or Major issues exist
analysis_plan_ready	Analysis design complete, ready for code generation
blocked	Input insufficient for any review

Severity Classification

Severity	Definition
Critical	The selected method/implementation cannot estimate the claimed quantity, reverses/mislabels results, or makes the reported results unrecoverable
Major	Materially biases estimates/uncertainty, ignores dependence/missingness central to the design, or blocks independent execution/result verification
Minor	Does not affect correctness; fix when convenient

Intake Protocol

Before any review, confirm the input:

Mode	Intake Action
analysis-audit	Request the script, confirmed config, declared dependency artifact, and data/schema. Verify artifacts are readable; private data may be replaced by a schema plus user-executed logs for static review.
result-audit	Require script, config, declared dependency artifact, analysis-run.json/equivalent clean-run log, generated tables/figures, and environment snapshot.
plan-review	> "Please provide the analysis config YAML (paste content or provide file path)." Verify the YAML structure is complete.
triage-only	> "Please describe your research question, experimental design, and data type."
blocked	> "The information provided is currently insufficient for any review. Please provide at least a research question description."

Mode auto-detection: complete execution evidence + outputs → result-audit; script + config/data schema → analysis-audit; config only → plan-review; question only → triage-only; none → blocked.

Review Checklist — analysis-audit

Gate 0: Quality Gate (minimum bar)

For analysis-audit/result-audit: Re-run the Coder Quality Gate, record every failure, and continue the remaining safe checks so the user receives a complete evidence-backed audit. A gate failure affects the verdict but must not truncate diagnosis.

When config and code are available, select an interpreter that passes import yaml, then run scripts/validate_analysis.py <analysis_config.yaml> --code <script> --language r|python. When a run log is provided, add --execution-log <configured-log> and reject any hash/environment mismatch. Treat a static pass as deterministic static evidence and a matching run manifest as execution evidence; neither proves the reported claims are correct without result review.

For plan-review mode: Skip Gate 0 (no script to check). Proceed directly to design-level review.

1. Statistical Validity

#	Check	Verify by
1	Estimand and model align	Verify outcome family, observation level, contrast, and interpretation answer the declared estimand; design labels alone do not select a model.
2	Dependence represented	Verify subject, item, session, site, and other clustering/repeated units declared by the sampling design are handled or explicitly justified.
3	Contrasts specified correctly	Are contrast weights orthogonal? Are planned comparisons justified?
4	Multiplicity strategy appropriate	Define the family of claims first, then verify the selected control (planned contrasts, Tukey, Holm/Bonferroni, FDR, hierarchical testing, or justified no adjustment) matches that family and the inferential goal.
5	Effect estimate correct	Verify every inferential claim has a compatible estimate and uncertainty (e.g. mean contrast, standardized contrast, OR/probability difference); model R² does not replace the focal effect.

2. Reproducibility

#	Check	Verify by
1	Stochastic control recorded	Require seed/backend controls only for stochastic steps; confirm config and code agree
2	Session info output	grep "sessionInfo|session_info|version"
3	Runtime/dependency evidence	Exact language version is checked; the declared pin/lock artifact exists, covers packages actually imported, and agrees with the clean-run environment snapshot
4	Data path configurable	No hardcoded paths. Acceptable: relative paths from project root (data/subject.csv), here::here(), or path from config. Unacceptable: absolute paths (/Users/..., C:\...), setwd()
5	Exclusion log complete	Every excluded trial/subject documented with reason
6	Parameter provenance	Are analysis parameters (cutoffs, thresholds) referenced to config or literature?

3. Assumption Checking

#	Check	Verify by
1	Distribution/model diagnostics	Use diagnostics appropriate to the estimator and target (e.g. paired differences, residuals, dispersion, convergence, posterior predictive checks), not universal Shapiro tests per condition
2	Sphericity	Only for an ANOVA whose within-subject factor/contrast structure makes sphericity relevant; use a justified diagnostic/correction rather than a ceremonial test
3	Variance structure	Check homoscedasticity/variance modeling when required by the chosen Gaussian test/model; Levene is not a universal gate
4	Diagnostic response	Is the prespecified remedy appropriate to this estimator and estimand (e.g. covariance correction, alternative likelihood, robust uncertainty, sensitivity model)?

Check adaptation rules: Adjust assumption checks based on model type:

• lmer/glmer → skip Mauchly's sphericity (not applicable); check convergence warnings + singular fit + random effects variance
• paired t-test → inspect the distribution/robustness of paired differences (not raw condition scores)
• aov within-subjects → check Mauchly's sphericity + Greenhouse-Geisser correction
• glmer(binomial) → check overdispersion

4. Reporting Completeness

#	Check	Verify by
1	All conditions reported	Every condition from design has descriptive stats
2	Effect estimates for inferential claims	Each substantive claim has an estimate and uncertainty; diagnostic-test p-values do not require ceremonial effect sizes
3	Confidence intervals	Effect sizes reported with CI, not just point estimates
4	n reported per analysis	After cleaning, how many subjects/trials per condition?
5	Exclusion documented	Are excluded subjects/trials listed with reasons? Counts and percentages reported?

5. Figure Quality

#	Check	Verify by
1	Error bars defined	SE or CI stated in caption or code
2	Individual data shown	For within-subjects designs, individual data points visible
3	Axes labeled	Clear axis titles with units
4	Color-safe	Colorblind-friendly palette?

R Anti-Patterns

• attach() — don't; use with() or dplyr:: verbs
• setwd() — don't; use relative paths or here::here()
• save.image() — don't; save specific objects with saveRDS()
• options(stringsAsFactors = TRUE) — don't; modern R defaults to FALSE
• summary(model)$r.squared for mixed models — wrong R²; use performance::r2()
• Automatic Type III ANOVA without matching contrasts/hypotheses — choose sums of squares from the estimand/design, not imbalance alone

R Anti-Pattern Grep Patterns

When auditing R scripts, scan for these patterns:

#	Anti-Pattern	Grep	Why
1	attach(	grep -q "attach(" script.R	Namespace pollution
2	setwd(	grep -q "setwd(" script.R	Non-reproducible
3	save.image()	grep -q "save\.image" script.R	Non-reproducible
4	summary(lmer.*r.squared	grep -q "summary.*lmer.*r\.sq" script.R	Doesn't exist
5	aov() repeated measures	grep -q "aov(" script.R + inspect formula/Error structure	Requires design-aware review; not automatically wrong
6	Absolute paths	`grep -qE "/Users/	/home/

Python Anti-Patterns

• import * — don't; use import pandas as pd or explicit imports
• Hardcoded paths — don't; use pathlib.Path or config-driven paths
• Missing random_state/seed on an actually stochastic function — deterministic scipy.stats tests do not require one
• print(df) without .head() — floods output; use df.info() or print(df.head())
• No column existence check — use assert set(expected).issubset(df.columns)
• pd.set_option('mode.chained_assignment', None) — hides warnings; use .loc[] instead
• No claim-compatible effect estimate — each inferential claim needs an estimate on an interpretable scale plus uncertainty; pingouin is optional and is not suitable for every model
• Figure not saved — plt.savefig() required, not just plt.show()
• scipy.stats.f_oneway() for within-subjects designs — use pingouin.rm_anova() instead
• Plain statsmodels.Logit() for within-subjects binary data — use a design-appropriate, verified GLMM/GEE/Bayesian implementation; pymer4 and Bambi are examples, not universal defaults
• scipy.stats.ttest_ind() for within-subjects — use scipy.stats.ttest_rel() or pingouin.ttest()
• Aggregation that discards trial/item structure needed by the confirmed model — inspect intent rather than banning groupby().mean()
• smf.ols() for repeated measures — use smf.mixedlm() with groups='subject_id'
• pingouin.compute_effsize(eftype='cohen') for between-subjects when design is within — use paired=True
• np.corrcoef() for within-subjects repeated measures — use pingouin.rm_corr()

Python Anti-Pattern Grep Patterns

#	Anti-Pattern	Grep	Why
1	import *	grep -q "import \*" script.py	Namespace pollution
2	Absolute paths	`grep -qE "/Users/	/home/
3	No saved figure despite declared figures	inspect output code	Declared artifacts not persisted
4	chained_assignment	grep -q "chained_assignment" script.py	Hides warnings
5	f_oneway() within-subjects	grep -q "f_oneway" script.py + inspect design	Ignores dependence
6	Plain Logit() for repeated binary trials	grep -q "Logit(" script.py + inspect hierarchy	No random effects; use a validated GLMM/GEE implementation
7	ttest_ind() within-subjects	grep -q "ttest_ind" script.py + inspect design	Ignores pairing
8	smf.ols() repeated measures	grep -q "smf.ols" script.py + inspect design	Ignores dependence unless justified with another covariance estimator
9	compute_effsize.*cohen no paired	inspect call and design	Pairing must match the estimand
10	np.corrcoef() within-subjects	grep -q "np.corrcoef" script.py + inspect design	Ignores repeated measures

Scope Limitation Rule

At the start of every review output, state what was and was not reviewed:

Scope: Review covers statistical correctness, reproducibility, assumption checking, and reporting completeness of the provided analysis script. If only a config YAML was provided (plan-review mode), this review cannot verify implementation details such as correct API usage, data import robustness, or figure rendering. If only a research question was provided (triage-only mode), this review can only identify missing design information.

Note: The output format adapts to the review mode. In analysis-audit mode, include the full output below. In plan-review mode, skip Anti-Pattern Scan Results (no script to scan). In triage-only mode, output only the missing-information list.

Output Format

## Review Mode
{result-audit | analysis-audit | plan-review | triage-only | blocked}

## Scope
{What was and was not reviewed}

## Readiness Label
{ready_for_publication | ready_for_execution | not_ready | analysis_plan_ready | blocked}

## Critical Issues
- {issue} — {how to fix}

## Major Issues
- {issue} — {how to fix}

## Minor Issues
- {issue} — {how to fix}

## Anti-Pattern Scan Results
- Context-relevant patterns checked: {list}
- Findings requiring semantic inspection: {list}

## Assumption Check Results
- Distribution/model diagnostics: {results and applicability}
- Dependence/convergence/dispersion: {results and applicability}
- Prespecified fallback/sensitivity status: {results}

## Reproducibility Evidence
Exact runtime + dependency artifact: {verified/mismatch/missing}
Stochastic controls: {verified/not applicable/missing}
Session/environment info: {present/absent}
Exclusion log: {present/absent}
Relative paths: {yes/no}

## Execution and Result Evidence (result-audit only)
Command/exit/timestamps: {verified/mismatch/missing}
Config + code + dependency + input hashes: {verified/mismatch/missing}
Output inventory + hashes: {verified/mismatch/missing}
Warnings and diagnostics reviewed: {yes/no + summary}
Claim-to-table/figure trace: {verified/mismatch/missing}

## Overall Verdict
{1-2 sentence summary}

Verdict Rules

• plan-review: complete design contract → analysis_plan_ready; otherwise not_ready/missing-information report.
• analysis-audit: zero unresolved Critical/Major → ready_for_execution; static inspection alone cannot yield publication readiness.
• result-audit: require successful clean execution; matching hashes for the reviewed config/code/dependency/input; reviewed output inventory, tables/figures, warnings/diagnostics and environment evidence; and zero unresolved Critical/Major. Then use ready_for_publication; report any non-claim-affecting Minor findings separately.
• Any failed execution, result/code mismatch, or unresolved Critical/Major → not_ready.

Recovery Loop

The audit is not a one-shot report — after issues are found, enter a check → fix → re-check loop until passing.

Loop Flow

psy-ana-coder generates code
       │
       ▼
psy-ana-reviewer audits
       │
       ├── Critical/Major issues → fix (see fix paths below) → re-audit
       │                              ↑                    │
       │                              └────────────────────┘
       │                                   loop until 0 Critical + 0 Major
       │
       └── 0 Critical + 0 Major → static pass (`ready_for_execution`); only result-audit evidence can advance further

Fix Paths

Issue Type	Who Fixes	How to Fix
Incorrect statistical method choice	psy-ana-designer	Revise questions[].selected_method/estimand rationale → psy-ana-coder regenerates
Code implementation error (API misuse)	psy-ana-coder	Based on specific line numbers and correct API pointed out by reviewer, directly fix the code
Inappropriate scientific parameter/threshold	psy-ana-designer	Confirm the revised config decision → regenerate
Missing effect size / figure / environment info	psy-ana-coder	Add missing code block → reviewer re-audits
Hardcoded data path	psy-ana-coder	Replace with config-driven path → reviewer re-audits

"Round N audit: {X} issues found. After fixes, proceed to round N+1."

A re-audit is mandatory after each fix. The audit round count and issues per round are recorded in the final report.