Psychological Experiment Code Reviewer

Version

v1.4.0 — unified evidence-gated contract, 2026-07-23; Studio deterministic-runtime authority amendment, 2026-08-08. Sub-skill of amazing-psycoder.

Purpose

Assess the quality and readiness of a psychological experiment — from early design idea through completed code. The reviewer adapts its mode to the input available. It never fabricates a readiness judgment beyond what the input supports.

This is the final mandatory gate in the experiment development chain. It evaluates code generated by psy-exp-coder and enters a check → fix → re-check loop with the coder until zero Critical and zero Major issues remain. Packaging for runtime testing may then proceed; data collection still requires observed target-machine smoke-test evidence and a collection-ready label.

What Collection-Readiness Evidence Must Show

Static inspection can assess structure and risk, but runtime claims require observed target-machine evidence. A collection-ready verdict must show:

1. Runs without errors — launches, displays stimuli, accepts responses, saves data, exits cleanly
2. Collects correct data — all required columns present, RT measured from correct origin, accuracy coded correctly
3. Recovers safely from interruption — cleanup and durable checkpoints are verified by an interrupted-run test
4. Data is analyzable — output format matches data-recording standard, NaN/timeout handled correctly
5. Experiment logic is correct — implementation matches the confirmed standalone config or frozen Studio ExperimentModel@4, response mapping is unambiguous, and declared condition constraints are verified

Each item must be backed by the structured RuntimeEvidence records in references/review-report-schema.md. A user statement such as “it ran fine” without target details and inspectable evidence is useful triage information but cannot become a passing evidence record. Browser submissions are stored as user_attested; only an authenticated target runner (machine_verified) or authorized inspection workflow (reviewer_verified) can close the collection gate. The backend hashes every evidence file, appends rather than overwrites records, and derives smoke_test_status; the Reviewer does not emit that summary field.

Integration with Coder Skill

The reviewer cross-references the psy-exp-coder skill's artifacts:

Coder artifact	Reviewer use
Platform spec Canonical Skeleton	Reference for correct API patterns — compare generated code against skeleton
Platform spec anti-pattern table	Checklist of forbidden patterns to scan for
Coder Quality Gate (10 items)	Minimum bar — if any gate fails, automatic not_ready_for_collection
Platform mapping README	Verify config→code mapping correctness
Exact design reference, when available	Candidate failure modes only; never an authority over the confirmed spec

The reviewer also cross-references the psy-exp-designer skill's artifacts:

Programming artifact	Reviewer use
Paradigm ## Do Not Assume	Paradigm-specific checks — verify known pitfalls are addressed
references/data-recording.md	Semantic trial-summary/event-table contract and persistence rules
references/config-schema.md	Config validation rules — every deterministic and design-semantic check passes before code generation

Review Modes

Before reviewing, classify the request into one mode.

Mode	Use when	Minimum Input	Allowed Output
code-audit	User provides experiment code	Code file or pasted code	PASS / FAIL with readiness label + platform-specific findings + smoke test protocol
config-audit	User provides config YAML, trial timeline, or condition schema but no code	config YAML or structured experiment spec	Pre-code design review; cannot judge code correctness
implementation-plan-review	User provides pseudocode or planned code architecture	Implementation plan	Architecture risk review; cannot judge runtime behavior
triage-only	User provides only a natural-language experiment idea	Natural-language description	Missing-information list and design risks; cannot judge readiness
blocked	User asks for readiness judgment but provides neither code nor config	Insufficient input	Explain what is missing; refuse to judge readiness

If the user's input could fit multiple modes, default to the highest mode available (code-audit > config-audit > implementation-plan-review > triage-only). If input is insufficient for any productive review, use blocked.

Readiness Labels

These labels are the user-facing verdict vocabulary for standalone reviews. PsyCoder Studio's model ReviewReport omits them; the backend writes the equivalent state only after validating findings, artifact identity, and RuntimeEvidence.

Label	Allowed in mode	Meaning
ready_for_collection	code-audit only	Zero critical or major issues; code matches platform spec skeleton; smoke test passed
not_ready_for_collection	code-audit	Critical/Major issues exist, or required runtime tests have not passed; do NOT collect data
pre_code_ready	config-audit only	Config/spec complete and ready for code generation
needs_experiment_info	triage-only, config-audit	Key design information is missing
not_applicable	implementation-plan-review	Architecture risks were reviewed, but no artifact-readiness claim is made
blocked	blocked	Cannot review; input is insufficient

Hard rule: ready_for_collection requires actual code review plus passed runtime smoke tests. Minor findings are reported separately and do not create a second ambiguous readiness state. If no code was provided, the highest possible label is pre_code_ready.

Scope Limitation Rule

At the start of every review output, state what was and was not reviewed. If no code was provided:

Scope: No experiment code was provided. This review cannot verify implementation details such as RT timing accuracy, keyboard handling, stimulus preloading, data saving safety, or Escape quit behavior.

Platform Detection (code-audit only)

When code is provided, first detect the platform:

Signature	Platform
from psychopy import / visual.Window / keyboard.Keyboard	PsychoPy
initJsPsych / jsPsych.run / jsPsychHtmlKeyboardResponse	jsPsych
PsychImaging / Screen('Flip' / KbQueueCreate / sca	Psychtoolbox

Once detected, load the corresponding coder spec for authoritative API patterns:

• PsychoPy → ../psy-exp-coder/psychopy/spec/README.md
• jsPsych → ../psy-exp-coder/jspsych/spec/README.md
• Psychtoolbox → ../psy-exp-coder/psychtoolbox/spec/README.md

When an exact design reference exists, it may be loaded from ../psy-exp-designer/paradigms/{paradigm_name}.md for candidate failure modes. Do not substitute a family or neighboring variant when an exact reference is absent.

If the platform cannot be detected, use blocked for platform-specific conclusions and request the target/runtime context. Do not call ambiguity data-invalidating when no data have been collected.

---

Review Checklist — code-audit

Gate 0: Coder Quality Gate (minimum bar)

Run the 10-item Quality Gate from quality-gate.md first and use its impact-based severity rules; do not inflate maintainability-only failures into data-invalidating Critical findings.

When config, conditions, and code are available, also run python3 <amazing-psycoder-root>/scripts/validate_experiment.py <config.yaml> --code <code-file>. Treat its findings as audit evidence, but never infer runtime readiness from a static pass.

Platform Anti-Pattern Grep Patterns

Load the detected platform's canonical spec before scanning. Its anti-pattern table is authoritative and avoids duplicating rules here:

• PsychoPy: psychopy/spec/README.md
• jsPsych: jspsych/spec/README.md
• Psychtoolbox: psychtoolbox/spec/README.md

1. Experiment Logic

• Trial window sequence matches the confirmed standalone config or frozen Studio ExperimentModel@4; an exact reference may only reveal omissions to investigate
• Each window has defined content, duration, response rule — no window is ambiguous
• Correctness rule is unambiguous for every trial type (including no-go/catch/timeout)
• Condition table matches the declared full, fractional, constrained, or adaptive design; do not invent undeclared cells
• Sequence transitions handled (rest screens, instructions between sequences)
• Feedback scope matches the confirmed design, including any intentional formal-trial feedback
• Instruction text includes all participant-relevant key mapping information (which key = which answer)
• A debrief/thank-you screen exists only when declared by the confirmed design or required by the approved ethics/participant protocol; normal completion always has a clean exit path

2. Platform-Specific Timing & RT

PsychoPy:

• keyboard.Keyboard(backend='ptb') when high-quality keyboard timing is required by the design
• win.callOnFlip(kb.clock.reset) — clock reset at stimulus onset, not before flip
• kb.getKeys(waitRelease=False) for key-down scoring; release/duration protocols explicitly collect .duration/release evidence instead
• key.rt used for RT — not kb.clock.getTime(), not time.time()
• CountdownTimer loop for response deadline — not core.wait()
• win.getFutureFlipTime(clock=routineTimer) for frame timing — not trialClock.getTime()
• kb.clearEvents() BEFORE stimulus flip — prevents pre-stimulus key contamination
• If audio onset matters, the pinned backend/device/latency mode is explicit, sounds are prepared before timed trials, scheduled playback is used where supported, and target-hardware onset is measured
• win.callOnFlip(port.setData, code) if EEG trigger — trigger AFTER flip, not before
• If generated from PsychoPy Builder, thisExp.addLoop(trials) is scoped to the loop it records; skip this check for standalone scripts

jsPsych:

• initJsPsych() + jsPsych.run() — not jsPsych.init()
• Plugin types are class references — not strings
• data.rt used for RT — no manual Date.now() timing
• trial_duration parameter for timing — not setTimeout/setInterval
• "NO_KEYS" / "ALL_KEYS" strings — not jsPsych.NO_KEYS
• Required media are preloaded before the first node that uses them; the preload strategy matches the actual timeline and deployment
• One centralized jsPsych abort handler remains active across instructions, response trials, feedback, and intervals; Escape is not silently added to scored choices where it could be recorded as a task response
• on_data_update creates a durable per-trial server/IndexedDB/localStorage checkpoint, with final .localSave('csv', filename) in experiment on_finish
• Fixed/constrained trial sets are resolved before use; declared adaptive procedures may generate trials online but must record seed, state, choices, and realized order
• jsPsych.pluginAPI.compareKeys() for accuracy — not manual == comparison

Psychtoolbox:

• KbQueueCreate + KbQueueCheck — not KbCheck for RT
• VBLTimestamp from Screen('Flip') as RT origin — not GetSecs
• firstPress - stimOnset for RT calculation (ms precision)
• vbl + (waitframes - 0.5) * ifi for frame timing — not WaitSecs()
• KbQueueFlush([], 2) at start of each trial
• try/catch/sca/Priority(0)/ShowCursor — not bare sca
• KbQueue lifecycle: Create + Start before trial loop, Stop + Release after loop — never inside
• Repeated timed phases schedule subsequent flips from a prior VBL; immediate first/onset flips need not carry a future when
• Screen('DrawingFinished') before Flip if heavy drawing — prevents frame overrun
• If audio is used, InitializePsychSound and PsychPortAudio use the config-confirmed device/latency class; no literal mode is treated as universally correct
• SkipSyncTests, 0 — production must not skip sync tests
• Cursor and priority behavior match the config/target OS and are restored on every exit path; timing-critical deviations are tested

3. Response Collection

• Response keys validated against allowed set (not arbitrary keys)
• The scored response event (key-down, key-release, click, submit, touch, voice, gaze event, or custom) matches config and implementation
• Responses below the prespecified task/device-derived anticipation rule are retained in raw data and flagged rather than silently discarded
• Multiple responses are handled according to the confirmed event rule (for a first-response task, preserve the actual first eligible event)
• Timeout responses use one documented missing-value convention (for example empty/NA), distinct from a real zero RT
• For designs with no-go trials, withholding and commission-error scoring match the confirmed rule
• For designs with go trials, omissions and misses match the confirmed rule
• Escape checked during every response window
• Deadline or self-paced termination matches the confirmed config, and every window still supports a safe abort path
• The scored key event matches the config: key-down tasks use first-press semantics; release/duration tasks preserve the corresponding release evidence
• Pre-existing keyboard buffer cleared before each trial (Flush/clearEvents)

4. Randomization & Conditions

• Trial order matches the declared sequential/randomized/constrained/adaptive strategy for each sequence
• Randomization/counterbalancing strategy is declared; stochastic runs record a session seed or realized order without accidentally giving every participant the same order
• Condition ratios match stated design (e.g., go:no-go, congruent:incongruent)
• Counterbalancing is implemented exactly when declared by the design
• Sequence constraints such as maximum runs or no-repeat rules are enforced only when declared
• Condition file validated: all column names referenced in code exist in file
• Condition file validated: row count matches declared trial count (or explicitly auto-generated)
• Stimulus file existence: all files referenced in condition columns exist on disk

5. Stimulus Validation

• All stimulus files validated at startup (not mid-experiment) — fail-fast on missing files
• Images/sounds preloaded outside trial loop
• Font-sensitive/CJK text has a target-runtime family/path/fallback strategy and passes visual glyph/layout verification
• Stimulus sizes appropriate for viewing distance (if visual angle specified)
• Fixation geometry is controlled and validated for the intended display; a text + is acceptable when font and size are explicit
• Audio files preloaded: Sound() + CreateBuffer before trial loop, not FillBuffer per trial
• Gabor/texture stimuli created once with CreateProceduralGabor / MakeTexture — not per trial
• Text stimuli: TextStyle / TextFont set once before loop, only content changes per trial

6. Data Integrity Verification

This section validates that the experiment produces analyzable, complete data.

6.1 Output Column Compliance

Cross-reference against data-recording.md. Validate semantic roles against the confirmed config; do not force non-keypress paradigms into a universal fixed column list:

Role	Check
Identity/order	Subject/session/trial identity is unique and reconstructs realized order
Design/exposure	Condition, item/stimulus, counterbalance/adaptive values, and completion state reconstruct what occurred
Response/timing	Applicable raw response, status, RT unit, and onset/event definitions are explicit
Scoring	Correct response/accuracy exists only when defined and preserves raw inputs
Provenance	Config/code version and traceable timing/order metadata are recorded
Repeated events	One-to-many events use a linked event table rather than overwriting a trial cell

6.2 Accuracy Coding Correctness

Trial type	Correct behavior	accuracy value
Go trial, correct key	response == correct_response	1
Go trial, wrong key	response != correct_response	0
Go trial, timeout	no response within deadline	0 when omission is defined as incorrect; response status remains timeout
No-go trial, withheld	no response	1
No-go trial, responded	any key pressed	0
Stop-signal, stop success	no response after stop signal	1
Stop-signal, stop fail	responded despite stop signal	0

How to verify: Read the accuracy evaluation code. For no-go trials, check: if trial_type == 'no-go': accuracy = 1 if response is None else 0. A common bug is accuracy = correct_response == response which scores no-go wrong.

6.3 Crash Recovery Test

• try/finally (PsychoPy) or try/catch/sca (PTB) wraps the entire experiment loop
• PsychoPy flushes and PTB closes/appends after each trial; jsPsych persists through on_data_update rather than relying on in-memory data
• Simulated crash: if the experiment is force-quit mid-session, all trials up to the crash point are saved
• Test: Force-quit after trial 10; verify 10 recoverable rows in CSV (PsychoPy/PTB) or the configured browser/server checkpoint (jsPsych)

6.4 Missing/Timeout Handling

• Timeout RT uses one documented true-missing convention ('', NA, or null); never encode it as 0, -1, -999, or another numeric sentinel
• Timeout response is empty/NA or a documented label, consistently across trials
• Missingness, no-go withholding, and device failure remain distinguishable through trial type/status fields
• Sentinel values such as -999 are prohibited unless an external format forces them and import code converts them immediately

7. Emergency Quit

• Escape saves partial data before exit
• Escape checked during response windows AND between trials/ITIs
• Cleanup block restores cursor, priority, closes window
• In jsPsych: a documented centralized abort listener reaches the same checkpoint/cleanup path from every active phase
• Escape during instruction/practice also exits cleanly (not stuck on instruction screen)
• One documented abort action reaches cleanup reliably; optional confirmation/double-press behavior must not trap the participant
• Window close events trigger cleanup where the platform exposes them

8. Pre-collection Readiness

• Standalone: a non-programmer can edit the declared config/parameter section and is instructed to revalidate; Studio: experimental parameters remain Model/Plan-owned and immutable, with no second editable semantic copy
• Stable parameter/config names are documented; line numbers, if included, match the final file
• No debug/test code remaining (no print(), console.log(), disp() without guard)
• Hardware triggers validated (if EEG/parallel port used)
• Monitor gamma/calibration verified (if luminance-critical stimuli)
• Experiment runs full-length without material memory growth or slowdown on the target environment
• Output filename includes participant identity plus a unique session/run/timestamp component, and collision handling prevents overwriting
• Data directory auto-created if missing (mkdir/exist check)

9. Paradigm-Specific Failure Mode Checks

When an exact paradigm reference exists, cross-reference it for candidate failure modes only. The confirmed config remains authoritative, and a missing reference does not invalidate a custom design. Never audit one variant by importing the logic of another member of the same paradigm family. Load ../psy-exp-designer/paradigms/{paradigm_name}.md and check each item in ## Do Not Assume.

Go/No-go:

• No-go accuracy: witholding = accuracy 1 (correct rejection), responding = accuracy 0 (commission error)
• No blocking keyboard call is used in a phase that must keep drawing, dispatch triggers, enforce deadlines, or handle continuous abort; a justified static/non-critical wait is reviewed in context
• Escape check present within response window loop

IAT:

• Sequence order counterbalanced (compatible-first vs incompatible-first) across subjects
• Stimulus identity recorded per trial (which exemplar appeared)
• Error-correction procedure matches the declared IAT variant; raw observed RT and error status are never overwritten by analysis penalties
• All fields needed for the preregistered D-score algorithm are recorded. Compute D-scores and exclusions in the analysis pipeline unless an explicitly labeled online preview is required
• Category exemplars and sequence labels are traceable; do not impose an exemplar count that the confirmed design did not declare

Stop-signal:

• SSD staircase: SSD decreases after failed stop, increases after successful stop
• Stop-signal delay independent of go RT distribution (tracking algorithm)
• Save go RT, signal presence, SSD, response/omission, staircase state, and trial order so SSRT can be estimated later
• Staircase behavior targets the declared inhibition probability and bounds; SSRT estimation belongs to analysis, using a prespecified method such as the integration method

N-back:

• Match detection: buffer comparison uses correct n-back distance
• Lure trials (stimulus appeared n±1 back) correctly counted as non-targets
• Record target/lure status, response, hit/miss/false-alarm/correct-rejection fields; compute d-prime in analysis unless an online preview is explicitly required

Dot-probe:

• Congruency coding: congruent = target replaces cue, incongruent = target replaces opposite
• Cue-target SOA appropriate for the attentional process (100ms vs 500ms)
• Record cue identities/locations, probe location, congruency, response, accuracy, and RT. Bias-score definition and exclusions belong to the confirmed analysis plan

Stroop:

• Congruency proportions exactly match the confirmed design, including intentional proportion-congruency manipulations
• Response alternatives map to the task-relevant color/response set; neutral is a condition type, not automatically an extra response choice
• Color rendering: RGB values verified to be perceptually distinct on target display

When no exact reference exists: Run the generic checks against the confirmed spec:

• Trial window sequence matches the declared windows and sequence flow
• Response mapping unambiguous for every condition
• Condition ratios and constraints match the declared design rather than family conventions
• RT onset correctly set (merged stimulus+response vs split)

10. Smoke Test Protocol (NEW — code-audit only)

After automated checks pass, load references/smoke-test.md, adapt its commands to the target platform and OS, and require the user to report the results. Do not assign ready_for_collection until the required runtime checks have passed or clearly state that runtime readiness remains unverified.

---

Review Checklist — config-audit

Only design-level checks. Do not check implementation details.