/rl:check — preflight one Lego-RL run

Answers one question: is it safe to launch this config right now?

Deterministic checks come from the runner itself (never re-implemented here); this skill adds the local live layer and prints one report ending in a clear YES / NO. Read-only — it never edits a config and never launches.

/rl:check <config>
   │
   ├─ 1. PREFLIGHT_ONLY=1 <kind>.sh <config> ... resolved parameters + OK/WARN/FATAL
   │
   ├─ 2. lib/live_probe.sh <kind> .......... local facts; this skill judges ownership
   │
   └─ 3. report ............ verdict + status table + run parameters + next steps

Each check lives in exactly one layer:

Layer	Run by	Covers
Deterministic	scripts/lib/preflight.sh via the runner	tool_parser × model × scaffold · topology · SP/device-mesh · VRAM · veomni constraints · R3 × model family (MoE/dense) × engine × verl · agent_name × scaffold · image source · kubeconfig · lr_scheduler · val timeout · rollout_is · path existence
Live (judgment)	this skill, from scripts/lib/live_probe.sh facts	is a run already in flight? · is that GPU/port mine, stale, or a foreign job's? · venv imports · disk / shm headroom

Everything is local host only — no SSH to 221/222/240/243, no cluster mutation. Cluster-side faults (registry down, kyverno crash-looping, a node missing its insecure-registry config) are out of scope here; they surface as env_setup_failed once a run is live, which is /rl:status's job.

Step 0 — Orient

Repo root is the Lego-RL checkout (contains scripts/lib/preflight.sh). If the user ran this from elsewhere, locate it; if there is none, abort — but still print the Step 3 report with a NO verdict whose reason is the abort.

Resolve the config. The argument may be a full path, a bare filename, or absent:

• full/relative path that exists → use it;
• bare name (qwen35a3b_4n_mix1582, smoke_sync_cc_qwen35a3b_243.env) → search scripts/{train,eval,infer}/configs/ then .../templates/;
• absent, or more than one match → list the candidates (ls scripts/*/configs/*.env) and ask which one. Do not guess.

Infer the kind from the resolved path: scripts/train/… → train, scripts/eval/… → eval, scripts/infer/… → infer. If the path is ambiguous, read the config: TRAIN_MODE/N_NODES_ROLLOUT → train, DATASET_PATH/DATASET_NAME → eval, RESULTS_DIR/OUTPUT_INDEX → infer.

Read scripts/README.md for context if you need the axis vocabulary. Do not echo config or README content into the report.

Step 1 — Deterministic checks

PREFLIGHT_ONLY=1 bash scripts/train/train.sh <config> 2>&1; echo "EXIT=$?"

This one command sources the config and its templates, resolves every derived value, prints the run configuration block, then runs preflight.sh and exits without launching. Read its output as-is — never re-run a probe separately and never overrule an OK.

Only the train runner has PREFLIGHT_ONLY. For an eval or infer config, use bash scripts/<kind>/<kind>.sh --dry-run <config>: it resolves and prints everything the same way but does not run preflight.sh, so for those two kinds the deterministic layer is thinner — say so in the report rather than implying the config passed rules that never ran.

Marker	Status	Blocks launch?
✓ OK	pass	—
⚠ WARN	warning	no
✗ FATAL	fail	yes

Two things need follow-up:

• the run configuration (<kind>) block → copy verbatim into the report; on an SAO / critic run it carries extra gae:, bypass: and critic lines — quote them too, they are what rule 10 (SAO / critic) of preflight.sh judged;
• a non-zero EXIT with no ✗ FATAL line → the runner died before preflight (bad config syntax, missing PROJECT_NAME/EXP_TAG, unreadable venv). Treat as a blocking failure and quote the error.

preflight.sh failing is never something to work around. Every ✗ FATAL has an entry in the Troubleshooting section of the docs site; point the user at it.

Step 2 — Live checks

bash scripts/lib/live_probe.sh <kind> 2>&1

The probe prints facts only (OK / WARN / INFO); deciding whether a process is ours is this skill's job. Four outcomes block a launch — job:running, gpu:foreign, port:conflict, import:missing. Everything else is advisory.

2a — Is a run already in flight?

A second run on the same GPUs OOMs or corrupts both.

Probe lines	Name	Status
any WARN job:trainer or WARN job:runner	job:running	✗ blocks
only WARN job:vllm / job:harbor (no trainer/runner)	job:orphan	⚠ — a leftover server or someone else's serving job; classify in 2b/2c
OK job:none	job:none	✓

On job:running, record the PID set — call it the run tree; it anchors the ownership tests below. Report pid + etime, and tell the user to let it finish or stop it themselves (kill -INT <pid>). Never kill anything.

2b — Are the GPUs free, mine, or a foreign job's?

Input: each WARN gpu:busy pid=<pid> comm=<comm> mem=<mem> line.

Owner	Name	Status
in the run tree from 2a	gpu:mine	⚠
any other pid	gpu:foreign	✗ blocks
OK gpu:idle	gpu:idle	✓

Ownership test: walk the parent chain (ps -o ppid= -p <pid>, repeated) and see whether it reaches a PID in the run tree. A VLLM::Worker_TP* / EngineCore holding ~130GB with no runner above it is someone else's serving job — block, report pid + memory + etime, and let the user decide whether to wait or ask its owner. This box is shared; a foreign vLLM squatting on all 8 GPUs is the common case, not an anomaly.

If nvidia-smi is absent (INFO gpu:absent), emit gpu:unknown (⚠) and say the GPU layer could not be judged — do not silently pass it.

2c — Who owns a busy port?

Input: each WARN port:<P> line. Only ports this config will bind can block; the rest are informational.

kind	ports that matter
train	6379 (ray), 8265 (ray dashboard)
eval	the port= in the summary's serving line (default 8000)
infer	VLLM_PORT / VLLM_MASTER_PORT / VLLM_DP_RPC_PORT from the summary

Owner	Name	Status
in the run tree	port:mine	⚠
anything else, on a port this run needs	port:conflict	✗ blocks
busy but irrelevant to this kind (e.g. 8090 webui)	port:other	✓ note only

For port:conflict, name the pid and suggest either stopping it or moving this run's port in the config — never suggest killing a process you cannot attribute.

2d — Environment sanity

Probe line	Name	Status
WARN import:veomni on a train run	import:missing	✗ blocks — the runner exits on this
WARN import:<mod> otherwise	import:degraded	⚠
WARN venv	venv:fallback	⚠ — runner will use a non-.venv python
WARN shm	shm:dirty	⚠ — train.sh clears it at bring-up; only worrying if a live run owns it
WARN disk:<mnt>	disk:low	⚠ — quote the mount; a full root disk evicts pods and truncates logs

Step 3 — The report (always the last thing you print)

The report is the deliverable. Print it every time, including on an abort (then: heading + NO verdict with the abort reason, nothing else). Prose in Chinese — the field labels and probe names stay as written here.

## harbor check — <kind> · <config basename>

**SAFE TO RUN: <✅ YES | ❌ NO>** — <R> blocking · <W> warnings

| Layer | Check | Status | Detail |
|-------|-------|:------:|--------|
| det  | preflight (tool_parser · topology · device-mesh · VRAM · veomni · R3 · agent · image · paths) | <✓/✗> | ok=<N> warn=<N> fatal=<N> |
| det  | <one row per ✗ FATAL or ⚠ WARN> | <✗/⚠> | <verbatim text> |
| live | job   | <✓/⚠/✗> | <job:none / job:running pid=<P> etime=<T> / job:orphan> |
| live | gpu   | <✓/⚠/✗> | <gpu:idle / gpu:mine / gpu:foreign pid=<P> mem=<M>> |
| live | ports | <✓/⚠/✗> | <port:free / port:mine / port:conflict:<P>> |
| live | env   | <✓/⚠/✗> | <imports ok / import:missing:<mod> / venv:fallback> |
| live | disk  | <✓/⚠>   | <shm=<pct> root=<pct> / disk:low:<mnt>> |

**Key parameters for this run**
```
<paste the whole "run configuration (<kind>)" block from PREFLIGHT_ONLY, verbatim>
```

**Log destination / dashboard visibility**
```
train log   <TRAIN_LOG, taken from the runner's "train log:" line — never assembled by hand>
dashboard   <pid=<P> port=<P> serving <log-dir> → visible / not visible / no instance running>
```

Not a pass/fail check, but it belongs in the report because it is invisible
otherwise: `scripts/templates/verl/common.env` puts `TRAIN_LOG` under
`${HARBOR_LOG_DIR}`, and a real config overrides that to a per-experiment
directory under the shared trials root — **not** `<repo>/logs`. The webui globs
its `--log-dir` one level with no recursion (`webui/server.py`), so such a run
trains normally and never appears on the board. Read the served dirs from
`pgrep -af 'server\.py.*--log-dir'` and say which way it falls. `/rl:run` turns
this line into a question before launching; `/rl:check` only has to report it.

**Next steps**
1. <one line per blocker, blockers before warnings; quote kubectl/curl/git errors verbatim>
2. ...
Re-run `/rl:check <config>` once they are fixed.

The four invariants:

1. Verdict — ✅ YES iff R == 0, where R = ✗ FATAL count + any job:running + any gpu:foreign + any port:conflict + any import:missing. Warnings never change the verdict.
2. Glyphs — ✓ pass · ✗ blocks · ⚠ advisory · · skipped.
3. Collapse — fold all passing preflight checks into the first row; add a row only for each check that is ✗ or ⚠.
4. Verbatim — the parameter block and every quoted error are copied character-for-character. Never paraphrase a config value into the report; if it is not in the runner's output, do not state it.

Add one italic line under the table when it applies:

• train with NNODES > 1 → (multi-node: this check covers the head node only — run /rl:check once on every worker node too)
• infer with VLLM_NNODES > 1 → same, keyed on node_rank.

Guardrails — never do these

• edit the config, lib/site.env, or anything under src/ to make a check pass
• launch anything, including a "quick" DRY_RUN that starts vLLM or ray
• re-implement or overrule a preflight.sh check — add only the live layer
• kill a process, ray stop, clear /dev/shm, or delete a log — surface it, let the user decide
• SSH to another node, or mutate the k8s cluster in any way
• report a value you did not read out of the runner's own output