/rl:run — launch one Lego-RL run

Preflight, review the parameters, confirm, then launch in the background. Refuses if any check blocks, or if a run is already live on this box.

Step 0 — Orient

Same as /rl:check Step 0: find the repo root, resolve the config (path / bare name / ask when absent or ambiguous), infer the kind (train | eval | infer). Never guess which config the user meant — a wrong guess here burns a cluster for hours.

Step 1 — Refuse if a run is already in flight

bash scripts/lib/live_probe.sh <kind> 2>&1 | grep -E '^WARN +job:'

If any job:trainer or job:runner line is alive, abort:

A run is already in flight; refusing to start another:
  pids:  <pid + etime, from ps -p <pid> -o pid=,comm=,etime=>
  log:   <most recent *.log under logs/>
Let it finish, or stop it yourself (kill -INT <pid>), then re-run /rl:run.

A bare job:vllm with no runner above it is a foreign serving job, not ours — that is Step 2's business (it blocks on GPU ownership), not an abort message about "our" run.

Step 2 — Preflight via /rl:check

Run the check skill's logic on this config (invoke that skill, or inline its Steps 1–3). If the verdict is SAFE TO RUN: ❌ NO, abort with the same consolidated report, prefixed:

Preflight failed — fix the items below before /rl:run.

Do not invent values, skip a check, or pass a force flag; there is no force flag. If the user says "just launch it anyway", explain exactly which check blocks and what it costs to ignore it (each ✗ FATAL maps to a real past incident, written up in the Troubleshooting section of the docs site), then ask them to fix the config. Configs are user-owned; this skill is a launcher, not an editor.

Step 3 — Show the parameters and confirm (mandatory, never skip)

PREFLIGHT_ONLY=1 already printed the run configuration (<kind>) block in Step 2. Show that block — verbatim — and then draw attention to the handful of fields that decide whether the run is worth its hours. Keep this short; the block above already has the detail.

Worth double-checking before this run:
  • data     <train/val index, or dataset, or shard>   ← is this the one you meant?
  • scale    <N nodes × 8 gpu>, <trials/step>, epochs=<N>  ← roughly <estimate> hours/step
  • resume   <RESUME_FROM, or "auto: picks up the newest ckpt in the exp dir">
  • naming   project=<...>  exp=<...>
  • logs     <the real TRAIN_LOG path>
             dashboard <visible / not visible: it is serving <log-dir>>
  • first step  <R3 on → pearson ≈ 0.999; otherwise routing misalignment silently
                destroys the gradients>

Where the log lands, and whether the board will see it — this line is not cosmetic. scripts/templates/verl/common.env derives TRAIN_LOG=${HARBOR_LOG_DIR}/${TRAINER_EXPERIMENT_NAME}.log, and a real config overrides HARBOR_LOG_DIR to a per-experiment directory under the shared trials root. Only the template default is <repo>/logs. The dashboard globs its --log-dir one level, no recursion (webui/server.py) — so a config that overrides HARBOR_LOG_DIR produces a run that trains fine and is invisible on the board for its whole life unless something links it in. Resolve both sides before asking for the yes:

bash scripts/train/train.sh --dry-run <config> 2>&1 | grep -E 'train log|vLLM log'
pgrep -af 'server\.py.*--log-dir'      # which dirs a board actually serves

If dirname(TRAIN_LOG) is not one of the served dirs — and "no board running at all" counts as not served — say so and ask. Do not launch silently, and do not pick for them:

This config writes its training log to
  <TRAIN_LOG>
but the dashboard (pid=<P>, port=<P>) is serving
  <served log-dir>
One-level glob, no match → this run will never appear on the board
(training itself is unaffected).

  [1] symlink it into <served-dir>/<name>.log right after launch
      (no dashboard restart — recommended)
  [2] leave it, watch wandb only
  [3] stop and change HARBOR_LOG_DIR in the config first
Which one?

Option 1 is one ln -s in Step 6, after the real exp_name is known. It is the only choice needing neither a restart nor a config edit, and the symlink name becomes the run's id in the UI — the analysis panels get their exp-dir mapping from the log contents, so any readable name works. Never move the log (the runner holds it open through tee), and never leave a dangling symlink under a log dir — that breaks /api/runs for every run.

Then ask, in one line:

Launch with this configuration? [Y]/[N]  (N = edit the config first)

Never launch without an explicit yes. If the user says no, stop cleanly and tell them to edit the config, then re-run /rl:run.

exp_name gotcha — unless the config pins EXP_NAME, train.sh derives it as harbor-<scaffold>-<EXP_TAG>-<timestamp> at launch time, so the name in the preflight output is not the name the run will have. Never report the preflight-time name as final; read the real one back in Step 6. If the user needs a stable name (resuming, comparing runs), suggest pinning EXP_NAME= in the config before launching.

Step 4 — Decide launch mode

Default = background. These runs last hours; foreground holds the session.

Launch mode? [B]ackground (default: nohup setsid, logs to logs/launch_<ts>.log)
             [F]oreground (holds this session until the run ends)

Accept B / F / <enter> (= background).

Step 5 — Multi-node: print the per-node command, do not SSH

Read topology (train) or vllm topo (infer) out of the parameter block.

• train, NNODES > 1 — ray_bringup.sh branches on local IP: the node whose IP equals MASTER_ADDR (default: the launching node's first IP) becomes the ray head and drives training; every other node joins and exits. So the same command must be run on each of the NNODES nodes.
• infer, VLLM_NNODES > 1 — same shape, keyed on VLLM_HEAD_HOST; rank-0 serves the API, the others run --headless DP shards.
• eval — always single-node.

Print the command block and tell the user to run it on the other nodes themselves. Never SSH to another node, and never assume the workers are up just because the head started.

Multi-node: run the command below once on every node (this machine is already covered
by the launch just performed)
  nodes <ip2>, <ip3>, ...:
    cd <repo_root> && MASTER_ADDR=<head_ip> nohup setsid bash scripts/train/train.sh <config> > logs/launch_<ts>_$(hostname -s).log 2>&1 &
The head node waits for all <NNODES> nodes to join before training starts; a worker that
never came up shows as the head spinning in its ray status poll.

Step 6 — Launch

Timestamp: TS=$(date -u +%Y%m%d-%H%M%S). Log: logs/launch_${TS}.log (mkdir -p logs first).

Background (default)

cd <repo_root> && nohup setsid bash scripts/<kind>/<kind>.sh <config> > "logs/launch_${TS}.log" 2>&1 < /dev/null &
PARENT_PID=$!
disown

setsid matters: it detaches the run from this session so a disconnect does not take the training down with it.

Then, without polling in a tight loop:

1. Wait up to 60s (say, 5s sleeps) for the launch log to grow past 0 bytes.

2. Read its first ~120 lines and pull out:

   • the real exp= value from the run configuration block,
   • the preflight verdict line (it runs again inside the real launch),
   • any early [FATAL].

3. Once they appear, capture the PIDs: pgrep -f 'scripts/<kind>/<kind>.sh' and, for train, pgrep -f 'fully_async_main|main_ppo' (the trainer takes a few minutes to show up — report it as pending rather than waiting for it).

4. If Step 3 chose [1], link the log into the served dir now that exp_name is real. Verify with the API rather than assuming — the run only shows up once it has training steps, so an empty steps right after launch is expected:

   ln -sfn "<TRAIN_LOG>" "<served-dir>/<readable-name>.log"
   curl -s -m 300 "http://127.0.0.1:<P>/api/runs" | grep -c '<readable-name>'
   

   For a multi-node run, link only the ray-head node's log — that is the one carrying the trainer output; the worker exp dirs hold just *_train_gpu_wandb.log, and the analysis panels recover the sibling exp dirs on their own.

If the log stays empty for 60s, or the runner exits non-zero within 60s, print the tail of the launch log and stop. Do not retry automatically and do not "fix" the config to get past it.

Foreground (only if chosen)

cd <repo_root> && bash scripts/<kind>/<kind>.sh <config> 2>&1 | tee "logs/launch_${TS}.log"

Stream it; on exit report the exit code and the log tail.

Step 7 — Report

Tight summary, ≤ 15 lines:

Launched (background)
  kind / config:  <kind> · <config path relative to the repo>
  exp_name:       <the real name read back from the launch log; if not there yet,
                   "starting — check the log in 30s">
  launch log:     logs/launch_<TS>.log
  train log:      <absolute TRAIN_LOG path, read from the launch log's "train log:" line>
  vllm log:       <absolute VLLM_LOG path, same source>
  trials:         <HARBOR_TRIALS_DIR>
  dashboard:      <URL, run name = <readable-name> / not visible (user chose [2])>
  pids:           runner=<pid>  trainer=<pid or "starting">
  stage:          vLLM loading weights + cuda-graph capture; ~10–20 min before step 1

Watch:
  tail -F logs/launch_<TS>.log
  nvidia-smi
  /rl:status            # once it is running, check that the first step is healthy
Stop:
  kill -INT <runner_pid>    # let it wind down; do not kill -9

For train, close with the first-step gates — they are cheap to check and each one has cost a full run before:

Check these at the first step (~20–40 minutes in):
  • R3 on → pearson ≈ 0.999 in the log. Clearly below that means the routing replay is
    misaligned and the gradients are already garbage — stop and investigate.
  • lr is not 0 (fully-async + cosine collapses to 0; this runner forces constant, but
    it is worth confirming).
  • First-step reward / num_turns are non-zero — all zeros usually means the environment
    images cannot be pulled, not a model problem.

What this skill must NOT do

• never force past a blocking preflight, and never add a flag to bypass one
• never kill a live run to make room — ask the user to stop it themselves
• never edit the config, lib/site.env, or src/ to make a check pass
• never SSH to another node, or launch anything on a node other than this one
• never run in the foreground without asking — it locks the session for hours
• never report the preflight-time exp_name as the run's real name (Step 3)
• never print logs/<exp_name>.log as the training log without checking — that is the old scripts' path; the config decides, and it is usually under harbor_trials/. Read the runner's own train log: line instead of composing one
• never launch without telling the user whether the dashboard will see this run
• never delete or move logs, harbor_trials/, or checkpoints; a dangling symlink under a run dir is enough to break the webui run list