Run GR00T Fine-Tuning

Own the requested workflow through validated artifacts. Execute stages when the user asks for a run; do not stop after printing commands. Keep source-data success, pipeline validity, and fine-tuned task success as three separate results.

Resolve an embodiment contract

Resolve the contract from task registration, environment configuration, HDF5 data, converter, modality config, and the registered closed-loop adapter. Never infer it from a robot name alone.

Record these fields before launching expensive work:

contract ID
record task and inference task
motion/scene source
record observation terms
state layout, units, transforms, and dimension
absolute action layout, order, semantics, and dimension
camera term -> video modality mapping
modality config path and closed-loop adapter
source-success definition
task-success evaluator and thresholds
episode horizon and FPS
reset, warmup, curriculum, and randomization behavior
safe camera-free and rendered parallelism

Read the matching reference:

• Floating-hand Sharpa: references/floating-hand.md
• Vega/Dexmate Sharpa: references/vega-sharpa.md
• Contract schema and invariants: references/embodiment-contract.md
• New embodiment work: references/adding-embodiment.md

If no contract matches, stop before collection and follow the new-embodiment procedure. Never silently fall back to a floating-hand or Vega layout.

Keep runtimes explicit

Stage	Runtime
Checkpoint resolution	host or IsaacLab container
Source rollout, record, and rerender	robotic_grounding IsaacLab container
HDF5 to LeRobot conversion	robotic_grounding container
Statistics, training, open-loop evaluation	external Isaac-GR00T Python 3.10 environment
GR00T server	external Isaac-GR00T environment
Closed-loop client	IsaacLab container over host-network ZMQ

Before using a container, inspect its mounts and prove it contains the current worktree. Do not reuse a similarly named container mounted from another checkout. Keep long-running server and simulator processes monitored and clean them up on success or failure.

Follow the gated workflow

1. Preflight and plan

Record the branch, commit, worktree status, checkpoint source and SHA-256, motion source, seeds, GPU, disk, container/image, external GR00T checkout, and resolved contract. Use one unique output root and preserve commands and metrics in a run report.

Install or verify pyzmq and msgpack-numpy in the IsaacLab interpreter. Verify statistics and training dependencies in the external GR00T environment.

Use the generic planner rather than estimating collection size or optimizer steps manually:

python -m groot_finetune.tools.plan_groot_run \
  --target-successes 100 \
  --measured-success-rate 0.45 \
  --collection-safety-factor 1.125 \
  --episodes 100 \
  --frames-per-episode 699 \
  --action-horizon 16 \
  --epochs 5 \
  --global-batch-size 32

Supply camera capacity only when it has been measured for the active renderer. Treat camera-free collection and rendered evaluation as different capacity classes.

2. Collect and select timeout-eligible source episodes

Choose the source route declared by the contract:

• Native source rollout: run the registered policy in its dynamics environment.
• Native semantic rerender: use rerender_demo_visuals.py.
• External rollout: validate its manifest and use replay_record.py.

Run the planned number of camera-free environments in one parallel wave when the simulator supports it. Do not replace a requested large wave with repeated small batches without reporting the reason. If a measured eligibility rate is unavailable, run a small pilot, report its denominator, then plan the full wave.

Source eligibility is distinct from closed-loop task success. For the joint-rollout route, an episode is eligible only when it reaches its timeout. Select deterministically with:

python -m groot_finetune.tools.select_successful_episodes \
  --input <EXPORT_DIR_1> \
  --input <EXPORT_DIR_2> \
  --output <OUTPUT_ROOT>/selected \
  --target 100 \
  --expected-frames <CONTRACT_HORIZON>

Fail if successful inputs are insufficient, any selected episode has inconsistent time-series lengths, or the success marker is false. Preserve source paths and counts in the selection manifest.

3. Record or rerender semantic demonstrations

Write the semantic HDF5 contract:

data/demo_i/obs/<semantic_term>
data/demo_i/actions

Do not substitute an unrelated flat-policy recording format. Replay only selected successful episodes. Enable the contract's training visual randomization unless the user explicitly requests a diagnostic no-randomization run.

Require:

• exact requested demo count;
• exact contract horizon for actions, state terms, and every camera;
• finite numeric arrays and expected action dimension;
• RGB (T,H,W,3) uint8;
• source/replay state and action agreement within the route's tolerance;
• nonblank, correctly mounted sample frames from every camera.

4. Convert and audit LeRobot data

Run the converter from robotic_grounding/:

python -m groot_finetune.convert_to_gr00t \
  --input <RECORDING>/data.h5 \
  --output <OUTPUT_ROOT>/gr00t_dataset \
  --contract <EMBODIMENT_CONTRACT> \
  --task-profile <TASK_PROFILE>

The converter exports exactly the cameras declared by the embodiment contract. Verify that state/action slice order, camera keys, transforms, and dimensions match the config and closed-loop adapter.

Audit the complete artifact chain:

python -m groot_finetune.tools.audit_groot_run \
  --root <OUTPUT_ROOT> \
  --episodes <N> \
  --frames <CONTRACT_HORIZON> \
  --contract <EMBODIMENT_CONTRACT> \
  --task-profile <TASK_PROFILE> \
  --selected-export selected \
  --hdf5 recording/data.h5 \
  --dataset gr00t_dataset

The audit must confirm exact HDF5, Parquet, and per-video frame equality. Do not accept merely nonzero or approximately matching lengths.

5. Generate statistics and fine-tune

Use the same modality config for statistics, training, and open-loop evaluation. Compute --max-steps from usable samples, action horizon, epochs, and global batch size with the planner. Do not equate raw video frames with usable action-window samples.

Confirm meta/stats.json before training and the requested checkpoint afterward. Record consumed samples, optimizer steps, losses, runtime, model shards, processor, statistics, and trainer state.

6. Gate on open-loop evaluation

Evaluate every action modality key in the order declared by meta/modality.json. Require finite MSE/MAE and save the plot. Treat this as a data/model plumbing gate, not task-success evidence. Diagnose a failed open-loop gate before spending time on closed-loop evaluation.

7. Evaluate task success in closed loop

Use the inference task, reset behavior, visual mode, and embodiment contract together with the task profile's evaluator. A free-running policy must not inherit source trajectory-deviation terminations.

Require a one-episode smoke before a large evaluation:

• first post-warmup camera observations are nonblack;
• reset frame, finger state, curriculum, and randomization match the contract;
• only intended safety/timeout terminations are active;
• the episode reaches its expected horizon;
• structured JSON reports task metrics independently of termination.

Use the lifecycle wrapper for full evaluation:

bash robotic_grounding/groot_finetune/closed_loop/run_eval.sh \
  --gr00t-dir <ISAAC_GR00T_DIR> \
  --model <CHECKPOINT> \
  --container <ISAAC_CONTAINER> \
  --client-workdir <ROBOTIC_GROUNDING_PATH_IN_CONTAINER> \
  --task <INFERENCE_TASK> \
  --contract <EMBODIMENT_CONTRACT_IN_CONTAINER> \
  --task-profile <TASK_PROFILE_IN_CONTAINER> \
  --motion-file <MOTION_DIR_IN_CONTAINER> \
  --output-json <EVAL_JSON_IN_CONTAINER> \
  --episodes 100 \
  --num-envs <RENDERER_SAFE_ENVS> \
  --episode-horizon <CONTRACT_HORIZON>

Pass additional client arguments through --client-extra-arg. Timeout survival is source eligibility, never closed-loop manipulation success.

8. Retain successful recordings

Record complete episodes, decide success using the same task evaluator, discard failed frames, and retain only requested cameras and successful rollouts. For supported lift-and-hold tasks use --success-video-dir, --success-video-camera, and --max-success-videos with the lifecycle wrapper.

Probe every retained MP4 for codec, resolution, FPS, duration, and exact contract frame count. Visually inspect the beginning, interaction, success moment, and end.

9. Report and clean up

Report:

• source input/success/failure/selected counts and criterion;
• exact frame counts through every data representation;
• dataset keys, dimensions, views, and statistics;
• checkpoint path and training metrics;
• open-loop MSE/MAE;
• closed-loop denominator, task successes, rate, partial-progress distribution, and termination counts;
• successful recording paths;
• any requested parallelism that was reduced and the measured capacity error.

Stop servers and isolated containers. Preserve intentional run artifacts and do not delete unrelated user files.

Validation

Run repository contract tests for every affected embodiment, then:

python "${CODEX_HOME:-$HOME/.codex}/skills/.system/skill-creator/scripts/quick_validate.py" \
  .codex/skills/groot-finetune
bash -n robotic_grounding/groot_finetune/closed_loop/run_eval.sh
PYTHONPYCACHEPREFIX=/tmp/groot-finetune-pycache \
  python -m py_compile robotic_grounding/groot_finetune/tools/*.py
git diff --check

Completion requires audited data, statistics, checkpoint, open-loop result, task-level closed-loop result, viewable successful recordings when requested, and clean process shutdown.