remote-gpu-trainer — the DL Experiment Lifecycle

Overview

One skill for the whole arc of a DL experiment: RUN → VERIFY → DELIVER.

• RUN — get a long GPU job to start, survive, and finish, then get the result off the box. On a machine you own there is no meter; on a rented box the core insight is that you are a short-term tenant on someone else's machine — so the job is to detach the work, make the result outlive the instance, and stop the meter safely, not to provision a cluster. Platform-specific at the edges (one profiles/<platform>.md owns every path, proxy, billing verb, and spot rule), invariant at the core.
• Remote ownership boundary — this skill is the compute/control layer: it binds inputs, runs and verifies compute, and closes one run into export/<run-id>. Long-term project organization belongs to research-artifact-hygiene; any durable/local/cloud mirror begins only at a validated export and belongs to the generic mirror-research-artifacts skill. Never mirror a mutable active/ tree.
• VERIFY — is this number a bug, a real effect, or noise? A surprising result is a hypothesis, not a fact to report. Platform-agnostic.
• DELIVER — organize the result so every shipped number/figure/table is a deterministic function of one immutable evidence layer; provenance and cross-document consistency are locked by mechanism, not by a human remembering to update three documents. Platform-agnostic.

Two stances run through VERIFY and DELIVER: user sovereignty (the science — seed count, which samples, whether an aux channel exists — is the user's call; the skill organizes and discloses a tradeoff once, then stops nagging) and audit → disclose, not enforce (the skill is an honest auditor, not a gate guard — an integrity issue must surface with the conclusion it affects, but the skill never blocks the user from shipping). Mantra: "disclose it, or don't claim it."

Route first

1. RUN — own the box or rent it?
   • Local (a workstation/laptop you own, no meter) → references/run-local/ and profiles/local.md.
   • Rented / remote (any metered or shared box you don't own) → references/run-remote/, and pick your profiles/<platform>.md FIRST (it owns every path/verb/proxy the phases delegate to).
2. Then ALWAYS → VERIFY the result (references/verifying/). A green run is not a real number.
3. When publication synthesis is requested, optionally consult references/delivering/ for legacy/non-canonical publication synthesis guidance. It never defines the artifact layout: canonical runs, hardware evidence, trust records, figures and tables belong to research-artifact-hygiene.

Already debugging a model that won't converge / OOMs / hangs / NaNs, regardless of where it runs? Jump straight to references/training/ (the 8-file debug layer), then come back to VERIFY before you report.

Operating principles (the spine)

The load-bearing invariants. One line each; the full cross-platform set (10 invariants for the remote lifecycle) is in references/run-remote/principles.md — read it before Phase 0 of a remote run.

• Checkpoint-to-durable + idempotent resume is the universal spine. File-checkpoint to the durable location + unconditional load-latest-on-startup is the one mechanism that survives an SSH drop, a Slurm walltime kill, a K8s reschedule, a spot preemption, a Colab disconnect. The detach primitive (tmux / sbatch / Job) is the swappable plug; this is the invariant.
• Trust the artifact you loaded, not a log line that claims success. "synced / saved / done" lies under a silently-failed write; a watcher's own state is also a claim — reconcile it against the real process / artifact / pixels / bytes.
• Cheap checks before expensive compute. A 1–2 batch CPU smoke (logger off) kills import/config/ shape/scale bugs for ~free, before they bill GPU-hours.
• Cost and destructive actions are the user's call. Never auto-release/terminate, never delete durable files without confirmation; if cleanup can't free space, ask to expand the disk, don't silently shrink the experiment.
• Execution permission is not task authority. Sandbox / Full Access only controls whether a command can run. Keep operational authority and scientific promotion separate: once a bounded, non-overwriting delivery objective is authorized, same-scope diagnostics, verifier/schema version bumps, tests, hashes, and control-plane repairs do not require a fresh confirmation merely because they mint a new immutable ID. Re-ask for new billable compute, destructive/irreversible actions, science-protocol changes, metric promotion, publication, or any other material scope expansion. A standing unattended contract applies only after its own activation rule is satisfied.
• Make the control plane cheap and the data plane rare. Validate small schemas, identities, paths, and contract hashes before rereading multi-GB bundles. A synthetic fixture may test rejection behavior but may never invent the producer's positive schema; freeze a redacted real-shape fixture and prove its test is live. Recompute every large payload once per trust boundary—producer, independent remote acceptance, and local pull—not once per wrapper or verifier revision.
• One-way run closure. Mutable work stays under active/<run-id>; only a validated capsule may move from export/.partial/<run-id> to export/<run-id>, and failed closeouts move to quarantine/. AutoDL binds this exactly as /root/autodl-tmp/<project>/{cache,active,export/.partial,export/<run-id>,quarantine}. A closed export is fully isomorphic to canonical local runs/<run-id>: run.json, config.yaml, train.csv, best.pth, optional frozen last.pth, and test/<test-id>/{metrics.json,results.parquet,vis/<condition-id>/<task-native-role>/<sample-id>.png}. Every declared software test must include visualization coverage for all declared conditions × task-native roles × K fixed selected samples. Real capture/hardware results never enter this software capsule; close them separately as export/hardware/<hardware-run-id> (capture/decode/model-run bindings, no copied weights) or hand them to research-artifact-hygiene. A hardware test without machine-readable ground truth must declare that status and metric non-applicability; it keeps finite-forward rows and prediction/overlay visuals but never invents ground truth or GT-derived metrics. Caches and whole active trees never cross a mirror boundary. Layout and gates → references/run-remote/artifact-layout.md.
• Before teardown, prove the evidence outlives the host. Teardown is irreversible and "I scp'd it back" is just another log line. The gate is not "files copied" but "every number I reported re-reads from the local copy" — diff each claim against the pulled artifact, then write a provenance note next to the data (protocol, reference frame, caveats) so a later reader can retrace it without the chat log. Two traps: (1) what you did not pull is a decision, not an oversight — say which (checkpoints are usually re-derivable from config+seed if determinism is established; results are not); (2) re-read the inventory, don't trust its prose — a note saying "none of these are local" may mean none was trained here, not none is stored here; the two differ by everything when you are about to press destroy.
• Audit → disclose, not enforce. What is mandatory is disclosure, not the fix. An integrity finding (no disjoint val, leakage, test touched during selection, a number you can't re-derive) must ride with the conclusion — but the skill discloses, it does not block.

RUN — local (a box you own)

No meter, no teardown clock — the risks move from money to resource contention and your machine's stability. The discipline that does not relax: env hygiene, resource awareness, artifact/checkpoint care, and "state the seed." Start at profiles/local.md, then the matching doc:

• Env hygiene — never train/install in conda base on a persistent box; the 4-step gate (enumerate → pick the project env → confirm sys.executable → run) → references/run-local/env-hygiene.md.
• Launch & detach — nohup/tmux, log + alive probe, don't foreground-block → references/run-local/launch.md.
• Single-node multi-GPU — torchrun/accelerate DDP env contract, the first-run rank/hang basics → references/run-local/multi-gpu.md (multi-node → references/run-remote/multinode.md).
• Local OOM — the fit-it ladder on hardware you can't rent bigger → references/run-local/local-oom.md.

RUN — remote (a box you rent)

Pick your profile FIRST — it binds every concrete path/proxy/credential/billing verb/spot rule the phases delegate to. Mental verb model (one API across platforms; the profile binds each verb to real commands): up (rent+reach) → push (code/data on) → run (detached + checkpointing) → watch (durable monitor) → pull (results off + verify) → down (stop the meter).

Windows + Clash/Mihomo high-port SSH gate. Use OpenSSH direct first with strict host-key checking. A Paramiko fallback is allowed only after recorded banner_timeout, fake_ip, or tun_interference evidence; it must use a DoH-selected address and Windows IP_UNICAST_IF on the single socket handed to the SSH transport. Never treat authentication failure, host-key mismatch, connection refusal, or an unexplained error as proxy evidence. Never mutate system routes, DNS, proxy settings, or Clash/Mihomo configuration. Once fallback is authorized, you must not report the host unreachable or a live refresh blocked before a bounded Paramiko single-socket attempt completes or host identity fails closed. A transport failure proves only transport unavailability; it never proves that a remote run is completed, live, failed or stalled. Full parameterized decision ladder and offline planner → references/run-remote/ssh_transport.md §4A.

You're on…	Profile	Meter-stop verb (the trap)
AutoDL (deepest, battle-tested)	profiles/autodl.md	关机 stops meter, keeps disk (the AutoDL exception)
RunPod	profiles/runpod.md	terminate (stop still bills 2×; destroys volume disk)
vast.ai	profiles/vastai.md	destroy (stop bills disk forever)
Lambda	profiles/lambda.md	terminate (no stop state)
Paperspace	profiles/paperspace.md	destroy + release IP + delete storage
恒源云 / 矩池云 / Featurize / 揽睿星舟	profiles/china.md	per-platform (data disk often bills while stopped)
Bare SSH / Slurm / K8s / Colab	profiles/generic-ssh.md	manual (a forgotten box bills 24/7)

The 6-phase lifecycle (full per-platform checklist → references/run-remote/lifecycle_checklist.md): 0 env + storage-layout audit (df -i not just df -h, cgroup memory.max, checkpoint/inode budget) · 1 SSH + credentials (the prebuilt image is the env — don't conda create on a rental; secrets via stdin) · 2 identity-bound inputs + isolated active run + CPU-smoke gate before renting · 3 detached launch (probe, then hand back — never a blocking sleep) · 4 durable monitoring (the four-layer architecture → references/run-remote/monitoring_patterns.md; a session-bound watcher dies with the session) · 5 close active → export, verify/pull or hand the closed export to the generic mirror skill, then teardown.

Iron Law — teardown gate: NO release / terminate / destroy / file-delete until the remote durable result root has an external immutable PULL_MANIFEST.json built by scripts/aggregate_to_fs.sh + scripts/build_pull_manifest.py from an explicit expected roster (never embedded in canonical run.json; the mirror workflow's custody manifest is an additional layer, not a replacement for this one), the pull matches that exact roster + every byte size + SHA-256, every checkpoint loads, and scripts/verify_local.py writes local PULL_VERIFIED.json; then the user must still explicitly approve the cost-affecting action. A directory count, a size heuristic, an old loadable checkpoint, or "it looked done in the log" is not evidence. On most platforms the meter-stopping action is irreversible (deletes the disk) — confirmation matters more, not less.

Other remote references: ssh_transport.md (rsync/scp resumable, secrets-via-stdin, CRLF) · artifact-layout.md (active/export/quarantine boundary, canonical capsule and atomic-close contract) · spot-resilience.md (preemption grace, Young/Daly cadence, atomic-write resume) · china-network.md (mirrors + HF_ENDPOINT + the no_proxy trap) · parallel_ablation.md (fan-out independence + reconciliation) · multinode.md (NCCL/fabric, advanced) · production-matrix-acceptance.md (shared-disk multi-node patrol, producer→independent acceptance, control/science failure separation) · gotchas_universal.md (the full U1–U44 catalog with a grep index).

When training itself breaks (the model, not the platform)

Once the box runs, training breaks in its own ways — local or remote, the same debug layer (references/training/, 8 files; each entry symptom → root cause → fix with cited docs). Route by symptom:

• OOM / won't fit (CUDA-VRAM or host-RAM, OOM-at-a-step, the fit-it ladder) → oom-memory.md.
• Multi-GPU launch / HANGS (torchrun/accelerate/deepspeed env contract, DDP/FSDP/ZeRO) → distributed-launch.md.
• NaN / Inf / loss spikes (fp16/bf16/tf32, AMP/GradScaler, LLM divergence) → precision-stability.md.
• Too slow (GPU- vs data- vs comms-bound, dataloader knobs, torch.compile traps) → throughput-profiling.md.
• Resume bugs (full-state + sharded save/resume; epoch restart, reshuffle, scaler/EMA dropped) → checkpoint-resume.md.
• Per-domain gotchas (LLM, vision det/seg, diffusion, RL, multimodal/VLM) → by-domain.md.
• Runs but won't learn (overfit-one-batch, params-not-updating, LR/schedule, loss-function footguns, freezing) → convergence-debugging.md.
• Dataloader correctness (worker-RNG aug duplication, IterableDataset sharding, RGB-vs-BGR / ÷255 / set_epoch) → data-pipeline.md.

VERIFY — is the number real?

Before you trust or report any metric, ablation delta, or "it works now": classify it bug / effect / noise, hold a comparison to exactly one changed variable, and probe leakage / fair-comparison / variance / metric-direction. A number you can't re-derive from the saved artifact is not a result yet. Stance: audit → disclose — surface an integrity issue with the conclusion, never silently pass or hard-block.

• Full methodology (the 14-section probe ladder + the 6 invariants) → references/verifying/methodology.md.
• Constant / degenerate output, real == shuffle, model-ignores-input → references/verifying/representation-collapse.md.
• A green smoke that hides undertraining vs a real bug; loss-low-but-samples-bad → references/verifying/smoke-hidden-failures.md.
• For an atomic producer bundle, run scripts/verify_artifact_bundle.py on a different node; compare two independently produced acceptances with scripts/compare_acceptance.py. A hash match proves identity, a safe load proves checkpoint structure, and a fresh evaluator proves the metric—none substitutes for the others.

State the metric's direction when comparing (PSNR/SSIM/mAP ↑ better; LPIPS/NMSE/loss ↓ better) — never assume. Tracker forensics / pruning duplicate runs → scripts/wandb_forensics.py.

DELIVER — legacy/non-canonical publication synthesis guidance

On a remote rental, delivery ends at the validated closed export/<run-id> capsule. Do not turn the compute host into the long-term project archive, figure workshop or mirror manager; hand canonical organization to research-artifact-hygiene and replicas to mirror-research-artifacts.

The references/delivering/ group is retained only for publication-synthesis principles such as generated-not-transcribed reporting, claim reconciliation, disclosure and pixel re-open QA. It is not an artifact-layout authority. Any directory/manifest example there that differs from research-artifact-hygiene is legacy and non-canonical.

For synthesis, read evidence from canonical runs/<run-id>/test/<test-id>/{metrics.json,results.parquet,vis/} or hardware/runs/<hardware-run-id>/test/<test-id>/...; never create a parallel results/<exp-id>/runs/ tree, checkpoint subdirectory, qualitative tree or selected symlink. Figures/tables use the canonical flat workshops (figure.json/table.json, root-level source, build.py, final.*) rather than README portals, nested output directories or provenance sidecars. The fixed selection roster remains the canonical run.json-bound _trust/selections manifest; a publication montage may cite existing atomic PNGs but may not redefine the run's visualization roster.

This skill owns one execution attempt, not project-wide closeout state. Route authorization gates, node queues/heartbeats, legacy checkpoint acceptance, and the separate training/evaluation/pull/paper axes to supervise-research-closeout; that controller must call this skill rather than duplicate its SSH or trainer logic.

Legacy synthesis navigation: references/delivering/data-architecture.md states the boundary and current flat-workshop mapping; principles.md, figures.md, delivery-gate.md and completeness-reconciliation.md provide advisory synthesis/QA checks only. None may override canonical paths, manifests or retention rules.

Companion skills (all OPTIONAL — this skill is standalone)

Recommended separate installs that deepen RUN / VERIFY / DELIVER; the skill needs none of them and works fully standalone. One-line-each list, what each adds, and the no-companion fallback → references/companions.md. In short: figure drawing (nature-figure), data availability (nature-data), experiment verification (the experiment-verifier agent), parallel ablation (superpowers:dispatching-parallel-agents), durable artifact mirroring (mirror-research-artifacts), and — one layer above — an idea→conclusion orchestrator (auto-research-pipeline: human gates + stage wiring; this skill executes, that one decides when each stage fires and what a human signs).

Getting better over time

The skill is static, but every run can teach it a gotcha — without corrupting it. Protocol → references/self-improvement.md: only sediment a root-caused, reproduced, generalizable gotcha (a one-off flake is a hypothesis, not a gotcha); route user/project-specific facts to the host's memory and generalizable ones to a proposed catalog edit; never silently rewrite a skill file — draft the symptom → root cause → fix and let the user approve. Platform facts carry a verified <month> stamp — re-verify any teardown/billing fact against current docs before betting money or data (scripts/check_staleness.py).

Bundled resources

Load only what the current phase needs (the body sections above name the individual files).

• references/run-local/ — own-a-box: env-hygiene · launch · multi-gpu · local-oom.
• references/run-remote/ — rented-box: principles · lifecycle_checklist · artifact-layout · monitoring_patterns · ssh_transport · spot-resilience · china-network · parallel_ablation · multinode · production-matrix-acceptance · gotchas_universal (U1–U44).
• references/training/ — the DL-training debug layer (8 files; local/remote-agnostic) — routed above.
• references/verifying/ — is-the-number-real: methodology · representation-collapse · smoke-hidden-failures.
• references/delivering/ — legacy/non-canonical publication synthesis guidance only: advisory principles · canonical-boundary data-architecture · historical manifest/figure notes · delivery gate · completeness reconciliation. research-artifact-hygiene owns every canonical path and schema.
• references/companions.md (optional skills + fallbacks) · references/self-improvement.md (capture-a-gotcha loop).
• profiles/<platform>.md — per-platform substrate (7 rental profiles + local.md; _schema.md = the fields).
• scripts/ — wrappers (run_one/run_queue), monitors (mem_monitor, gpu_health, health_patrol.sh.template, reap_vram_zombies.sh), transfer (download_loop, aggregate_to_fs, build_pull_manifest.py, setup-china-mirrors), verify_local.py, delivering (manifest_scaffold.py, reconcile.py, repro.sh.template), atomic acceptance (verify_artifact_bundle.py, compare_acceptance.py), wandb_forensics.py, check_staleness.py.
• examples/autodl_sweep/ — one runnable worked case · evals/ — the regression harness.