Health preflight (the doctor before you spend)

There is no npa doctor. The readiness surface is npa workbench health, and it has exactly two public commands. Run them before anything that costs GPU time or provisions cloud resources.

npa workbench health preflight --json
npa workbench health preflight --checks nebius --json
npa workbench health access --json

All three exit non-zero on any FAIL. That is the point: they are gates, not reports. Use --warn-only only when you deliberately want the report without the gate (for example while collecting a status snapshot), never to make a red run green.

What each command answers

preflight answers "do I hold the credentials nearly every workbench tool needs?" — one PASS/WARN/FAIL/SKIP row per check over Hugging Face, NGC, S3, and Token Factory. Request the optional nebius check before provisioning:

npa workbench health preflight --checks all
npa workbench health preflight --checks nebius --json
npa workbench health preflight --checks s3,token_factory
npa workbench health preflight --checks all --offline  # presence only; Nebius is SKIP

Valid --checks values are all, hf, ngc, s3, token_factory, nebius. The default remains hf,ngc,s3,token_factory, so hosted-inference work does not require a Nebius Cloud profile. Explicit all includes nebius. Empty selections and unknown check names are errors, including unknown names combined with all. Repeated names run once.

Online nebius runs the selected Nebius CLI profile through iam whoami and iam get-access-token with browser launch and update checks disabled. It discards both commands' output and removes ambient NEBIUS_IAM_TOKEN and NEBIUS_IAM_TOKEN_FILE values so a stale token cannot mask profile readiness. PASS requires both calls to succeed. In offline mode it returns SKIP because a local profile file is not proof of usable authentication.

A failed Nebius identity probe does not establish that a token expired. Profile configuration may be missing, or access or connectivity may have failed. For diagnosis or authorized human login on an operator VM, see skills/atomic/vm-nebius-auth/SKILL.md. A readiness-only request does not authorize login or credential changes. Authentication PASS also leaves project permissions, quota, and capacity unverified.

Profile selection uses NPA_NEBIUS_PROFILE, then NEBIUS_PROFILE, then the CLI's active/default profile. The project selector does not choose an auth profile. PASS proves authentication; permission to provision a particular resource still depends on the selected identity's access to that project.

Online hf authenticates against Hugging Face whoami-v2; public repository metadata is not sufficient. Online ngc performs a registry token exchange; that proves the key, not entitlement to every NGC artifact. access performs the capability-specific repository/artifact probe.

access answers the different and more specific question "is my token actually entitled to fetch bytes from the gated assets this capability pulls?" It probes a representative payload path at the exact catalog revision with HEAD or a one-byte Range request. Metadata access is not proof. Holding an HF token is not the same as having account entitlement, and Ready is technical fetch evidence, not proof of legal acceptance.

npa workbench health access --capability all
npa workbench health access --capability cosmos3,groot

Capabilities: all, cosmos, cosmos3, cosmos3-serving, groot, lerobot, nurec, paidf, sim2real, sonic, token_factory, vlm_eval.

For anything still gated, access prints the exact "Agree and access repository" URL. Hugging Face gated licenses must be accepted interactively on the model page — there is no API that accepts them for you, so no amount of retrying or re-tokenizing will clear a gate. Open the printed URL, accept, then re-run. scripts/accept-model-access.sh collects the URLs for a batch.

Ordering: preflight before provisioning, not after

The recurring cold-start failure this prevents is a mid-run stop after you have already paid for a cluster. Run the checks in this order:

1. npa configure --show — confirm the project stanza, bucket, and endpoint you think you are using are the ones on disk.
2. npa workbench health preflight — credentials exist and authenticate.
3. npa workbench health preflight --checks nebius: the selected Nebius CLI profile can resolve identity and mint an IAM token before provisioning.
4. npa workbench health access --capability <the one you will run>: gated model entitlements for that capability only; all is slower and reports failures you do not care about today.
5. Only then npa provision-if-absent, npa cluster up, or npa workbench workflow submit.

Image pullability is a separate gate that health does not cover; see npa workbench workflow preflight-images <spec.yaml> in skills/atomic/submit-workflow/SKILL.md. A green health report with an unpullable image still hangs in ImagePullBackOff.

A green s3 row does not mean the submit is execution-ready. The generic s3 check lists its configured bucket. Submit resolves one effective target, verifies provider project/tenant/region and exact bucket ownership before any write, then writes and reads a unique object in each actual workload output, ledger and source-staging prefix using the executing credential pair. It also checks the exact Kubernetes context and requested GPU product/shape, including single-node requests. An explicit destination never falls back to another writable bucket. A mismatch or unknown ownership blocks creation, even with --skip-preflight. A denied task prefix can therefore follow a green generic health check:

Error: execution preflight storage_access: exact output prefix write failed (authorization)

Use submit's execution diagnosis. Reconcile the selected project and the exact endpoint/bucket/prefix before changing IAM. --s3-bucket / --s3-prefix bind the actual spec and ledger; --s3-endpoint binds both probe and worker endpoint. S3 keys are selected as a pair from one source, with non-secret provenance; an incomplete explicit pair cannot borrow a saved principal's other key.

The credential preflight resolves its bucket from NPA_CHECKPOINT_BUCKET, then NEBIUS_S3_BUCKET, then saved credentials. Setting only NPA_S3_BUCKET changes the workflow destination but does not select the credential preflight bucket. When checking an explicitly authorized workflow destination, set NPA_CHECKPOINT_BUCKET to an unsigned s3:// URI for that same bucket in the private process environment and use the matching endpoint and credentials. The list probe requires a URI, so a bare bucket name fails before S3 access. Verify bucket ownership first; do not change shared configuration or provision storage to repair a stale default selection.

Persisting credentials you already hold

If the tokens are in your environment but not in ~/.npa/credentials.yaml, persist them instead of re-exporting them in every shell:

npa workbench health access --save-env-credentials
npa configure --save-env-credentials

Both perform an atomic 0600 write and never print the values. This is the fix for "it worked in my shell but the submit could not resolve the secret": submit resolves each requested secret from the explicit process environment first, then the selected project's configured NPA credentials.

Hidden sim2real check

npa workbench health sim2real exists but is hidden, and is specific to the 14-stage Sim2Real graph rather than general readiness. It adds cluster-shaped checks (config, coherence, s3, registry, tokens, cluster) including schedulable GPU count and kube-context pinning:

npa workbench health sim2real --checks all --json

Use it from skills/workflows/sim2real-operate/SKILL.md, not as the generic preflight. Its cluster check counts schedulable nvidia.com/gpu but does not yet match the requested GPU product against available node products, so it can pass on a cluster that has GPUs of the wrong kind for your spec. Confirm the product with npa workbench workflow gpus --cluster <name>.

Gotchas

• --offline proves presence, not validity. It skips live provider probes. Existing credentials may pass presence checks, while nebius returns SKIP because profile configuration is not authentication proof. Use offline only where there is genuinely no egress.
• A SKIP is not a PASS. Checks skip when the corresponding capability is not configured. If you are about to run a Cosmos stage and the NGC row says SKIP, you have not verified anything about NGC.
• preflight does not check Kubernetes or SkyPilot. Cluster readiness is npa skypilot verify --cluster <exact-context> and npa cluster status; registry pullability is workflow preflight-images. Three separate gates, three separate commands.
• Stale NEBIUS_IAM_TOKEN defeats provider calls even when health is green. The Nebius provider prefers an ambient (often expired) token over the fresh CLI token. unset NEBIUS_IAM_TOKEN NPA_NEBIUS_IAM_TOKEN before provisioning or submitting.

Verify

npa/.venv/bin/python -m pytest npa/tests/guardrails/test_skills_index.py -q

With an authenticated operator profile, exercise the real CLI's successful, missing-profile, and offline paths (including stale ambient token scrubbing):

NPA_INTEGRATION_E2E=1 npa/.venv/bin/python -m pytest \
  npa/tests/e2e/test_nebius_auth_preflight.py -q