NPA Agent VM

The NPA agent is a public HTTPS workbench VM with basic-auth UI, grounded chat, Sim Assets + Cameras panels, embedded Rerun viewer, and Sim2Real submit hooks.

When To Use

• Deploy, bootstrap, or verify an agent VM (npa agent …)
• Fresh deploy / teardown loops: load skills/workflows/agent-fresh-operate/SKILL.md
• Debug chat hallucinations (raw GET /api/… replies) or false “Loaded Franka” messages
• Fix Rerun iframe black screen (basic auth + wasm fetch)
• Operate customer HTTPS access and sign-in UX
• Describe this / viewer feedback: load skills/atomic/agent-visual-feedback/SKILL.md

Bootstrap And Verify

npa/.venv/bin/npa agent fresh-setup --project <alias> --name agent \
  --project-id <project-id> --tenant-id <tenant-id> --region <region> \
  --tf-var ssh_cidr_block=<operator-cidr> \
  --tf-var application_cidr_block=<operator-cidr>
npa/.venv/bin/npa agent bootstrap --project rtxpro --name agent
# Existing agents missing credentials: refresh long-lived npa-agent SA + restage VM env
npa/.venv/bin/npa agent bootstrap --project rtxpro --name agent --refresh-credentials
NPA_AGENT_CHAT_LIVE=1 npa/.venv/bin/npa agent verify-live --project rtxpro --name agent
bash npa/scripts/verify_agent_franka.sh
bash npa/scripts/verify_agent_rerun_bundle.sh
bash npa/scripts/verify_byof_onboarding_live.sh

Audit the capability surface without a VM

Before claiming the agent does or does not support something, render the exact backend.py that bootstrap installs, run it against a sandbox state root, and probe it. No cluster, no VM, no Token Factory call, no quota:

npa/.venv/bin/python npa/scripts/audit_agent_capabilities.py --json audit.json

It reports the registered route count, each parameterless GET's real outcome, and whether every advertised chat intent still matches and produces a grounded reply.

Add --serve-live to probe a real uvicorn backend:app process on loopback, started with the same argument list as the deployed npa-agent-backend systemd unit, instead of driving the ASGI app in-process:

npa/.venv/bin/python npa/scripts/audit_agent_capabilities.py --serve-live

Use the served tier when the question is "would this work on a VM", because it is the only tier that exercises import-time and lifespan behavior under the real server and the real websocket flags. It needs uvicorn and websockets in the venv (bootstrap installs both on the VM; a repo venv may not have them). The two offline tiers should report identical route counts and outcomes — a divergence is itself the finding. Neither binds a public port, so both are safe on a shared machine.

Point --base-url at a deployed agent to audit the VM itself. Routes are then enumerated from the deployment's own /openapi.json, and the report diffs that against the local render, which is how you find a VM running older code:

npa/.venv/bin/python npa/scripts/audit_agent_capabilities.py \
  --base-url https://<agent-ip>/api \
  --auth-env ~/.npa/agents/<project>/<name>/auth.env \
  --insecure --allow-mutations

--insecure is normal — the ingress uses a self-signed certificate. The capability probes POST to /chat, run memory, and retrieval, so they need --allow-mutations against a real deployment; route probing is read-only and always runs. routes_missing_on_deployment and routes_absent_from_render should both be 0.

A deployed agent legitimately reports better outcomes than the sandbox, and the difference tells you what the sandbox could not know: /artifacts/runs answers 200 once S3 is staged (sandbox: 400), /sim-viz/rrd and /sim-viz/rrd-blob answer 200 once the stock demo recording exists (sandbox: 404), and the LeIsaac routes return 404 "No LeIsaac runtime is registered" rather than the sandbox's 403 transport refusal because real public HTTPS satisfies that guard. Most important: chat workflow authoring emits a runnable YAML on a configured deployment where the sandbox correctly declines with unresolved placeholders — the fail-closed refusal is a configuration state, not a capability ceiling.

Read the outcome classes rather than a pass/fail count:

• answered — the capability responded.
• needs_arguments — required query parameters missing (by design).
• gated — refused by a transport/auth guard (LeIsaac requires same-origin authenticated HTTPS, so 403 off a real deployment is correct).
• absent_in_sandbox — depends on staged state a sandbox has no reason to have (.rrd recordings, the LeIsaac client bundle).
• error — the only class that indicates a defect.

Routes that need a staged dependency say so instead of returning empty success: GET /api/artifacts/runs answers 400 with "S3 discovery is not configured on this agent" when no bucket/credentials are staged. Treat that as the honest answer it is, not as a broken route — and never report it as "no runs found".

With verified configured storage, npa agent deploy creates or reuses a dedicated long-lived npa-agent service account and verifies an editor grant on the exact deployment project. The VM identity and configured S3 credentials are resolved separately; a failed project grant stops this path before deployment. The legacy storage-bootstrap path retains its saved-credential fallback. Deploy persists ssh_key_path and non-secret deployment identity on the agent record; storage credentials remain in the owner-only project credential store. Bootstrap stages llm.env, s3.env, and nebius.env on the VM and resolves SSH from the agent record (or --ssh-key / NPA_SSH_KEY) — not from workbench SSH config.

Deploy/bootstrap persists pre-mutation through health-verification checkpoints and emits secret-free structured heartbeats during long calls. After lost client transport, reconcile the exact remote setup marker and authenticated /api/models: adopt matching healthy evidence, resume incomplete phases, and preserve ambiguous/mismatched evidence without replacing the VM.

If service installation completed before credential staging lost its SSH connection, a retry can reuse the private remote installer receipt. Reuse requires the same verified owner and an identical rendered installer (source revision, settings, and authentication included); the original bootstrap timestamp is preserved. Credentials are restaged and deployment identity and health are checked again. Missing or changed receipts run installation normally, and an explicit bootstrap of a healthy agent still reinstalls services.

Agent VM creation is credential-free: Terraform/cloud-init receives no S3 HMAC keys, product tokens, or basic-auth password. After the exact VM identity and SSH channel are verified, bootstrap stages runtime credentials with owner-only SFTP uploads and atomic installs. A client failure resumes staging on that VM; it does not recreate the instance or copy secrets into Terraform state/user-data.

SSH and application ingress are separate and empty by default. Agent deploy and fresh-setup require explicit ssh_cidr_block and application_cidr_block Terraform values because verified bootstrap and public HTTPS health checks use those paths. Any /0 additionally requires the matching allow_world_open_ssh=true or allow_world_open_application=true acknowledgement. Post-deploy reconciliation reuses the application CIDR and never creates a world-open fallback rule.

All npa agent … and nebius IAM commands run on the operator/dev VM. The agent VM only receives staged /opt/npa-agent/*.env files. For human no-browser profile setup or recovery on a remote operator/dev VM, load skills/atomic/vm-nebius-auth/SKILL.md. Do not use that flow to replace the agent VM's attached-service-account metadata profile.

Legacy storage-bootstrap credential fallback

When bootstrap must provision storage, its legacy fallback tries in order:

1. npa-agent SA — create or reuse if IAM allows
2. Saved operator credentials — ~/.npa/credentials.yaml S3 keys + optional nebius.service_account_id
3. Project terraform_state keys — projects.<alias>.terraform_state from the original deploy
4. SA id discovery — parse lerobot-training id from IAM errors when agent-sa cannot read IAM

Bootstrap persists the resolved SA id into the agent record, credentials block, and ~/.npa/credentials.yaml when discovered.

For the full BYOF live pipeline (agent + container + GPU on the configured project):

export NPA_E2E_PROJECT=rtxpro
export NPA_BYOF_LIVE_PIPELINE=1
bash npa/scripts/verify_byof_onboarding_live.sh

Project Kubernetes settings resolve from ~/.npa/config.yaml (projects.<alias>.kubernetes) and ~/.npa/clusters/<cluster>/kubeconfig — not from any operator VM hostname.

For real BYOF container build/push/inspect, set NPA_BYOF_LIVE_CONTAINER=1 and run bash npa/scripts/verify_byof_onboarding_live.sh on a host with Docker and explicit credentials for the operator-controlled registry. Default validation repo is LeIsaac; override with NPA_BYOF_REPO_URL / NPA_BYOF_REPO_REF.

For full BYOF GPU smoke (SkyPilot submit), also set NPA_BYOF_LIVE_GPU=1 and run the same script on a host with Docker, nebius, sky, and any required explicit registry pull access. GPU train YAML and SkyPilot config resolve from the project kubernetes block (gpu_profile: rtxpro, byof_train_yaml, skypilot_config).

Auth secrets live at ~/.npa/agents/<project>/<name>/auth.env (AGENT_USER, AGENT_PASSWORD). Agent bootstrap now stages operator config + credentials on the VM at ~/.npa/{config,credentials}.yaml so the VM can run infra commands without re-entering project metadata. Bootstrap also installs Nebius CLI (if missing) and seeds a cursor-sa profile backed by /mnt/cloud-metadata/token when the VM has attached SA metadata; if token-backed profile setup is present but unusable, bootstrap fails fast instead of silently skipping it. Token Factory model selection is configurable via --llm-model and --llm-models (NPA_AGENT_LLM_MODEL and NPA_AGENT_LLM_MODELS on the VM), with /api/models exposed for UI/model picker refresh.

Customer HTTPS Access

• Public URL: https://<public_ip>/ (self-signed cert on VM IP)
• npa agent status may print that canonical HTTPS endpoint in local operator output or an explicitly requested handoff only after its authenticated probe succeeds and an unauthenticated request returns 401, proving HTTP Basic Auth is enforced. The status payload records endpoint_disclosure_allowed=true and basic_auth_enforced=true when this narrow exception applies.
• This exception never covers direct_url, credential-bearing URLs, usernames, passwords, auth-file contents, or an endpoint whose protection was not just verified. Keep those values in the owner-only 0600 credential store.
• Sign-in form at /login-help.html and /welcome (mobile-safe XHR/fetch sign-in; URL-embed fallback on desktop only)
• On phones: open /healthz first to accept the self-signed certificate, then sign in at /login-help.html
• Mobile chat uses sessionStorage basic-auth fallback — sign out by clearing site data or use /login-help.html again
• All fetch calls use credentials: "include" for session basic auth
• Never suggest localhost, 127.0.0.1, or port 8080 — use same-origin /api/… paths

Optional LeIsaac UI configuration

The exact operator-config key is projects.<project-alias>.agents.<agent-name>.ui.leisaac_enabled in ~/.npa/config.yaml (or $NPA_CONFIG_DIR/config.yaml). Only the YAML boolean true enables it; absent/false, strings, numbers, and malformed UI sections fail closed. The default page has no LeIsaac navigation and makes no LeIsaac capability requests. Never add a browser enable/disable control, localStorage opt-in, or URL override.

Bootstrap reads the named agent record and renders the setting into static HTML. After changing the operator config, run npa agent bootstrap --project <alias> --name <name> against that same agent, then reload open browser pages. A backend/nginx restart or reboot alone does not regenerate the UI. Existing record writes must preserve ui settings. The flag exposes the normal tab and readiness checks; it does not launch infrastructure or replace LeIsaac's transport/controller authorization. See operator docs for the minimal YAML example.

For a flag or UI change, run npa/tests/cli/test_agent_ui_config.py and the real Cypress suites (cd npa/tests/browser && npm run cy:mock). Exercise the rendered disabled and enabled pages: absent/false must have no navigation or capability polling even with an old localStorage opt-in; true must render readiness and perform status checks. Validate both states on the same live deployment when authorized, preserving its original explicit opt-in or restoring the false default afterward. Agent Access browser coverage must select multiple and single available project/bucket options and verify dependent details and artifact-action provenance, using actual DOM events.

Run npm run cy:live-access from npa/tests/browser with the existing NPA_AGENT_BASE_URL, NPA_AGENT_USER, and NPA_AGENT_PASSWORD supplied through the protected runner environment. It checks real access selections and expects LeIsaac hidden by default. Set NPA_AGENT_EXPECT_LEISAAC=true only to assert an already enabled deployment; this test expectation does not enable the feature. Keep all live runner output and screenshots in access-controlled evidence outside Git.

Chat Maturity Patterns

Typed GPU placement failures and consented preemptible fallback use skills/atomic/gpu-allocation-fallback/SKILL.md and the grounded /api/agent/gpu-allocation/* routes.

Intent router in npa/src/npa/cli/agent_chat.py (embedded in remote backend.py at bootstrap).

Routed intents are listed below with trigger phrases verified to match (npa/scripts/audit_agent_capabilities.py exercises exactly these). Do not assume an unlisted capability is missing without re-running that audit, and do not add a rule for one of these without checking which existing intent already claims the phrasing — earlier rules win.

Run / viewer state

Intent	Verified trigger	APIs
sim2real_status	"what is the current sim2real status"	sim-viz/status, workflows/sim2real/status
watch_sim	"watch the sim until the blob and iframe both report success"	sim-viz/status, sim-viz/rrd, sim-viz/rrd-blob
start_sim2real	"start the sim2real pipeline"	workflows/sim2real/submit
drive_sim2real	"autonomously drive the sim2real outer loop"	agent/sim2real/drive, workflows/sim2real/{submit,status}
load_franka	"load the franka demo"	sim-viz/load-franka-demo, sim-viz/status
list_recordings	"list the available recordings"	sim-viz/recordings, sim-viz/runs
find_artifacts	"what can I view?"	artifacts/runs, artifacts/run/{run_id}, sim-viz/load-artifact
foxglove_viewer	"open foxglove"	foxglove/status, foxglove/config, foxglove/load-artifact
sim_assets	"show me the sim assets"	sim-assets, sim-assets/selection
cameras	"which cameras are selected"	sim-assets/cameras

Workflow authoring — all four templates share workflows/draft, workflows/validate, workflows/plan. Picking the wrong one is the common failure, so match the qualifier, not just the word "workflow":

Intent	Verified trigger	Picked when
create_vlm_rl_workflow	"create a sim-to-real workflow yaml"	sim-to-real authoring, or any "quality gate" / "policy rollout" / "heldout eval" / "vlm critic" phrasing
create_gate_workflow	"create a token factory gate workflow"	Token Factory / Cosmos scene-reasoning gate
create_loop_gate_workflow	"create a sim2real workflow with a loop gate"	explicit "loop gate" / "decision gate"
create_data_factory_workflow	"create a PAIDF workflow yaml"	PAIDF / video augmentation / scenario fan-out
create_rl_policy_workflow	"create an RL policy training workflow"	RL policy training
create_workflow	"create a 2-step sim2real npa.workflow"	explicit two-step, or generic npa.workflow
validate_workflow	"validate the saved workflow YAML"	validate fenced YAML, or the saved draft, without rewriting it
plan_workflow	"plan the saved workflow YAML"	validate and plan that exact specification without submission
workflow_execute_guidance	"how do I actually run this workflow"	validate/plan/submit + tools

create_vlm_rl_workflow is matched before create_gate_workflow and claims "quality gate", so reach the Token Factory gate through its own wording.

Infrastructure

Intent	Verified trigger	APIs
infra_backends	"which infra backends are available"	infra/k8s, infra/provision, workflows/submit
mk8s_provision	"provision an mk8s cluster"	infra/mk8s, infra/mk8s/provision, infra/k8s
live_infra_loop	"run the live infra loop"	infra/k8s, infra/provision, workflows/*, tools
soperator	"deploy a slurm cluster"	infra/soperator/{validate,deploy,status/{name}}
tenant_resources	"what tenant resources do I have"	resources
configure_s3	"configure S3 bucket access"	tools (nebius-infra)
onboard_solution	"containerize this github repo as a workbench solution"	tools, workflows/validate, workflows/plan
cosmos3	"set up cosmos3"	tools (skill steps run on the operator machine)

Tool capability questions — all answer from tools. "what can <tool> do" is a verified trigger for each of cosmos, lancedb, sonic, lerobot, groot, genesis, mjlab, isaac lab (*_capabilities), and component_capabilities answers the generic "what components are available".

Every routed intent must have an INTENT_APIS entry. That map is not just reply metadata: _semantic_route derives the semantic fallthrough's known_intents from its keys, so an intent absent from it can never be reached by a paraphrase the regex misses. test_every_intent_declares_its_apis enforces this.

BYOF onboarding: load skills/workflows/byof-onboard/SKILL.md (source of truth for base profiles, workloads, live verify). Chat replies reference this skill path — do not paste the full procedure into agent_chat.py.

Rules:

• Never return only GET /api/… as the chat reply
• Always unpack run_id, stage, rerun_ready, camera in markdown (**key**: \value``)
• Grounded replies set "grounded": true and "apis_used": ["sim-viz/status", …]
• LLM fallback injects format_live_context_block(state) JSON snapshot into the system prompt
• Workflow drafting should pick a template by intent + workflow capabilities (sim2real loop-gate, VLM-RL loop, tokenfactory-cosmos gate, or simple two-step), not by hardcoded endpoint-only replies.
• PAIDF and sim-to-real drafting resolves the staged agent bucket and configured Kubernetes accelerator/profile before rendering. A conflicting requested GPU fails closed; absent infrastructure remains an explicit placeholder/warning.
• Chat-generated sim-to-real loads the single canonical compositional npa.workflow/v0.0.1 graph. It must not emit the retired workbench.sim2real.run monolith or legacy demo/echo toolRefs.

Workflow Draft / Validate / Plan / Submit Loop

These workflow operations are provider-neutral: the caller owns its model or reasoning configuration, while NPA owns validation, translation, and execution. The complete bounded lifecycle is documented in docs/workbench/agent-workflow-operations.md.

Use the VM as a grounded drafting surface, then run operator-machine commands for real workflow execution:

# Agent VM draft surface
GET  /api/workflows/draft
POST /api/workflows/draft
POST /api/workflows/validate
POST /api/workflows/plan
POST /api/workflows/submit

# Operator machine (authoritative execution path)
npa/.venv/bin/npa workbench workflow validate-spec <spec.yaml> --json
npa/.venv/bin/npa workbench workflow plan-spec <spec.yaml> --run-id <run_id> --json
npa/.venv/bin/npa workbench workflow run-spec <spec.yaml> --plan-only --scheduler-plan --json

POST /api/workflows/draft saves a draft (it requires a yaml body and returns validation + plan); it does not generate one. Chat is the generator.

Chat emits workflow_yaml only after validation and planning both succeed. Without a staged bucket and configured accelerator it returns Could not generate runnable workflow YAML yet. and names the unresolved placeholders (<configure-s3-bucket>, <configure-gpu-accelerator>). That is the fail-closed contract working, not a defect — do not "fix" it by relaxing the gate, and do not report a placeholder refusal as a broken drafting path. It also means workflow authoring cannot be fully exercised on an agent with no staged infrastructure; npa agent verify-live asserts the YAML branch, so run it against a bootstrapped VM.

Guidance:

• Keep config grouped: runtime knobs first, then *_uri keys under prefix paths.
• For multi-step specs, include explicit state descriptions, resources, inputs/outputs schemas, loop/gate transitions, and terminal leaves.
• If transitions exist, plan with --assume-decision promote_checkpoint|loop_back.

Rerun Iframe Fix

Rerun wasm inside /rerun/?url=… cannot send HTTP basic auth and does not reliably consume parent-created blob: URLs across browsers. Bootstrap publishes the active recording to unauthenticated /rerun/recordings/sim2real.rrd for the iframe, while the parent page still validates authenticated access by fetching /api/sim-viz/rrd-blob.

Use this order:

1. Publish/copy the .rrd to /opt/npa-agent/recordings/sim2real.rrd.
2. Point iframe url= at same-origin /rerun/recordings/sim2real.rrd?t=....
3. Also fetch("/api/sim-viz/rrd-blob", { credentials: "include" }) as the authenticated blob/bytes health gate and fallback.

Do not point the iframe directly at /api/sim-viz/rrd (black screen / auth failure in browser contexts).

Submitted Sim2Real runs must not reuse the stock Franka/demo .rrd as if it were run-specific data. POST /api/workflows/sim2real/submit and chat requests such as "start/run the Sim2Real pipeline" should launch the agent-local Sim2Real runner, update the standalone Run status/logs panel from /api/workflows/sim2real/status and /api/workflows/sim2real/runs/{run_id}, and only mark Rerun ready after a real run recording URI is present. Run-specific recordings should open on a useful 3D scene overview (world/table, world/franka/*, world/cube) plus rollout/signal panels, not only sparse rollout or held-out image streams.

HTTP API Reference

All paths are under /api/ (nginx proxies to FastAPI backend on :8787).

GET /api/health

{
  "ok": true,
  "tool_refs": 19,
  "capabilities": {
    "gpu_allocation_fallback": {
      "status": "available",
      "grounded": true,
      "routes": [
        "POST /api/agent/gpu-allocation/attempt",
        "POST /api/agent/gpu-allocation/consent"
      ]
    }
  }
}

The GPU allocation routes are embedded-backend capabilities, not workbench toolRefs. attempt accepts typed placement evidence and returns a zero-token decision; consent declines without consuming another action's confirmation or accepts only the exact single-use, action-digest-bound confirmation token.

GET /api/session

{
  "selection": {"robot_preset": "franka", "sim_backend": "isaac", "scene_spec_uri": "stock://scene/default"},
  "sim_viz": {"run_id": "franka-demo", "stage": "demo", "camera": "workspace", "rerun_ready": true},
  "latest_submit": {},
  "camera_selection": ["workspace"],
  "chat_history": []
}