Agent Development (cheap-token chat backend)

How to develop the NPA agent chat backend. For operating a deployed agent (deploy/bootstrap/verify, chat UX, API grounding, Rerun) use skills/tools/npa-agent/SKILL.md; for fresh deploy/teardown loops use skills/workflows/agent-fresh-operate/SKILL.md.

Guiding principle: the cheapest token is the one you never spend. Protect and widen the zero-token grounded path; make the unavoidable model calls small, structured, and cheap-model-first.

Architecture (three layers)

1. Grounded intent router (zero tokens) — npa/src/npa/cli/agent_chat.py. match_chat_intent() classifies a turn; build_grounded_reply() answers from live session state with no model call ("grounded": true). Most operator turns end here.
2. Cost-tier routing (cheap model calls) — npa/src/npa/cli/agent_routing.py. Only turns that fall through the router reach Token Factory, and this layer keeps them cheap.
3. Token Factory client — npa/src/npa/clients/token_factory.py (operator/SDK path) and the embedded _provider_chat / _chat_with_resilience in npa/src/npa/cli/agent.py (agent-VM path).

The embedded-backend mechanism (critical)

The agent VM runs backend.py, which is built as one big f-string inside _bootstrap_agent_stack in npa/src/npa/cli/agent.py. Pure helper modules are inlined into it via placeholder substitution — the pattern to reuse when adding logic:

• Real module (normal Python, no brace escaping): agent_chat.py, agent_workflow.py, agent_routing.py.
• Embed helper _embedded_agent_<name>_source() strips the docstring + from __future__ line.
• Placeholder constant _AGENT_<NAME>_EMBED appears in the template and is replaced in the .replace(...) chain near the end of _bootstrap_agent_stack.

Rules:

• Put testable logic in a real module and embed it; do not write new logic directly inside the f-string unless it must touch template variables.
• Code written inside the f-string must escape literal braces as {{ / }} and newlines in strings as \\n; substitutions use single {var}.
• After editing the template, validate the rendered backend compiles (see Testing). A stray brace is a SyntaxError at import of agent.py.
• Embedded routes register before the template's own routes. The _AGENT_*_EMBED placeholders sit earlier in the f-string than most @app.* blocks, and Starlette resolves the first matching route, so an embedded module silently wins over a same-path handler written further down in agent.py. /artifacts/file/{filename} and /artifacts/download each carried two definitions this way, and the shadowed copies in agent.py were the weaker ones — no Content-Disposition/nosniff headers, no run-scoped inventory authorization — so the file a maintainer would open described a contract the deployment did not serve. Before adding a route, grep the embedded modules for its path; test_rendered_backend_registers_no_shadowed_routes fails the build on any method+path registered twice.

Cost-tier routing (agent_routing.py)

Pure, side-effect-free functions (no network) so they unit-test cheaply:

• classify_tier(text, intent, messages) → cheap (default) / standard (long/compound) / reasoning (analytical language) / vision (image content).
• build_model_ladder(tier, configured, interactive, requested_model, allow_tier_defaults) → cheapest-capable first; explicit user model wins; operator allowlist (NPA_AGENT_LLM_MODELS) respected when set.
• flavor_variants + filter_available — use only documented concrete model IDs and drop unavailable defaults. The current public replacements have no verified -fast variants; never invent one. Explicit selections are tried even when absent from the model list (dedicated endpoints can differ).
• chat_extra / thinking_enabled — disable hidden reasoning traces off the reasoning tier (don't pay for discarded tokens).
• enforce_input_budget — cap oversized pastes (head+tail preserved).
• usage_summary — surface per-turn token usage.

The /chat handler classifies the tier, enforces the input guardrail, honors an explicit model override, and returns tier + usage + input_budget_ok.

Adding a capability cheaply (decision order)

1. Grounded intent — can a regex intent + grounded state reply answer it? (0 tokens) Add to _INTENT_RULES / build_grounded_reply in agent_chat.py and cover it in test_agent_chat.py. Prefer this for anything high-frequency. Three invariants apply to every new or edited rule:
   • Add an INTENT_APIS entry. It is not only reply metadata: _semantic_route builds the semantic fallthrough's known_intents from its keys, so an intent missing from it is unreachable by any paraphrase the regex misses and its grounded replies report an empty apis_used. test_every_intent_declares_its_apis fails the build otherwise.
   • Earlier rules win, and match_chat_intent applies several hard-coded checks before the list at all. Adding a phrasing an earlier rule already claims silently changes nothing; check what currently matches first.
   • Keep sibling rules symmetric. The tool-capability rules are near copies, so a verb added to one belongs in all of them. "what can lancedb do" degraded to the generic component reply for exactly this reason while the identical Sonic/LeRobot/Genesis wording stayed tool-specific. A phrasing that matches nothing is not neutral — it falls through to a paid model call for an answer the grounded layer already had.
2. Cheap model — if it needs generation, let routing pick the cheap tier; only add reasoning/vision signals to classify_tier when the turn truly needs them.
3. Escalate deliberately — reserve MiniMaxAI/MiniMax-M3 for analytical/physical-AI/vision turns; it is overkill for routine chat.
4. Visual feedback — UI Describe this captures the active viewer frame and posts multimodal /api/chat with visual_context. Helpers live in agent_visual_feedback.py (embedded). Never ground these turns; use vision when an image is attached. See skills/atomic/agent-visual-feedback/SKILL.md.

Token Factory notes

• OpenAI-compatible: base https://api.tokenfactory.nebius.com/v1/, key NEBIUS_TOKEN_FACTORY_KEY. Same chat/completions shape everywhere.
• Public defaults follow the August 2026 migration notice: nvidia/Nemotron-3_5-Lightning for cheap/standard text, MiniMaxAI/MiniMax-M3 for reasoning/vision. No default -fast IDs are added. Vision never falls back to the text-only Lightning default.
• Cost-ordered default ladder lives in DEFAULT_LLM_MODELS (cheap first). A bare npa agent deploy seeds this whole ladder, so per-turn routing reaches every tier without --llm-models. Explicit --llm-models is a governance allowlist; /api/models still surfaces every model the key can serve for per-request selection.
• Reasoning-trace handling: split_reasoning() normalizes Cosmos3 inline <think> and Kimi/GLM reasoning fields.
• chat_extra(tier, model) selects thinking parameters per attempted model: Lightning uses chat_template_kwargs.enable_thinking=false; MiniMax uses chat_template_kwargs.thinking_mode="disabled". Analytical turns retain reasoning, and unknown custom models receive no guessed template options.

/api/models and /api/session share observed model availability. Prefer the configured default only when the provider lists it as chat-capable and it is within the explicit operator allowlist; otherwise choose the first eligible configured fallback. Keep the actual full provider catalog for explicit model selection. Never append an unavailable configured model to make a default look valid. The documented GET /v1/models?verbose=true supplies architecture modality; basic listings identify only known chat families, not arbitrary embedding or custom models. Empty successful discovery reports unavailable; failed discovery or unknown chat suitability reports unknown, with a null default and no provider diagnostics. Grounded/session operations still work. test_agent_model_availability.py exercises the rendered routes, refresh and failure cache, allowlist, explicit selection and non-chat exclusion. A catalog entry proves listing availability, not successful inference; retain live chat verification after deployment.

Testing tiers (keep tokens out of CI)

Follow skills/atomic/testing-conventions/SKILL.md; use npa/.venv/bin/python.

• Tier 0 — pure logic (0 tokens): npa/tests/cli/test_agent_routing.py (tiers, ladder, flavor, availability filter, budget, usage) and test_agent_chat.py (intent router, grounded replies). Highest-value coverage.
• Tier 1 — mocked LLM (0 tokens): patch _provider_chat / _chat_with_resilience; assert prompt assembly, resilience fallthrough, and tier/usage in the response.
• Rendered-backend check: confirm the embedded backend compiles with all wiring inlined — render setup_script with mocked SSH, extract the backend.py heredoc body, and ast.parse + compile it. Guards the f-string.
• Whole-surface audit (0 tokens): npa/scripts/audit_agent_capabilities.py goes one step further than the compile check — it runs the rendered backend against a sandbox state root and probes every parameterless GET plus every advertised intent. Use it to prove a change did not silently unregister a route or re-route an intent, and to answer "does the agent really support X" without a VM. Compiling is not running: an import-time failure in a shipped agent_backend module passes the compile check and fails here. --serve-live runs the same probes against a real uvicorn process on loopback with the deployed systemd unit's arguments, which is the only tier that covers lifespan and websocket-flag behavior.
• Tier 2 — live e2e (bounded tokens): gate behind NPA_AGENT_CHAT_LIVE=1 / NPA_INTEGRATION_E2E=1; pin the cheapest model; assert grounded: true where possible so most turns cost 0 tokens.
• Live public-model migration coverage: npa/tests/e2e/test_agent_token_factory_e2e.py runs the rendered backend under uvicorn on loopback, sends synthetic text/reasoning/vision requests through /chat, and checks real Token Factory answers and model selection. Only the deployment SSH transport is replaced while rendering; inference is real. It uses token_factory_e2e and skips without configured provider credentials.

For action-loop validation, inspect the final answer against the original goal as well as the successful tool observations. stopped_reason="done" can come from a deterministic empty-result reply; it does not prove the model produced a final answer or that every requested part was answered. An empty lookup is terminal only when its subject and scope satisfy the whole request. A filtered query with zero matches cannot establish an empty store, and an unrelated empty run lookup cannot replace an available status or tool-catalog answer. Preserve these distinctions in evaluation receipts and regression tests.

npa/.venv/bin/python -m pytest npa/tests/cli/test_agent_routing.py \
  npa/tests/cli/test_agent.py npa/tests/cli/test_agent_chat.py \
  npa/tests/smoke/test_agent_smoke.py npa/tests/smoke/test_agent_chat_smoke.py \
  npa/tests/guardrails/test_agent_secret_guard.py -q

Guardrails

• Never leak credentials/auth env/secrets into chat, logs, or workflow YAML.
• Do not hardcode project IDs, tenant IDs, bucket names, registry IDs, usernames, or public IPs in code or examples.
• Preserve the chat contract: grounded-first, then a cheap LLM fallback. Do not regress the agent into a chat-only (always-LLM) design.

Agentic surface (fallthrough beyond grounded)

Everything below runs only after the grounded intent router misses; the zero-token path stays the default. Design doc: docs/architecture/agent-competitive-plan.md.

• Bounded tool-calling loop — npa/src/npa/cli/agent_actions.py (run_action_loop): classify → plan → call → observe → decide → stop with a hard max_steps guard, an explicit TOOL_ALLOWLIST, and a confirmation-gate contract. GPU/destructive tools need a token bound to the action digest (action_digest); tokens are single-use. Route: POST /api/agent/act.
• Autonomous Sim2Real drive — npa/src/npa/cli/agent_sim2real_loop.py (drive_sim2real_loop): launch→status→gate→diagnose→adjust→re-run, mirroring the engine promote_checkpoint/loop_back gate. Every launch is confirmation-gated; stages complete only when live status confirms the run (no fabrication); stops on insufficient signal / no-adjustment to avoid runaway GPU. Route: POST /api/agent/sim2real/drive; chat intent drive_sim2real returns grounded guidance.
• Semantic fallthrough — npa/src/npa/cli/agent_semantic_router.py (classify_intent_semantic): keyword + cache (0 tokens) then one cheap structured call to map regex-missed paraphrases to a known intent/action. Wired into the /chat fallthrough; degrades to none on failure. Parity intents still match in match_chat_intent and never reach it.
• Quantitative viewer eval + memory — agent_visual_feedback.py (extract_quantitative_signals, compare_rollouts) and the shipped package npa/src/npa/agent_backend/memory.py (RunMemory, storage-injected, no hardcoded bucket). Routes: GET/POST /api/agent/memory/*.
• Task-eval harness — npa/tests/agent_eval/ (mocked, 0 tokens): scenarios
  • scorecard (success_rate/avg_steps/avg_tokens); live variant gated on NPA_AGENT_CHAT_LIVE=1.

Shipped vs embedded (Phase G): new logic still uses the embed mechanism by default; agent_backend/ is the shipped-package migration target (uploaded to /opt/npa-agent/agent_backend/, imported via sys.path). agent_memory is the migrated pilot; cli/agent_memory.py is a re-export shim. Rendered-backend compile check: npa/tests/cli/test_agent_backend_render.py.

Blueprint incorporation (retrieval, observability, adversarial eval)

Open-source-only parity with the Nebius "Blueprints" reference agent — Token Factory (LLM + embeddings) + AI Cloud only, no LangSmith/Pinecone/Tavily/ Snowglobe. Design doc: docs/architecture/blueprint-incorporation-plan.md.

• Retrieval / grounding (Phase H) — shipped npa/src/npa/agent_backend/retrieval.py (shim cli/agent_retrieval.py). index_corpus() chunks + embeds repo docs/+skills/ (and optional live web) into an injected vector store (InMemoryVectorStore / JsonVectorStore / build_lance_store for LanceDB); retrieve() returns typed Citations; format_grounded_answer() is extractive (0 generation tokens). embed and web_search (SearXNG-shaped, provider- agnostic) are injected. Read-only retrieval_search tool in TOOL_ALLOWLIST; routes POST /api/agent/retrieval/index, GET /api/agent/retrieval/search, GET /api/agent/retrieval/status; grounded-first /chat fallthrough that only fires when a corpus is indexed and the top match clears the score floor.
• Observability (Phase I) — shipped npa/src/npa/agent_backend/trace.py (shim cli/agent_trace.py). spans_from_action_loop / spans_from_drive wrap the existing step/iteration traces in structured spans emitted through an injected tracer (NullTracer default; build_langfuse_tracer / build_otel_tracer guarded). redact_attributes keeps secrets/PII out of spans. analyze_traces clusters traces + flags silent failures (truncation, empty tool results, unsurfaced errors, max-steps). Routes: GET /api/agent/trace/spans, POST /api/agent/trace/analyze.
• Adversarial eval (Phase J) — npa/tests/agent_eval/adversarial.py: persona (Token-Factory-generated, mocked in CI) × prompt-injection scenarios run against the real modules; validate_output always runs the repository-owned pure-Python checks. test_agent_adversarial_scorecard.py gates defense_rate delta-vs-baseline (adversarial_baseline.json).

Optional extras (injected/guarded, absent degrades gracefully): npa[agent-eval] is a compatibility alias with no extra dependencies. npa[agent-trace] (langfuse / opentelemetry-sdk). The LanceDB store is now part of the base install. Embeddings default to NPA_AGENT_EMBED_MODEL (confirm with npa workbench token-factory models); LanceDB URI, SearXNG URL, and tracer keys are env/config-resolved, never hardcoded. New agentic tests: test_agent_retrieval.py, test_agent_trace.py, and agent_eval/test_agent_adversarial_scorecard.py.

Evidence-backed improvements

The shipped agent_backend/improvements.py queue detects explicit tool failures, nonterminal empty results, truncation and exhausted action loops. Direct action, Sim2Real drive and semantic-action chat record observations linked to the active trajectory episode. Successful recovery and empty discovery that answers the whole request are not implementation defects. Premature empty-result termination requires separate goal-answer review even when the loop reports done. The queue produces triage work packages; it does not run shell commands, modify source, launch workers or publish code.

Enable it explicitly with NPA_AGENT_IMPROVEMENT_CONFIG, pointing to an owner-only JSON file outside the checkout. Configure directory, repository, evidence_directory, reviewers, optional private_literals, and scopes. Each scope requires a stable scope_id, known component, exact relative files, full base_revision, required_checks, supported lesson_keys, and optional version. Queue and evidence directories must be outside the source repository, mode 0700; configuration, database and reports are mode 0600. Dataset URI/prefix and the configured trajectory literal denylist also apply before queue persistence. Never put a live configuration or raw report in Git.

Use ImprovementStore.claim to obtain an exact work package and opaque claim_token. Its field name ensures trajectory redaction removes the credential. SQLite transactions prevent overlapping file ownership across processes; every update requires the current generation and lease. Release only after joining the worker. Scope comes from coordinator configuration, never an exception or model-supplied path. Changed scope configuration requires explicit reconciliation.

The coordinator captures begin_candidate before running the scope's real checks with npa/.venv/bin/python. Supply the complete candidate changed-file list, excluding separately owned concurrent work. Feed the actual subprocess.CompletedProcess and nonempty report bytes to write_validation_receipt, then record_validation. The adapter binds the report digest, check identity, exit code, source snapshot, scope and claim generation. It never executes commands supplied by a finding. Failed, partial, missing or changed evidence cannot advance to independent review.

An independently obtained review report enters through the local trusted write_review_receipt adapter and review. Reviewer identities are explicitly coordinator-attested external evidence; the agent's shared HTTP login does not establish multiple independent reviewers. HTTP routes under /agent/improvements accept existing protected receipt references, never a passed flag or an asserted reviewer name as proof. Workers cannot create receipts through HTTP. Retain the separate actor's actual report privately.

Verified lessons are fixed behavior keys with evidence references, not arbitrary prose inserted into prompts. Current keys are trajectory_observation_conservation and inspect_failed_tool_evidence. Relevant skill/action context retrieves them without a model call. Reuse checks the source snapshot and retained evidence, records consumption and action outcome, and deactivates after a distinct recurring failure. A storage failure returns pending feedback while preserving the completed product action; never repeat a successful GPU operation to repair feedback. Missing opt-in config reports disabled.

Regression checks include test_agent_improvements.py, test_improvement_episode_context.py, and test_agent_improvement_render.py. The rendered tests exercise all three action hooks and actual shipped modules. They do not replace a deployed zero-token lifecycle exercise with independent review evidence. The opt-in test_agent_improvements_live.py uses NPA_AGENT_IMPROVEMENT_LIVE=1, a dedicated approved component supplied through NPA_AGENT_IMPROVEMENT_LIVE_COMPONENT, and an owner-only lifecycle bundle at NPA_AGENT_IMPROVEMENT_LIVE_BUNDLE. The coordinator prepares that bundle from actual check receipts and a separately obtained review; the live test cannot invent either. Use the existing private live-agent credential configuration.

Source Layout

• CLI + bootstrap + embedded backend: npa/src/npa/cli/agent.py
• Grounded intent router (testable): npa/src/npa/cli/agent_chat.py
• Cost-tier routing (testable): npa/src/npa/cli/agent_routing.py
• Agentic tool loop / sim2real drive / semantic router: npa/src/npa/cli/agent_actions.py, agent_sim2real_loop.py, agent_semantic_router.py
• Shipped backend package: npa/src/npa/agent_backend/ (memory pilot)
• Token Factory client: npa/src/npa/clients/token_factory.py
• Routing tests: npa/tests/cli/test_agent_routing.py; agentic tests: test_agent_actions.py, test_agent_sim2real_loop.py, test_agent_semantic_router.py, test_agent_memory.py, test_agent_backend_render.py; eval harness: npa/tests/agent_eval/