Testing Conventions

Use the repository virtualenv. Never use bare python; use npa/.venv/bin/python.

Correct command from repo root:

npa/.venv/bin/python -m pytest npa/tests/ --ignore=npa/tests/e2e --timeout=120 -q

Correct command from inside npa/:

cd npa
.venv/bin/python -m pytest tests/ --ignore=tests/e2e --timeout=120 -q

Run it in parallel. The suite is xdist-safe (pytest-xdist is in the dev extra); -n auto takes ~6 min down to ~2 min on 8 cores with an identical pass count, which is the difference between running the whole suite each iteration and only running a slice:

.venv/bin/python -m pytest tests/ -q -n auto

Use the serial form when a failure needs a readable, ordered traceback. CI still runs serially with coverage, so treat a parallel pass as the fast signal, not as a replacement for the gate.

Docs drift is a required gate and is slow to re-run blind. scripts/build_docs.sh memoizes and prefetches its npa --help walk (~1 min, was ~5), but it still costs a minute — so change CLI options first, then regenerate once, rather than checking after each edit. Per-subcommand options do not change the generated pages; a new top-level command adds docs/cli/<name>.md plus a README index line.

ruff is available in the venv:

npa/.venv/bin/python -m ruff check <files>

Existing Failures And Gates

• Known pre-existing failure: tests/smoke/test_cosmos_serverless_smoke.py has 5 tests gated by NPA_COSMOS_SERVERLESS_SMOKE=1; they fail with Unable to list Nebius VPC networks for project project-smoke: unsupported.
• E2E tests are gated by NPA_INTEGRATION_E2E=1 and excluded from standard runs with --ignore=npa/tests/e2e.
• Pipeline E2E tests use the e2e_pipeline pytest marker.
• Live Nebius Token Factory tests use the token_factory_e2e marker (in npa/tests/e2e/test_token_factory_e2e.py). They self-skip without a real NEBIUS_TOKEN_FACTORY_KEY; the marker is in conftest _LIVE_MARKERS so the key is not scrubbed. Run with NEBIUS_TOKEN_FACTORY_KEY=... pytest npa/tests/e2e/test_token_factory_e2e.py.

The gate for make test is 0 failures. Most recent measurement: 10836 passed, 37 skipped, 12 deselected, 1 xpassed, ~14 min serial (2026-08-18 at 1b89b3ba).

Treat that pass count as a floor, never as an equality. It is stale by construction — it rises whenever tests land, and it was previously recorded as a closed total that went wrong at the next merge. A higher number is normal; only a count that has fallen indicates tests stopped being collected, and the reliable comparison is against your own merge base. Two things move it legitimately: a few tests self-skip without node, tmux, or docker, and make test deselects the live/GPU markers so it collects a different tree from test.yml.

The suite is hermetic: it needs no kubectl, no cluster, and no venv at a particular path. A failure that names a missing binary or ModuleNotFoundError: No module named 'npa' is a bug in the test or the script it drives, not a missing prerequisite — three of those were fixed in 516396ec.

A hermeticity fix needs a test that fails without it in any environment. CI pip-installs the package, so a fix that only matters when the npa console script is absent from PATH cannot regress-fail there: prune PATH inside the test, or assert on the helper directly, rather than relying on the ambient environment.

Failures that appear only on an operator/dev VM are usually an ambient env var the conftest does not scrub, not a real regression. tests/conftest.py scrubs credential and infra-targeting variables for every non-live test precisely because CI runs with them unset and a working machine does not. When a test asserts on argv or rendered output, check that every env var the product reads to build it is in that scrub list: NPA_NEBIUS_PROFILE / NEBIUS_PROFILE were missing, and because product code prepends --profile <name> whenever either is set, ten mk8s argv assertions failed on any machine where an operator had selected a profile. Add the variable to the conftest tuple rather than working around it in the test; the tests that exercise the variable's own behavior set it with monkeypatch.setenv after the scrub runs.

Use evidence-based convergence: report numeric pass counts and exact failure messages, not subjective assessment.

Unit Test Rules

Tests must not hit real infrastructure. Mock SSH, S3, Nebius APIs, GPUs, and network calls at the call site. CLI tests use typer.testing.CliRunner against npa.cli.main:app.

When testing a later runtime failure, isolate successful prerequisite setup and assert that execution reached the intended failing boundary. For example, a Cosmos inference-denial test must mock the separately tested guardrail tokenizer preparation before its mocked inference subprocess. Otherwise a missing optional dependency or empty host cache masks the denial and privacy assertions. Preserve those assertions; never prime a real cache or download vendor data to make a hermetic test pass.

Live-Infra Testing Is A Priority (not optional)

Smoke + mocked-unit tests are necessary but not sufficient. Any change to an npa.workflow spec, a workbench tool / toolRef, or the agent/deploy (Terraform / provisioner) path MUST also land committed live-infra coverage — do not stop at smoke. Definition of done: the change is wired into a live path and you report numeric results from running it.

• New/changed npa.workflow spec: register it in SUBMIT_LIVE_MATRIX (npa/src/npa/orchestration/npa_workflow/submit_matrix.py) with the right tier (cpu | gpu | multi). If it has a dynamic gate/loop, also add it to DYNAMIC_SPECS in npa/tests/e2e/npa_workflow_live_helpers.py so the runner supplies --assume-decision; if it actually executes (non-plan-only), seed inputs in seed_live_workflow_inputs. Use plan_only=True only when stages are stubs or a real run would burn a GPU on a stub (the repo convention: "do not burn GPUs on stubs"). Keep test_submit_live_matrix.py green. Run it:

  NPA_E2E_NPA_WORKFLOW_SUBMIT_TIERS=<cpu|gpu|multi> \
    NPA_E2E_NPA_WORKFLOW_SUBMIT_SPECS=<spec>.yaml \
    ./scripts/npa-workflow-submit-live-e2e.sh
  # plan-only preflight (no job launch):
  NPA_E2E_NPA_WORKFLOW_SUBMIT_PLAN_ONLY=1 ./scripts/npa-workflow-submit-live-e2e.sh
  

• Live workflow render/infra:

  NPA_INTEGRATION_E2E=1 npa/.venv/bin/python -m pytest \
    npa/tests/e2e/test_npa_workflow_live_e2e.py \
    npa/tests/e2e/test_npa_workflow_live_infra.py -q
  

• Token Factory-backed stages: the token_factory_e2e live tests must pass with a real NEBIUS_TOKEN_FACTORY_KEY.
• Agent/deploy changes: validate a real npa agent destroy + deploy (and, for credential-path changes, that it is reproducible). Known trap: a stale ambient NEBIUS_IAM_TOKEN shadows the fresh var.iam_token in the Nebius Terraform provider (PermissionDenied/Unauthenticated even though the CLI works); provisioner._run scrubs it, but when reproducing by hand unset NEBIUS_IAM_TOKEN NPA_NEBIUS_IAM_TOKEN first.
• If a full live run is genuinely infeasible in the environment, say so explicitly and still commit the plan_only live-matrix entry — never silently ship smoke-only.