Fleet (multi-cluster, multi-project Kubernetes)

When To Use

Use when a customer wants more than one Managed Kubernetes cluster stood up across one or many projects in a single Nebius tenant — e.g. per-team or per-tenant training clusters — and wants npa to drive it from one declarative file. npa fleet wraps the public nebius/nebius-solutions-library k8s-training recipe (the same recipe npa cluster up uses) once per cluster, and can create the target projects on demand.

For a single cluster, prefer npa cluster up. For Slurm-on-Kubernetes, use npa soperator.

Three-tier contract:

• CLI: npa fleet plan|deploy|destroy|status|verify-storage|verify-graphics --spec <fleet.yaml>.
• SDK: npa.sdk.fleet.deploy(spec) / destroy / plan / status with FleetSpec / ProjectSpec / ClusterSpec / NodePoolSpec.
• YAML / agent: apiVersion: npa.fleet/v0.0.1 spec; workflow toolRef: infra.fleet.deploy (config key fleet_spec).

Opt-in CPU RayCluster

Use kuberay: {enabled: true, worker_replicas: 2, worker_cpus: 2, worker_memory_gib: 4} in a cluster/default profile for a fixed CPU worker group. The shared backend validates strict types, explicit CPU placement and the exact reviewed recipe before provisioning. It also rejects head or worker pods larger than one declared CPU node and fixed requests above the pool's nominal aggregate CPU or memory; operators must still leave room for Kubernetes system pods. KubeRaySpec is exported by npa.sdk.fleet. A disabled block replaces enabled defaults atomically. Omitted policy remains off. Enabled deployments require the default local Terraform workspace without TF_CLI_ARGS* or TF_DATA_DIR overrides. Reapply rejects extra effective inputs and destination symlinks while preserving exact state/backup and provider caches. Retained backend metadata is unsupported; require managed implicit local state. The module manifest is rebuilt from the reviewed recipe before Terraform init. Historical opt-in provenance survives disabled/omitted policy reapply. Teardown checks the same effective-input restrictions and recorded materialized digest, then requires empty managed state before removing recovery files. Preserve state on refusal or incomplete teardown; missing provenance needs a reviewed reapply. See docs/fleet-kuberay.md and npa/examples/fleet/kuberay/cpu-raycluster.yaml for the one-entry fleet and native Ray Jobs journey. The separate CPU template pins Ray 2.58, disables autoscaling and bundled monitoring, uses non-root pods and installs namespace ingress isolation. GPU workers, RayService, arbitrary images and alternate recipe bytes are unsupported; do not recreate a Jobs submitter/controller or enable the old privileged vendor template. Preserve application outputs before owned teardown. Run npa/tests/e2e/test_fleet_kuberay_live.py with NPA_FLEET_KUBERAY_LIVE_CONFIG pointing at an owner-private JSON object with spec, kubeconfig, and evidence_dir. It inspects deployed values and executes real deterministic work on every Ray worker through native Jobs status/logs.

Verify an existing shared filesystem

Run npa fleet verify-storage --spec <private-fleet.yaml> --output json to qualify every CPU and GPU worker. The shared implementation is npa.fleet.storage_verification.verify_storage, also exported as npa.sdk.fleet.verify_storage. Use --only-projects, --only-clusters, --project-prefix, and --profile to preserve Fleet selection and identity semantics. Unknown selectors and missing or stale registered identity fail closed; explicitly disabled filesystems are skipped.

The verifier proves the exact read-write virtiofs source/path, reboot-safe nofail entry, capacity in binary GiB, unique host-file checksums, CSI health, and one RWX PVC shared across pods pinned to every exact worker. Every pod checks every worker's unique payload. It reads no pre-existing customer entries. Cleanup removes only owned probe paths and temporary resources with identity labels and UID preconditions, then proves absence on every node using server-synchronized Linux statx attributes after owned writers stop. Cached positive directory entries cannot substitute for fresh link-count evidence; unsupported synchronization and persistent linked entries fail closed. Partial evidence or cleanup failure cannot pass. Do not replace this with the vendored single-node shell smoke or infer storage health from node readiness.

Use --evidence-dir <owner-private-directory> outside the repository for exact receipts. Publication surfaces receive only sanitized counts, requested capacity, categories, hashes, and cleanup counts. This operation does not redeploy, resize, change IAM, or alter customer workloads. See docs/fleet-storage-verification.md for prerequisites and the installed SDK.

Run npa fleet verify-graphics --spec <private-fleet.yaml> --output json after an RTX driver, node, or runtime change. It uses the registered Fleet identity, waits for the declared health stability interval, and runs CUDA vectorAdd plus GLX, EGL, and Vulkan qualification on every selected 8-GPU RTX worker. All targets must declare gpu_workload_profile: rtx-rendering; a partial target or partial per-node result fails closed. Use --concurrency for independent clusters and the standard Fleet selectors to narrow scope. Exact provider and node evidence goes only to the owner-private --evidence-dir outside Git; CLI output remains safe to publish. The shared SDK entry point is npa.sdk.fleet.verify_graphics.

The all-worker live regression is npa/tests/e2e/test_fleet_storage_verification_live.py. Supply NPA_INTEGRATION_E2E=1, NPA_FLEET_STORAGE_VERIFY=1, an owner-private NPA_FLEET_STORAGE_VERIFY_SPEC, and NPA_FLEET_STORAGE_EVIDENCE_DIR. The daily runner reaches it only with that explicit opt-in. Its expected target, worker, and binary-GiB totals derive from the complete selected declaration.

RTX PRO 6000 hardware MIG is an additive cluster policy. Use mig: {enabled: true, strategy: mixed, config: all-balanced} only with two strict reserved-capacity gpu-rtx6000 / 1gpu-24vcpu-218gb workers and 128 GiB boot disks. NPA pins and live-verifies GPU Operator v26.3.3, driver 580.173.02, device plugin/GFD v0.19.3, MIG Manager v0.14.2, exact per-node resources, and zero whole-GPU capacity/allocatable. See docs/fleet-rtx-pro-6000-mig.md and npa/examples/fleet/rtxpro-mig.yaml.

RTX rendering without MIG uses the separate explicit cluster setting gpu_workload_profile: rtx-rendering. It selects RTX PRO 6000 (including an explicitly resolved zonal platform such as gpu-rtx6000-a), the supported single-GPU preset by default or accepts the explicit 8gpu-192vcpu-1744gb RTX PCIe preset, uses GPU Operator mounted drivers, and requires per-node GLX/EGL/Vulkan readiness. The 8-GPU RTX shape remains non-fabric and does not opt into the NVSwitch unsafe-operator path. The profile does not alter the empty/default managed-image contract and cannot be generalized to NVSwitch targets.

The rendering profile supplies the exact platform and preset to the marketplace GPU Operator's nebius.nvidiaDriverCRDPatch.profiles values. Its default gpu-rtx6000 selector does not match zonal gpu-rtx6000-a workers. The override contains only the requested RTX pool, keeps RDMA disabled, and uses a revision hash so Terraform reconciles write-only values on existing releases. Alternate recipes must expose and wire gpu_operator_rtx_driver_profile; incompatible recipes fail before cloud mutation. Do not relabel workers to hide a selector mismatch.

An RTX cluster may declare gpu_driver_package_repositories as a mapping of safe filenames to package-source text. Fleet passes it through the Operator's driver.repoConfig and an owned ConfigMap, retaining signature verification. Match the sources to the actual driver-image distribution. The five-cluster example supplies Ubuntu 24.04 HTTPS sources for networks that permit HTTPS but block HTTP. Empty configuration preserves the image defaults. Before workloads, verify the driver pod actually mounts the configured files and can fetch its kernel packages; do not infer network reachability from the operator VM.

The marketplace may disable automatic upgrades and use an OnDelete driver DaemonSet. A successful Helm update then changes the template without replacing existing pods. Compare actual pod mounts with the template. Replacing stale driver pods is maintenance: first verify no application GPU workloads are active, cordon each affected node, recheck workloads, and delete only the identified stale pod with a UID precondition. Wait for its ready replacement and restore the node's prior scheduling state. Retain private receipts and rerun the full CUDA/graphics health gate; never delete application pods or relax readiness to finish an upgrade.

Read-only live coverage checks exact selectors, per-node GPU quantities, ConfigMap contents, ready driver containers, and actual repository mounts:

NPA_INTEGRATION_E2E=1 NPA_FLEET_RTX_VERIFY_SPEC=<private-spec-path> \
  NPA_FLEET_RTX_KUBECONFIGS=<private-project-cluster-path-mapping.json> \
  npa/.venv/bin/python -m pytest \
  npa/tests/e2e/test_fleet_rtx_driver_config_live.py -q

The mapping is {project_key: {cluster_name: kubeconfig_path}}; keep its exact values outside Git. This check creates no pods and complements the real CUDA/graphics and representative workload validation.

For qualification on every individual eight-GPU RTX cluster, additionally run npa/tests/e2e/test_fleet_rtx_workload_live.py with the same private spec/mapping, NPA_FLEET_RTX_RUN_WORKLOADS=1, and an owner-only NPA_FLEET_RTX_WORKLOAD_EVIDENCE_DIR. It creates one Job per target using the provisioning gate's immutable, payload-clean image. Each Job executes verified CUDA arithmetic and matrix multiplication, dynamically loads GLX/EGL, and creates a Vulkan instance with an NVIDIA physical device. It verifies the runtime digest and deletes each Job with a UID precondition, then proves both Job and pod absence. The evidence directory holds private execution receipts and a separate sanitized result summary. This does not replace representative Isaac training or the canonical Sim2Real ladder.

The RTX override also sets toolkit RUNTIME_CONFIG_SOURCE=file. With newer containerd binaries, containerd config dump can migrate the configuration in memory. Using that output may produce a version-4 NVIDIA drop-in beside a version-2 root, preventing containerd startup. Reading the file preserves the host schema; see NVIDIA's configuration-source guidance. For an already broken node, apply the corrected toolkit configuration first. After verifying no application workloads are active, retain the root and drop-in files privately, move only the proven incompatible NVIDIA drop-in out of the imports directory, validate containerd config dump, and restart containerd. Verify the replacement toolkit produces a compatible fragment and the full health gate passes. Do not change version numbers blindly or discard unrelated host configuration.

A node group can report PROVISIONING again when its workers lose readiness. Fleet treats it as zero incremental reserved demand only after a complete Compute inventory proves the exact group/cluster/project labels, requested template, running worker count, reservation bindings, and attached disks. Target and actual node counts must also match. This capacity proof does not mark the cluster healthy: every deployment readiness gate still runs.

Spec (npa.fleet/v0.0.1)

The version remains additive. A cluster without backend is the historical mk8s shape. Explicit targets use backend: mk8s plus mk8s: {...} or backend: soperator plus soperator: {...}; both may appear in one project or fleet. Fleet owns target identity and shared-network inventory, while the selected backend owns one target's plan, materialization, apply, native status, verification, and destroy. See docs/cluster-backends.md for one-entry and mixed examples and the fail-closed state rules.

A defaults cluster profile is deep-merged under every cluster, so identical fleets are just project entries with no overrides. Projects may declare custom clusters (overrides and/or several clusters), and identical + custom may be freely mixed. Projects reference an existing project_id or a name that is created on demand as project_prefix + name.

Project-scoped object_storage is independent of a cluster's shared filesystem. storage_class: enhanced maps to Nebius Enhanced Throughput, size_gibibytes becomes the bucket's exact binary capacity cap, and deploy requires provider read-back plus a write/read/delete probe. Leave bucket_name empty in public specs so Fleet derives a stable private name at runtime. Fleet retains buckets and durable artifacts on cluster destroy; explicit npa storage bucket delete remains the destructive cleanup boundary. Fleet uses its own project-scoped storage service account and access key, separate from LeRobot bootstrap identities, and never enables the editors fallback. Storage reconciliation currently requires at least one selected mk8s target in each project that declares storage; soperator-only selections fail before mutation.

Before any project, bucket, subnet, or cluster mutation, deploy budgets all new bucket declarations against tenant storage.bucket.count and the matching storage.bucket.size.<class> allowance (enhanced-throughput for Enhanced). The size allowance uses bytes, separately from filesystem quotas. Missing, unreadable, or insufficient evidence fails closed. Every selected existing bucket must match the exact project, name, region, active/unsuspended state, storage class, and cap before reuse. Verified existing buckets are not charged again as new quota demand; a new storage declaration is still checked when its cluster is unchanged. Read/write success without verified probe deletion cannot mark Fleet storage ready.

A bucket cap is a maximum, not reserved storage. Available tenant capacity can change after preflight, including when other buckets grow. See the official bucket contract and storage quotas.

Run the read-only reservation and storage quota regression against a private planning spec even before its target projects exist:

NPA_INTEGRATION_E2E=1 NPA_FLEET_QUOTA_VERIFY_SPEC=<private-spec-path> \
  npa/.venv/bin/python -m pytest \
  npa/tests/e2e/test_fleet_storage_quota_live_e2e.py -q

This proves capacity evidence only; project identity, actual bucket provisioning, GPU readiness, and representative workloads remain separate deployment gates.

apiVersion: npa.fleet/v0.0.1
name: fleet1-test
tenant_id: ""                # resolved from ~/.nebius + ~/.npa when empty
region: us-central1
profile: ""                  # ~/.nebius profile to authenticate as; "" = active
project_prefix: "fleet1-test-"
defaults:
  # Replace the next two driver lines with
  # gpu_workload_profile: rtx-rendering for RTX/Isaac rendering clusters.
  gpu_driver_mode: auto
  managed_driver_preset: cuda13.0
  allow_unsafe_nvswitch_operator: false
  gpu_health_stabilization_seconds: 120
  gpu_health_timeout_minutes: 60
  gpu_cuda_smoke: true
  cpu_nodes: { count: 1, platform: cpu-d3, preset: 48vcpu-192gb }
  gpu_nodes:
    count: 1
    platform: gpu-rtx6000
    preset: 1gpu-24vcpu-218gb
    # Optional runtime-only ID; renders STRICT and never falls back to PAYG.
    capacity_block_group: ""
  enable_filestore: true
  filestore_disk_size_gibibytes: 1024
  filestore_mount_path: /mnt/data
  filestore_mount_tag: npa-shared-fs
  # Runtime/operator-supplied standards-based chart source. Keep private
  # registry endpoints out of committed specs and PR evidence.
  filesystem_csi_chart_repository: ""
projects:
  - name: a                  # -> project fleet1-test-a (identical profile)
    object_storage:          # separate from enable_filestore
      enabled: true
      storage_class: enhanced
      size_gibibytes: 1024
  - name: b                  # -> project fleet1-test-b (identical profile)
  # - name: c                # custom: overrides + a second cluster
  #   clusters:
  #     - name: train
  #       gpu_nodes: { count: 2, platform: gpu-h200-sxm, preset: 8gpu-128vcpu-1600gb }
  #       enable_gpu_cluster: true
  #       infiniband_fabric: us-central1-a
  #     - name: infer         # inherits defaults
  # - project_id: project-existing123   # deploy into an existing project by id
  #   clusters: [ {} ]

Example spec: npa/examples/fleet/fleet1-test.yaml.

Targeting another tenant (profile)

A Nebius service account belongs to exactly one tenant, so deploying into a second tenant means authenticating as that tenant's principal. Set the spec's profile: (or pass --profile <name>, which wins) to name a ~/.nebius profile; every nebius CLI call, the minted terraform TF_VAR_iam_token, and the generated kubeconfig's exec-credential args are pinned to it. The machine's active profile is never mutated, so concurrent fleets in different tenants stay independent.

With a profile set, tenant_id resolves from that profile's tenant-id (never the active profile's) and a profile with no tenant-id is a hard error instead of a silent deploy into the wrong tenant. destroy falls back to the profile recorded in each cluster's env sidecar at deploy time, so a teardown always authenticates as the principal that created the cluster.

Register a service-account profile non-interactively:

nebius profile create <name> --endpoint api.nebius.cloud \
  --service-account-id <sa-id> --public-key-id <public-key-id> \
  --private-key-file-path ~/.nebius/<name>.pem \
  --parent-id <sa-parent-project> --tenant-id <tenant> --skip-auth
nebius --profile <name> iam get-access-token >/dev/null   # verify

nebius profile create also activates the new profile; re-activate the previous one (nebius profile activate <prev>) if other tooling on the host depends on it.

Procedure

Parallel applies use a shared download cache only during initialization. NPA holds the cache lock through initialization and copying each target's selected providers into its own Terraform data directory. Later initialization can then rewrite cached downloads without changing binaries used by another apply or destroy. Dependency-lock checksum verification remains enabled.

Older recipe sources can omit newer regions from their default-node table. Materialization preserves known-region defaults while allowing explicit platforms and presets in other regions. It supplies no guessed GPU or fabric; resolve those values from the provider inventory before deployment.

1. Keep committed files public-safe: never hardcode tenant/project/registry IDs or SSH keys. The spec resolves tenant/region from ~/.nebius/config.yaml and ~/.npa/config.yaml when its fields are empty; the SSH public key comes from ssh_public_key or ~/.ssh/id_ed25519.pub / id_rsa.pub.

2. Plan first (no infra): npa fleet plan --spec fleet.yaml shows the projects (create vs existing) and per-cluster node config.

3. enable_gpu_cluster is auto: GPU clustering (InfiniBand fabric) is only valid on fabric-capable 8-GPU SXM presets. Single-GPU presets (e.g. RTX PRO 6000 1gpu-24vcpu-218gb) auto-set enable_gpu_cluster=false; set it true only with an 8-GPU preset and infiniband_fabric. Every GPU pool defaults to gpu_driver_mode: auto, which selects Nebius's managed driver image plus the provider device plugin; CPU-only clusters emit no GPU-driver input. managed_driver_preset defaults to the vendored recipe's supported cuda13.0 and is configurable. operator remains an explicit escape hatch, but NVSwitch topologies reject it unless allow_unsafe_nvswitch_operator: true acknowledges the Network Operator/MOFED versus Fabric Manager host-device race. The same strategy resolver applies to npa cluster up and Fleet.

4. Bind reserved GPU capacity explicitly when required. Set gpu_nodes.capacity_block_group to a runtime-supplied Capacity Block Group ID. Fleet renders gpu_nodes_reservation_policy = { policy = "STRICT", ... }, so an unavailable or incompatible block fails instead of falling back to ordinary on-demand capacity. Never commit a live capacity block ID.

   Discover and verify reservations read-only with:

   nebius --profile <p> capacity capacity-block-group list \
     --parent-id <tenant> --all --format json
   nebius --profile <p> capacity capacity-interval list \
     --parent-id <capacity-block-group> --all --format json
   nebius --profile <p> capacity capacity-block-group list-resources \
     --id <capacity-block-group> --format json
   nebius --profile <p> capacity resource-advice list \
     --parent-id <tenant> --all --format json
   

   Preflight requires the named block to be active, in the target tenant and region, and matched to the GPU platform and InfiniBand fabric. It checks the aggregate GPU requirement against remaining reserved capacity. Only after that validation does it exclude those GPUs from ordinary GPU quota; all node, boot-disk, GPU-cluster, Kubernetes, and storage quotas still apply.

5. Preflight quotas at the tenant, before anything else. Each cluster needs, in the target region: compute.instance.count (worker nodes only; the managed control plane is service-owned), compute.instance.non-gpu.vcpu for the CPU preset, compute.instance.gpu.<family> for the GPU preset (on-demand GPU quota is frequently 0), compute.disk.count/compute.disk.size.network-ssd, compute.gpucluster.count when enable_gpu_cluster, and compute.filesystem.count + compute.filesystem.size.network-ssd when enable_filestore. Each private worker consumes two vpc.allocation.count slots (its private address and pod alias range); the managed control-plane endpoint is service-owned. A create-on-demand project additionally needs one vpc.network.count, one vpc.subnet.count, two vpc.pool.count, one vpc.routetable.count, and one vpc.route.count. List them all at once with nebius --profile <p> quotas quota-allowance list --parent-id <tenant> --all --format json (each item carries metadata.name, spec.region, spec.limit).

   deploy does this automatically (--preflight, on by default) and refuses to apply when a capacity block or tenant limit cannot cover the in-scope clusters; --no-preflight attempts it anyway. Before calculating creation-only VPC requirements, preflight lists projects once and reuses an existing immutable project ID when a name already exists. An unreadable project inventory fails closed rather than assuming the project exists.

   Both explicit project_id targets and projects resolved by name require a provider read verifying exact ID, tenant, region, active state, and absence of suspension or deletion. An explicit ID may use a local Fleet role whose name differs from the provider project name. With --no-create-projects, a missing target aborts the entire preflight before any project's storage or workers are created; never substitute a stale or deleting project from local state.

   Run the read-only live identity regression against an owner-private spec with explicit tenant, region, profile, and project IDs:

   NPA_INTEGRATION_E2E=1 NPA_FLEET_PROJECT_VERIFY_SPEC=<private-spec-path> \
     npa/.venv/bin/python -m pytest \
     npa/tests/e2e/test_fleet_project_identity_live_e2e.py -q