Cosmos3-Super B200/H200 Benchmark

Use this skill for the production benchmark in workflows/testing/cosmos3-super-b200-benchmark.yaml or workflows/testing/cosmos3-super-h200-benchmark.yaml. It is different from Cosmos Framework native Ray Serve (cosmos3-ray-serve): this workload runs the public vLLM-Omni synchronous video endpoint. The default primary suite measures four independent-service arrangements on one complete B200 or H200 node. The b200-full suite reproduces the public machine-readable ten-cell B200 record and its 240 measured attempts.

The separate h200-single-gpu suite is a functional/performance validation of one TP-1 service on one H200. It fixes one warmup and 24 sequential measured requests with the same model, image, prompt, seed, workload, timeout, and MP4 gates. Report its rates per GPU-hour and service-hour. Never label it eight-GPU node throughput or compare it as the paper's 8x1 cell, which uses eight independent one-GPU services on a complete node.

Fixed contract

• Image: vllm/vllm-omni:cosmos3 at the digest in the workflow.
• Model: nvidia/Cosmos3-Super at revision e0262be9d8f7586bc24c069a2aed2b665bdff266.
• Arrangements, in order: one 8-GPU hybrid service; two TP-4 services; four TP-2 services; eight TP-1 services. Every arrangement occupies all 8 GPUs.
• Workload: BF16 text-to-video, 1280x720, 189 frames, 24 fps, 35 steps, guidance 6.0, flow shift 10.0, max sequence length 4096, pinned model anchor and negative prompts, seed cycle 17/23/41, guardrails disabled, synchronous timeout 5400 seconds.
• One technically valid warmup per service is excluded. Every measured cell is 24 attempts. The primary cells have exactly one request in flight per service.
• b200-full adds a concurrency-two cell for every topology and delayed repeats of the 1x8 concurrency-one and concurrency-two cells. Request concurrency is per service and is independent of replica count.
• h200-single-gpu fixes topology 1x1, H200, 24 measured attempts, one request in flight, and reports an explicit non-paper validation scope.

Do not turn the reference frontier in the documentation into expected output or a pass threshold. A new report is live evidence only when its per-attempt records and window were emitted by the real command.

Before submit

Load and follow skills/atomic/health-preflight/SKILL.md, skills/atomic/third-party-eula-preflight/SKILL.md, skills/atomic/solution-licensing/SKILL.md, and skills/atomic/protect-nebius-infra-details/SKILL.md.

Run the exact access gates before provisioning:

npa/.venv/bin/npa workbench health preflight --checks hf,s3 --json
npa/.venv/bin/npa workbench health access --capability cosmos3-serving --json
npa/.venv/bin/npa workbench workflow validate-spec \
  workflows/testing/cosmos3-super-<gpu>-benchmark.yaml

The operator must independently review the runtime terms and pass NPA_COSMOS3_ACCEPT_NVIDIA_SOFTWARE_LICENSE=YES at submit time. Never commit, persist, or bake acceptance, credentials, model weights, prompt text, generated clips, or runtime caches.

Run

Submit the shipped spec through npa workbench workflow submit, selecting the operator's exact Kubernetes context and bucket. Pass the acceptance value, HF_TOKEN, and S3 credentials through --secret-env; never render their values into YAML or logs. Use the recipe for the intended hardware. The resource profile must remain B200:8 or H200:8, must keep the 32-GiB /dev/shm, and must retain the exact external image digest. The H200 path sets PyTorch expandable segments for the lower-memory one-GPU service cell; B200 behavior is unchanged.

The command starts services sequentially and refuses to open a cell's measured window if any service warmup fails technical validation. It tears down one cell before starting the next. Do not run two cells concurrently on the same node.

The shipped workflow defaults to suite: primary. Select the complete B200 record with --var suite=b200-full; that suite fails closed unless the GPU family is B200, all four topologies are selected, and every cell has exactly 24 attempts. After each cell, its records and validated clips are uploaded before an immutable completion marker is created and read back. A resumed run reuses only a marker whose complete cell record matches the current model/image/workload/prompt/run contract; a partial cell is rerun and a conflicting completed cell fails closed.

Evidence and interpretation

The durable root contains benchmark.json, per-cell attempts.json, window.json, derived.json, cell.json, complete.json, and validated production MP4s. A valid attempt requires HTTP 200, non-empty bytes, full first-to-last decode, exact geometry, frame count and rate, and passing blank/frozen checks. Failures remain in the window and receive zero valid-video-second credit.

Use only the shared first-dispatch-to-final-completion window for node throughput. It includes routing, generation, encoding, skew, failure time, and tail idle time; it excludes startup, model load, and warmup. The final full-suite report derives primary-frontier, concurrency-two, and delayed-repeat comparisons from the live cell records. Report throughput as technically valid video-seconds per node-hour, not accepted or useful output.

Teardown

Cancel the exact workflow run before removing any cluster or controller. Follow skills/atomic/teardown-and-cost/SKILL.md; shared controllers and operator projects are not run-owned resources. Preserve exact infrastructure identifiers only in access-controlled evidence.

Verify

npa/.venv/bin/python -m pytest \
  npa/tests/workbench/test_cosmos3_super_benchmark.py \
  npa/tests/workflows/test_cosmos3_super_b200_benchmark_workflow.py \
  npa/tests/workflows/test_cosmos3_super_h200_benchmark_workflow.py \
  npa/tests/cli/test_cosmos3_cli.py -q