Token Factory (zero-GPU hosted inference)

Nebius Token Factory is an OpenAI-compatible hosted-inference API for open text and vision models. It is the cheapest tier in the workbench that produces a real artifact: no cluster, no GPU, no provisioning. Reach for it before standing up anything, both for real work and to prove a toolchain end to end.

Full reference: docs/workbench/token-factory.md. Key setup only: docs/workbench/token-factory-key.md.

The credential is not a Nebius IAM token

This is the single most common failure. A Token Factory key is a separate credential minted in the separate Token Factory console (https://tokenfactory.nebius.com/). It is a long opaque token starting with v1., read from NEBIUS_TOKEN_FACTORY_KEY or ~/.npa/credentials.yaml. Your nebius CLI IAM token returns 403 here — having Nebius CLI access is not enough, and no amount of re-authenticating the CLI will help.

Keys are shown once at creation. A project with no balance returns 402/403 on inference even with a valid key.

npa workbench token-factory status    # connection settings, no network call
npa workbench token-factory verify    # live models call; non-zero on auth failure
npa workbench token-factory models    # what this key can actually reach

Run verify before a batch job and models before pinning a model name — model availability is per-key, so a model in the docs may not be in your project.

Defaults: base URL https://api.tokenfactory.nebius.com/v1/, overridable with NEBIUS_TOKEN_FACTORY_BASE_URL. Requests retry on 429 and 5xx.

August 2026 migration

The official notice retires the old public text, vision, and reasoning defaults. The replacements are Nemotron-3.5-Lightning for text and MiniMax-M3 for vision/reasoning. See docs/workbench/token-factory-deprecation-verification.md for exact IDs, live observations, API differences, vendor terms, and verification commands. MiniMax-M3 is under the MiniMax Community License; hosted API access does not establish the operator's commercial entitlement. No vendor weights are bundled.

Direct-output client calls disable thinking with model-specific template keys: Lightning enable_thinking=false, MiniMax thinking_mode=disabled. Explicit client extra values win. Agent reasoning turns enable thinking deliberately.

Commands

Current pricing and usage

When the task requires a price check, fetch the official public catalog at https://tokenfactory.nebius.com/api/public/models_info. The field meanings are documented at https://tokenfactory.nebius.com/model-catalog.md; the site also exposes https://tokenfactory.nebius.com/llms.txt for discovery. An empty rendered pricing page is not evidence that pricing is unavailable.

Match pricing to the exact catalog model/flavor. Record the retrieval date and applicable input/output rates, including the serving mode. A public-price lookup does not require credentials. When inference is requested, also verify that the selected model is in the key-scoped model list. Public catalog presence does not prove account access or account-specific billing. If required pricing remains unresolved, stop before paid inference.

Report inference usage and finish reason only when the response exposes them. Keep the coding agent's token usage separate. Requested output tokens are a limit, not measured usage; missing provider usage means cost is unmeasured.

Inference commands

Every command takes local paths or s3:// URIs for both input and output, and supports --dry-run (compute without writing the artifact) and --output text|json.

Caption images — default model MiniMaxAI/MiniMax-M3:

npa workbench token-factory caption \
  --input-path s3://<bucket>/frames/ \
  --output-path s3://<bucket>/captions.json \
  --max-images 50 --max-tokens 512 --temperature 0.2 \
  --instruction "Describe the scene, objects, and any action."

Batch text generation over a JSONL/text prompt file — default model nvidia/Nemotron-3_5-Lightning:

npa workbench token-factory generate \
  --input-path prompts.jsonl \
  --output-path s3://<bucket>/generations.jsonl \
  --max-prompts 0 --max-tokens 512 --temperature 0.7 \
  --system-prompt "<applied to every request>"

--max-prompts 0 means all of them. Set a small non-zero value first: this is the command that turns a typo into a large token bill.

Prompt parsing depends on the file extension. Keep plain prompt lines in a .txt file. Each nonempty .jsonl line must be a JSON string or an object with a prompt, text, or instruction field, not bare text. Use a JSON serializer when converting prompts, including prompts written by a workflow shell step. For a local, no-inference check, the installed npa.workbench.token_factory._load_prompts(Path(...)) reader returns (id, prompt) pairs. Compare the prompt values and count with the requested input. This is an internal Python helper, not a CLI validation command; confirm it exists in the installed version before using it. It does not verify remote staging or model access.

Batch text generation — same prompt file, same generations.jsonl, batch token rates, default model openai/gpt-oss-120b:

npa workbench token-factory batch-generate \
  --input-path prompts.jsonl \
  --output-path s3://<bucket>/generations.jsonl \
  --model openai/gpt-oss-120b --completion-window 24h

# or submit now, collect later
npa workbench token-factory batch-generate ... --no-wait
npa workbench token-factory batch-status --operation-id <id> --output-path <same> --wait

Reach for batch-generate over generate whenever nothing is waiting on the answer, which is most bulk stages. Three properties are unique to it, and each one has already cost real debugging time:

• Batch routing is a per-model entitlement, unrelated to real-time chat. Most models that serve generate are rejected for batch. Historical measurements across eight text models on one key, exactly one — openai/gpt-oss-120b — was batch routable; meta-llama/Llama-3.3-70B-Instruct, Qwen/Qwen3-32B, Qwen/Qwen3-30B-A3B-Instruct-2507, Qwen/Qwen3-235B-A22B-Instruct-2507, google/gemma-3-27b-it, deepseek-ai/DeepSeek-V4-Flash, nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B, and zai-org/GLM-5.1 were not. That is why DEFAULT_BATCH_MODEL is openai/gpt-oss-120b and not DEFAULT_TEXT_MODEL. Treat the routable set as per-key and verify on a couple of prompts before pointing a large run at a new model.
• Batch is text-to-text only. A vision model is rejected at submit with Batch inference is only supported for text2text models, so there is no batch captioning path; use caption, which is real-time.
• The completion window is a deadline, not a latency. Observed live: batches of one and three prompts sat in_progress with completed: 0 for over an hour against a 24h window. Do not read a slow batch as a hung one, and never put --wait on a path that has its own timeout.

Where the failure reason actually lives matters. GET /operations/{id}/errors returns a single empty string for a failed batch — useless. The real per-row reason is in the batch record's error file (GET /batches/{id} → error_file_id → GET /files/{id}/content, which redirects, so redirects must be followed). batch-generate reads that file and reports it, and also surfaces request_counts (total, completed, failed, invalid) as the only genuine progress signal a pending batch offers.

Distinguish a degraded platform from your own bug. A batch that is accepted, reports in_progress with rows validated (total: 2, invalid: 0), and holds completed: 0 is usually not your job's fault. Batch execution has been observed unavailable while submissions were still accepted through the datasets/operations route. The cheapest tell is POST /v1/batches, the OpenAI-compatible submit, returning 403 Creating new batch job is temporarily unavailable. Confirm it is a server-side switch rather than your request by checking where the 403 lands: an empty body returns 422 naming the missing fields, but a valid payload with a genuinely uploaded input_file_id still returns 403, so the gate sits ahead of resource validation. Meanwhile the rest of the key stays healthy — real-time chat on the same model, POST /v1/files with purpose=batch, GET /v1/batches, and dataset create/delete all succeed — which rules out the key, the balance, the model, and the payload. When you see this, stop debugging your spec, cancel what you queued (POST /batches/{id}/cancel), and use generate until batch recovers. Do not wait it out: the same 403 was still being returned eight days after it was first seen, so "temporarily" can outlast any plausible stage timeout. Plan the run on generate and re-probe later rather than leaving a stage parked.

That 403 is not the quota rejection, and conflating the two sends you down the wrong path. The documented limits are 10 active batches per customer and 100 submissions per hour, a batch counts as active only until its processing finishes, and rate limiting surfaces as 429. So before blaming a limit, list your batches (GET /v1/batches?limit=100 — the default page is 10, which makes a long history look artificially short) and count the non-terminal ones. All terminal plus a 403 with no x-ratelimit-* headers means availability, not quota.

Physical-AI reasoning over a scene — default model MiniMaxAI/MiniMax-M3. Point it at scene images and ask what a robot should do:

npa workbench token-factory reason \
  --input-path s3://<bucket>/scene/ \
  --output-path s3://<bucket>/plan.json \
  --task "Describe this scene and give a step-by-step plan of action." \
  --max-images 8 --max-tokens 1024 --temperature 0.2

In workflows

These run as CPU-only npa.workflow steps with no accelerator request. The renderer injects NEBIUS_TOKEN_FACTORY_KEY for workbench.token_factory.* steps, so pass it as a secret at submit time and never in the YAML:

npa workbench workflow submit <spec.yaml> --secret-env NEBIUS_TOKEN_FACTORY_KEY

toolRefs: workbench.token_factory.caption, .generate, .batch_generate, .reason, .triage (digest a run's textual artifacts into a triage report).

npa workbench token-factory workflow prints exactly four: token-factory-caption.yaml, token-factory-generate.yaml, token-factory-cosmos-reason.yaml, and vlm-eval-token-factory.yaml. Several more are checked in but not listed by that command, so do not treat its output as the full inventory:

• token-factory-batch-generate.yaml — the batch-inference twin of token-factory-generate.yaml.
• token-factory-parallel-fanout.yaml — parallel batches.
• token-factory-gate-loop.yaml, tokenfactory-cosmos-gate.yaml — a hosted model as a gate that decides whether the pipeline continues.
• tokenfactory-rollout-judge.yaml, tokenfactory-scene-to-rollout-judge.yaml — reason about a scene, then judge a rollout against that plan.
• tokenfactory-train-triage.yaml — triage a training run's artifacts.

All live under workflows/testing/.

Choosing between Token Factory and VLM eval

They overlap and are easy to confuse. token-factory reason produces an analysis or plan. vlm-eval scores a rollout against a task and emits a pass/fail gate with a threshold. When you want a judged number for a gate, use skills/tools/vlm-eval/SKILL.md — and note it can consume a Token Factory reasoning artifact directly through --task-from, so a judge scores against a plan an earlier stage wrote rather than a hardcoded string.

Gotchas

• Canonical Sim2Real is scoring, not planning. Stage 8 uses MiniMaxAI/MiniMax-M3 as its only Stage 8 evaluator, on CPU with no self-hosted evaluator image. It sends a bounded, deterministic rollout-wide frame sample and requires event-local structured scores. Stage 9 compares the single evaluator result with the authoritative Stage 7 rollout set and rejects missing, duplicate, or extra evaluations before PPO. Preserve request IDs, token usage, latency, retries, and an authoritative returned cost or explicit null separately from model-agent tokens.
• Sim2Real preflight is stronger than model listing. Its submit and prepared action paths declare NEBIUS_TOKEN_FACTORY_KEY by name only, then require both key-scoped model availability and a minimal inference before provisioning.
• Model availability is per-key. Confirm with models before pinning a name in a spec; a spec that names an unavailable model fails at run time, not at validation.
• --max-images and --max-prompts are cost controls, not correctness knobs. Defaults are 50 images and unlimited prompts. Always bound the first run.
• --dry-run still calls the model. It skips writing the artifact, so it is not a free syntax check. For a free check, validate the spec instead.
• Hosted inference is not a rendering or simulation path. It has no access to your cluster, your PVCs, or a GPU; give it S3 or local inputs it can read.
• The key belongs in credentials, not in a spec or a shell history. Persist it with npa configure --save-env-credentials (atomic 0600 write, never printed).

Verify

npa/.venv/bin/python -m pytest npa/tests/guardrails/test_skills_index.py -q