Model benchmarks

Run paid benchmarks only when the user explicitly asks. Use the wrapper so raw results survive and failed evaluations still retain their timing data:

node .agents/skills/model-benchmarks/scripts/run-benchmark.mjs <model-id> [benchmark options]

The wrapper builds @llmgateway/benchmarks, creates an absolute output path, and writes results under .context/benchmarks. It defaults to every active mapping, the smoke profile, and a 120-second per-target budget. Override those defaults with ordinary CLI options, for example:

node .agents/skills/model-benchmarks/scripts/run-benchmark.mjs <model-id> \
  --mapping <provider> --profile load --budget 300000

Do not pass --output or --format; the wrapper owns them. Set BENCHMARK_OUTPUT_DIR when results belong elsewhere. The benchmark CLI reads LLM_GATEWAY_API_KEY, falls back to LLMGATEWAY_API_KEY, and loads the root .env file.

Profiles

smoke (default), standard, load, and coding. coding measures agentic performance: each case is a small task in a virtual repository that the model must finish through a multi-turn list_files / read_file / search / write_file / run_tests loop, and run_tests really executes the candidate code. It reports solve rate, turns, tool calls, invalid and repeated tool calls, end-to-end wall clock, and cost per solved task in an "Agentic coding" section. Budget it generously — one task is many upstream requests:

node .agents/skills/model-benchmarks/scripts/run-benchmark.mjs <model-id> \
  --profile coding --budget 300000 --timeout 120000

A budget is a maximum per target, not total run time. A request already in flight may finish after its target reaches the budget. Keep cache and fallback disabled unless the user explicitly asks otherwise, so results stay pinned to the selected mapping.

The wrapper preserves:

• .json: raw trials, responses, usage, cost, and stream events.
• .md and .html: built-in benchmark reports.
• -timings.md: timings grouped by target and case, including evaluator failures.
• -timings.csv: every trial's timing and usage fields.

Against a local gateway

Point the run at the local gateway with --url and an api-key env var holding a seeded token:

LOCAL_GATEWAY_KEY=test-token node .agents/skills/model-benchmarks/scripts/run-benchmark.mjs <model-id> \
  --url http://127.0.0.1:${GATEWAY_PORT:-4001}/v1/chat/completions --api-key-env LOCAL_GATEWAY_KEY

Do not use LLM_GATEWAY_API_KEY for this — the CLI loads the root .env, which may already define it and point at production.

Start that gateway under an entry filename other than serve.js, detached into its own session. Other worktrees on this machine kill stray gateways with pkill -f "dist/serve.js", which matches every worktree, and nohup does not ignore SIGTERM. A mid-run kill shows up as request_error: fetch failed with a near-zero totalMs on every remaining target, which is indistinguishable from a provider outage in the report:

echo 'import "./serve.js";' > apps/gateway/dist/bench-serve.js
( cd apps/gateway && nohup python3 -c 'import os,sys; os.setsid(); os.execvp(sys.argv[1], sys.argv[1:])' \
    node --env-file=../../.env dist/bench-serve.js > /tmp/gateway.log 2>&1 < /dev/null & )

macOS has no setsid binary, hence the python3 one-liner. Confirm the process shows PPID 1 before starting paid work, and kill it by PID afterwards.

If rendering fails after JSON is saved, render it without rerunning paid requests:

node .agents/skills/model-benchmarks/scripts/render-results.mjs <absolute-json-path>

If a run makes requests but saves no JSON, report the loss and ask before rerunning. After completion, link the reports and summarize quality, reliability, TTFT, total latency, throughput, buffering, truncation, cost, and provider errors. For coding, also summarize solve rate, turns and tool calls per task, invalid and repeated tool calls, and the stop reasons.