quark-onnx-eval-runner

Purpose

Walk a maintainer through manual verification of the ONNX skill family across the four contract categories: routing, planning, artifact, and recovery. Run this after modifying any quark-onnx-* skill, after a Quark ONNX upgrade, or before tagging a release.

Inputs

• The current ONNX skill files under .claude/skills-impl/{l1-atomic,l2-workflows,l3-recipes}/onnx/
• The entry stubs under .claude/skills/quark-onnx-*
• The contract schemas under .claude/skills-impl/shared/contracts/
• The example prompts under examples/agent_skills/prompts/ (add ONNX-specific cases as the catalog grows)

Outputs: validation_report.md

A markdown report recording per-category pass/fail and concrete evidence for each finding.

Schema: validation_report.schema.json

# ONNX Skill Verification Report

## Summary
| Category | Cases Run | Pass | Fail |
|----------|-----------|------|------|
| routing  | N         | N    | 0    |
| planning | N         | N    | 0    |
| artifact | N         | N    | 0    |
| recovery | N         | N    | 0    |

## Failures
### <category> / <case name>
- **Expected**: ...
- **Got**: ...
- **Impact**: ...
- **Fix**: ...

Manual Verification Protocol

For each category below, run at least one case and record the result in the report. As ONNX prompts are not yet enumerated in examples/agent_skills/prompts/, the cases below double as the seed catalog — add more as the ONNX skill set grows.

1. Routing

Goal: verify that quark-onnx-router (and Claude's auto-routing via the descriptions in .claude/skills/quark-onnx-*) maps natural-language ONNX goals to the correct downstream skill and never silently routes ONNX requests through a torch skill.

Manual procedure:

1. Pick a user-style prompt that names a .onnx artifact or ONNX-specific vocabulary.

2. In a fresh Claude Code session at the Quark repo root, paste the prompt.

3. Observe which skill Claude invokes first.

4. Compare against the expected target skill. Examples of expected mappings:

   • "Quantize my ./models/yolov8n.onnx to XINT8 for AMD NPU CNN" → quark-onnx-ptq (which loads quark-onnx-ptq-workflow)
   • "Run AutoSearchPro on this .onnx with the XINT8_SEARCH preset" → quark-onnx-autosearch-pro
   • "Analyze this .onnx — what opset is it, is it NPU-compatible, is it already QDQ?" → quark-onnx-model-intake
   • "Validate my quantized model.onnx — did QDQ insertion happen, are the non-quantized initializers byte-identical?" → quark-onnx-result-validator
   • "onnxruntime-gpu import fails, CUDAExecutionProvider not in providers list" → quark-onnx-install (or quark-onnx-debug if the user already attempted install)
   • "quantize_static failed with custom-op library load failure for BFPQuantizeDequantize" → quark-onnx-debug
   • "Is onnxruntime-rocm installed correctly? Show me the install matrix" → quark-onnx-install

5. Cross-backend guard: also run one negative prompt that mentions a .onnx path and confirm Claude does not route to quark-torch-* (e.g., "quantize ./models/foo.onnx with FP8" must not land on quark-torch-ptq).

Pass criteria: the first skill invoked matches the expected target, and no ONNX prompt is routed to a torch skill.

2. Planning

Goal: verify that quark-onnx-quant-plan produces internally consistent plans for typical ONNX inputs and that the deployment-target gates are respected.

Manual procedure:

1. Construct (or take from a prior session) a model_analysis.json produced by quark-onnx-model-intake for a representative model (e.g., YOLOv8n exported at opset 17, Conv-heavy, 6.2 MB inline).
2. Hand it to quark-onnx-quant-plan with a target preset (e.g., XINT8) and a deployment target (e.g., AMD NPU CNN).
3. Inspect the produced quant_plan.json for:
   • preset matches the requested preset
   • activation_spec and weight_spec are consistent with the preset (e.g., both XInt8Spec for XINT8)
   • EnableNPUCnn=True is set when the target is AMD NPU CNN
   • use_external_data_format is True iff the model is >2 GB
   • algo_config is a non-empty list when CLE or AdaRound was requested or recommended
   • exclude is a list (may be empty) and never contains an op the plan also quantizes
   • requires_confirmation is set when the plan deviates from preset defaults
   • Negative gate: the plan refuses incompatible combos (e.g., BFP16 + AMD NPU CNN) rather than silently downgrading

Pass criteria: the plan validates against quant_plan.schema.json, contains no internal contradictions, and explicitly rejects unsupported deployment-target / preset combinations.

3. Artifact

Goal: verify that ONNX workflow output artifacts conform to their JSON schemas and that the generated standalone script + manifest produced by quark-onnx-ptq-workflow agree with each other.

Manual procedure:

1. Take the quant_plan.json from the planning case.
2. Run quark-onnx-ptq-workflow to produce a run_manifest.yaml and the standalone <name>_ptq.py script in the user's working directory.
3. Validate the manifest against .claude/skills-impl/shared/contracts/run_manifest.schema.json (use any JSON-schema validator, e.g., the jsonschema Python package).
4. Spot-check that:
   • All required fields are present in the manifest
   • The manifest's command references the generated script path and uses python3
   • The manifest's resolved QConfig matches the plan's preset / algo_config / EnableNPUCnn / use_external_data_format
   • The generated script imports only from quark.onnx and the standard ORT calibration API (no editing of upstream examples/onnx/ or quark/onnx/ files)
   • Output paths reflect any .onnx_data sidecar when external data is enabled

Pass criteria: schema validation passes; the manifest's resolved config matches the plan; the generated script is self-contained in the user's working directory.

4. Recovery

Goal: verify that quark-onnx-debug correctly diagnoses known ONNX-side error patterns and that handoffs to quark-onnx-install happen for runtime/provider issues.

Manual procedure:

1. Pick a known ONNX error scenario. Examples:
   • RuntimeError: CUDAExecutionProvider not in available providers after installing onnxruntime (CPU build) instead of onnxruntime-gpu.
   • Custom-op library load failure for BFPQuantizeDequantize or MXQuantizeDequantize (missing C++ build, ABI mismatch).
   • model.onnx >2 GB and the run fails with "external data not found" because use_external_data_format was not set or the sibling .onnx_data was not staged.
   • OOM during calibration on a vision model with num_calib_data=1000 and batch_size=4.
   • AdaRound divergence with default learning rate.
   • NPU CNN run fails because activation scales are not power-of-two.
2. Present the error (full traceback) to quark-onnx-debug.
3. Verify the diagnosis:
   • Root cause is correctly identified
   • A concrete fix command or config change is provided
   • For runtime/provider issues, the skill explicitly hands off to quark-onnx-install instead of silently swapping execution providers
   • For OOM, the skill suggests the documented ladder: reduce num_calib_data → drop batch_size to 1 → move calibration to CPU (OptimDevice="cpu")
   • For custom-op load failures, the skill names the expected library path / build step rather than recommending the user ignore the op

Pass criteria: the diagnosis names the actual root cause, suggests a fix that would actually work, and respects the "never silently fall back to CPU" rule from quark-onnx-ptq-workflow.

Rules

• Run at least one case per category before any ONNX release — even small wording changes in preset names or custom-op names can shift routing or break generated scripts.
• A failing case means the skill is broken, not the procedure — investigate the skill first. Only update the expected behavior if the skill change was intentional.
• Report all results, not just failures. A clean run is positive evidence and worth recording.
• Document new cases inline in the report. As the ONNX skill set grows (e.g. new deployment target, new preset, new AutoSearchPro preset), add cases that cover the new triggers.
• Cross-backend isolation is part of the protocol. Every routing case must include a guard that confirms ONNX requests stay on quark-onnx-* skills.

Interaction Flow

1. Select scope: which categories to verify — all four, or a subset affected by a recent change?
2. Pick or write cases: start from the examples above; add ONNX prompt files under examples/agent_skills/prompts/ as the catalog grows.
3. Run each case manually following the protocol above in a fresh Claude Code session.
4. Record results in validation_report.md using the template.
5. Hand off failures to quark-onnx-skill-sync if they look like upstream drift, to quark-onnx-doc-drift-check if they look like stale user-facing facts, or directly to the affected skill's owner if it's a content bug.

Recovery

• If a case can't run because a prerequisite artifact is missing (e.g., no model_analysis.json for the planning case), report the missing producer skill and stop — do not fabricate the input.
• If the ONNX custom-op binaries are not built locally, the recovery case for the custom-op load failure may produce a real failure rather than a simulated one — note this distinction in the report so future runs don't confuse genuine environment gaps with skill bugs.
• If Claude Code itself is misbehaving (ONNX skill not discovered, stub not loading), that's an infrastructure issue separate from skill quality — record it distinctly in the report.
• If an ONNX prompt routes to a torch skill (or vice versa), this is a routing-category failure and must block the release — cross-backend mis-routing produces wrong artifacts silently.