quark-torch-eval-runner

Purpose

Walk a maintainer through manual verification of the skill system across the four contract categories: routing, planning, artifact, and recovery. Run this after modifying any skill, after a Quark upgrade, or before tagging a release.

Inputs

• The current skill files under .claude/skills-impl/
• The entry stubs under .claude/skills/
• The contract schemas under .claude/skills-impl/shared/contracts/
• The example prompts under examples/agent_skills/prompts/

Outputs: validation_report.md

A markdown report recording per-category pass/fail and concrete evidence for each finding.

Schema: validation_report.schema.json

# Skill Verification Report

## Summary
| Category | Cases Run | Pass | Fail |
|----------|-----------|------|------|
| routing  | N         | N    | 0    |
| planning | N         | N    | 0    |
| artifact | N         | N    | 0    |
| recovery | N         | N    | 0    |

## Failures
### <category> / <case name>
- **Expected**: ...
- **Got**: ...
- **Impact**: ...
- **Fix**: ...

Manual Verification Protocol

For each category below, run at least one case and record the result in the report. The example prompts in examples/agent_skills/prompts/ are good starting cases; add more as the skill set grows.

1. Routing

Goal: verify that quark-torch-router (and Claude's auto-routing via descriptions in .claude/skills/) maps natural-language goals to the correct downstream skill.

Manual procedure:

1. Pick a user-style prompt (e.g., from examples/agent_skills/prompts/torch_llm_ptq.md or a freshly invented one).
2. In a fresh Claude Code session at the Quark repo root, paste the prompt.
3. Observe which skill Claude invokes first.
4. Compare against the expected target skill. Examples of expected mappings:
   • "Quantize Llama-2-7B with FP8 and export to HuggingFace format" → quark-torch-ptq (which loads quark-torch-llm-ptq-workflow)
   • "What GPU do I have?" → quark-env-preflight
   • "pip install amd-quark fails" → quark-install or quark-torch-debug
   • "Convert my quantized model to GGUF" → quark-torch-export

Pass criteria: the first skill invoked matches the expected target.

2. Planning

Goal: verify that quark-torch-quant-plan produces internally consistent plans for typical inputs.

Manual procedure:

1. Construct (or take from a prior session) a model_analysis.json for a representative model (e.g., Qwen3-8B).
2. Hand it to quark-torch-quant-plan with a target scheme (e.g., fp8).
3. Inspect the produced quant_plan.json for:
   • global_scheme matches the requested scheme
   • exclude_layers is non-empty and includes lm_head for LLMs
   • No conflicting options (e.g., kv_cache_dtype: fp8 requires the FP8 scheme path)
   • requires_confirmation is set when the plan deviates from defaults

Pass criteria: the plan validates against quant_plan.schema.json and contains no internal contradictions.

3. Artifact

Goal: verify that workflow output artifacts conform to their JSON schemas.

Manual procedure:

1. Take the quant_plan.json from the planning case.
2. Run quark-torch-llm-ptq-workflow to produce a run_manifest.yaml.
3. Validate the manifest against .claude/skills-impl/shared/contracts/run_manifest.schema.json (use any JSON-schema validator, e.g., the jsonschema Python package, or eyeball required fields).
4. Spot-check that:
   • All required fields are present
   • Referenced artifact paths exist or are clearly marked as to-be-produced
   • Export configuration is included (e.g., export.formats)

Pass criteria: schema validation passes; required fields are present and consistent.

4. Recovery

Goal: verify that quark-torch-debug correctly diagnoses known error patterns.

Manual procedure:

1. Pick a known error scenario. Examples:
   • AttributeError: 'PreTrainedTokenizerFast' object has no attribute 'get_max_length' (transformers version mismatch)
   • CUDA out-of-memory during a 70B-parameter quantization
   • Missing lm_head exclusion causing accuracy collapse
2. Present the error to quark-torch-debug.
3. Verify the diagnosis:
   • Root cause is correctly identified
   • A concrete fix command or config change is provided
   • The affected version range or environment condition is stated

Pass criteria: the diagnosis names the actual root cause and suggests a fix that would actually work.

Rules

• Run at least one case per category before any release — even small wording changes can shift routing behavior.
• A failing case means the skill is broken, not the procedure — investigate the skill first. Only update the expected behavior if the skill change was intentional.
• Report all results, not just failures. A clean run is positive evidence and worth recording.
• Document new cases inline in the report. As the skill set grows, add cases that cover new triggers.

Interaction Flow

1. Select scope: which categories to verify — all four, or a subset affected by a recent change?
2. Pick or write cases: use prompts from examples/agent_skills/prompts/ or write minimal new ones.
3. Run each case manually following the protocol above.
4. Record results in validation_report.md using the template.
5. Hand off failures to quark-torch-skill-sync if they look like upstream drift, or directly to the affected skill's owner if it's a content bug.

Recovery

• If a case can't run because a prerequisite artifact is missing (e.g., no model_analysis.json for the planning case), report the missing producer skill and stop — do not fabricate the input.
• If Claude Code itself is misbehaving (skill not discovered, stub not loading), that's an infrastructure issue separate from skill quality — record it distinctly in the report.