quark-torch-quant-plan

Purpose

Convert a model analysis plus the user's intent into a confirmed quant_plan.json. This skill makes the quantization decisions — which scheme, which algorithm, what to exclude — without generating scripts or executing PTQ. The plan is the contract between the user's intent and the execution step.

Inputs

• model_analysis.json from quark-torch-model-intake
• env_context.json for accelerator-aware scheme recommendations
• User preferences (target precision, accuracy goal)

Outputs: quant_plan.json

Records the chosen scheme, algorithm, layer overrides, calibration settings, and evaluation intent.

Schema: quant_plan.schema.json

{
  "model": {
    "model_type": "qwen3",
    "analysis_ref": "./model_analysis.json"
  },
  "global_scheme": "fp8",
  "kv_cache_scheme": "fp8",
  "exclude_layers": ["lm_head"],
  "layer_quant_config": {},
  "algorithm": null,
  "calibration": {
    "dataset": "pileval",
    "num_calib_data": 128,
    "seq_len": 512
  },
  "evaluation_intent": "smoke",
  "requires_confirmation": false
}

Available Quantization Schemes (21 total)

Weight-Only INT4 (best for deployment size reduction)

Scheme	Description	Use Case
int4_wo_32	INT4, group size 32	Highest accuracy among INT4
int4_wo_64	INT4, group size 64	Good balance
int4_wo_128	INT4, group size 128	Smaller overhead
int4_wo_per_channel	INT4, per-channel	Least overhead
uint4_wo_32/64/128/per_channel	Unsigned INT4 variants	GGUF export compatibility

Weight+Activation INT8

Scheme	Description	Use Case
int8	INT8 per-tensor for both W and A	CPU deployment, good accuracy

FP8 (best accuracy-size tradeoff for GPU inference)

Scheme	Description	Use Case
fp8	FP8 E4M3 per-tensor	Standard GPU quantization
ptpc_fp8	Per-Token-Per-Channel FP8	Higher accuracy, dynamic activation quantization

OCP Microscaling Formats

Scheme	Description	Use Case
mxfp4	OCP MXFP4	Aggressive compression
mxfp6_e3m2	OCP MXFP6 (E3M2)	Better range
mxfp6_e2m3	OCP MXFP6 (E2M3)	Better precision
mxfp4_mxfp6_e2m3	MXFP4 weights + MXFP6 activations	Mixed precision
mxfp4_fp8	MXFP4 weights + FP8 activations	Mixed precision

AMD-Specific

Scheme	Description	Use Case
amdfp4	amdfp4, group size 16	AMD MI300X optimized
amdfp4_g32	amdfp4, group size 32	AMD MI300X, less overhead

Other

Scheme	Description	Use Case
nvfp4	NVFP4: FP4 group_size=16 with FP8 E4M3 scale	NVIDIA Blackwell/Hopper
mx6	MX6 format	Experimental
bfp16	Block Floating Point 16-bit	Experimental
int4_wa_64	INT4 weights + activations, group 64	Research

Available Algorithms (7 primary)

Algorithm	Compatible Schemes	Description
awq	INT4/UINT4 weight-only	Activation-aware weight quantization — finds optimal per-channel scaling
gptq	INT4/UINT4 weight-only	Second-order weight optimization — often better than AWQ for small models
smoothquant	INT8, FP8	Migrates quantization difficulty from activations to weights
autosmoothquant	INT8, FP8	Automatic SmoothQuant with optimal alpha search
rotation	Various	Rotation-based optimization to equalize weight distribution
gptaq	INT4/UINT4	GPTAQ variant combining GPTQ with activation quantization
qronos	Various	Custom algorithm for time-series-aware quantization

Algorithms can be combined: --quant_algo awq,smoothquant

KV Cache Quantization

• Only fp8 is supported for KV cache (--kv_cache_dtype fp8)
• Adds --min_kv_scale option (default 0.0) to prevent extreme scale values
• --kv_cache_post_rope quantizes KV cache after RoPE (inside cache) instead of at k_proj/v_proj outputs — can improve accuracy for some models

Decision Guide

Help the user choose based on their priorities:

"I want the best accuracy" → fp8 or ptpc_fp8, optionally with smoothquant "I want the smallest model" → int4_wo_32 with awq or gptq "I need CPU deployment" → int8 (the only scheme that works well on CPU) "I need GGUF format" → uint4_wo_32 with awq, export as GGUF "I'm on AMD MI300X" → amdfp4 for best hardware utilization "I'm on NVIDIA H100/Blackwell" → fp8 or nvfp4 "I want to experiment" → mxfp4 for aggressive compression research

Decision Table (MUST show to user)

ALWAYS present this table to the user and WAIT for confirmation before finalizing. Do not skip this step.

Fill in the "Value" column based on the user's request and model analysis, then show:

Decision	Value	Reason
global_scheme	(fill)	(why this scheme)
kv_cache_scheme	(fill: fp8 or null)	(explain)
exclude_layers	["lm_head"]	Standard — lm_head stays full precision
layer_quant_config	(fill: dict of pattern -> scheme, or {} if none)	(explain which patterns and why)
algorithm	(fill: algorithm or null)	(explain)
calibration_dataset	pileval	Fast default
num_calib_data	128	Standard default
seq_len	512	Standard default
evaluation_intent	smoke	Quick PPL check post-quantization

After showing the table, ask: "Confirm this plan? Any changes?"

Do NOT proceed until the user confirms.

Layer-Specific Overrides via layer_quant_config

The layer_quant_config plan field is a dict of pattern -> scheme pairs. It is the single mechanism for any "quantize layer/module X with scheme Y" intent — including attention modules, MoE experts, lm_head, etc. Each entry emits one --layer_quant_scheme PATTERN SCHEME CLI argument.

"layer_quant_config": {
  "*self_attn*": "fp8",
  "lm_head": "int8",
  "*experts*": "fp8"
}

translates to:

--quant_scheme <global_scheme> \
--layer_quant_scheme '*self_attn*' fp8 \
--layer_quant_scheme lm_head int8 \
--layer_quant_scheme '*experts*' fp8

When a user asks for attention-module quantization (e.g. "self_attn in fp8"), populate this field with the appropriate pattern (commonly *self_attn* for LLaMA-style models; adjust for models whose attention submodule has a different name). Do NOT introduce a dedicated attention field — keep all per-pattern overrides in layer_quant_config.

Rules

• Keep scope to plan creation only. Do not generate scripts, do not run quantization, do not export. Those are separate skills.
• Require a model analysis first. Without knowing the model architecture and layer count, you cannot make informed scheme recommendations. If model_analysis.json is missing, route back to quark-torch-model-intake.
• Always present the decision table before finalizing. The user should explicitly confirm the scheme, algorithm, and exclusions.
• If a risky scheme is chosen (e.g., mxfp4 on a model where accuracy loss may be significant), keep the user's choice but record the risk in the plan.

Interaction Flow

1. Check prerequisites: Is model_analysis.json available? If not, route to quark-torch-model-intake first.
2. Gather intent: What does the user care about most — accuracy, size, speed? What hardware will run inference?
3. Present the decision table: Show defaults, explain the tradeoffs, and let the user adjust.
4. Confirm: Always required. Show the final plan summary before writing it.
5. Emit: Write quant_plan.json.

Recovery

• If the model analysis is incomplete, produce a draft plan with requires_confirmation: true and note what facts are missing.
• If the user picks an unusual combination (e.g., awq with fp8 — AWQ is designed for INT4), explain why it might not work well and suggest alternatives, but respect the user's choice if they insist.
• If calibration dataset preferences are unclear, default to pileval with 128 samples — it is the fastest option and works for most models.