Qwen3-30B-A3B NVFP4 Evaluation Results (MoE, reasoning)

20 NVFP4 W4A4 variants of Qwen/Qwen3-30B-A3B, all scored against one BF16 baseline on six reasoning tasks under one fixed serving and harness config. The variants come from two batches that vary different things — a calibration & algorithm batch (12 variants: RTN / GPTQ / AWQ / SmoothQuant x perfect-blend vs UltraChat) and an observer batch (8 variants: nvfp4_expanded_imatrix vs nvfp4_expanded_mse x GPTQ x AWQ) — but every table below ranks all 20 together. Observer batch rows are marked † throughout.

Setup

Baseline — Qwen/Qwen3-30B-A3B (BF16)

Variants — 20 NVFP4 W4A4 models, all nvfp4-pack-quantized, group_size 16, tensor_group strategy, all 48 mlp.gate routers and lm_head left unquantized. Naming is NVFP4[-GPTQ][-AWQ|-SmoothQuant][-ultrachat|-IMATRIX|-MSE]:

The 12 unmarked variants all use the default observer; the 8 † variants all use the perfect-blend calibration set. config.json does not record the AWQ step — AWQ and non-AWQ checkpoints have byte-identical configs and safetensors headers. Confirm from recipe.yaml (look for the AWQModifier block).

Seeds — 1234, 2345, 3456. aime25 additionally uses 4567, 5678, 6789, 7890, 8901.

Sampling — temperature 0.6, top_p 0.95, top_k 20

Serving — vLLM v0.28.0, TP=1 (one GPU per model), max-model-len 40960, no YaRN, --reasoning-parser qwen3, --gpu-memory-utilization 0.9

Harness — eval-agent --category reasoning, num_concurrent 32, timeout 3600

Recovery — seed-mean(quantized) / seed-mean(BF16), per task, on the metric listed for that task. Every score in this document is a mean over seeds.

Recommended models

Ranked by mean recovery over the five non-gsm8k tasks (mmlu_pro_chat, ifeval, math_500, gpqa_diamond, aime25).

• NVFP4-GPTQ-IMATRIX † — Highest floor, best on the hard tasks. Its worst task (0.9706) is the best worst-task of all 20, and it is the only variant in the document to exceed BF16 on gpqa_diamond (1.0077) — the single hardest column here. Where it gives ground is mmlu_pro (0.9709, rank 15 of 20); it is last on nothing.
• NVFP4-GPTQ-AWQ-ultrachat — Best aggregate, safest default. Highest 6-task mean of any variant (0.9885) because it is the only one that also wins gsm8k (1.0066), and it holds 2^nd-best mmlu_pro (0.9986) and 2^nd-best aime25 (0.9883). Its soft column is gpqa (0.9536).
• NVFP4-AWQ-MSE † — aime25 / math-leaning pick. The only variant to match BF16 on aime25 (1.0000) and above BF16 on math_500 (1.0014), with a respectable mmlu_pro (0.9806) — but it pays ~6 pts of gpqa recovery (0.9485) for it.

Caveats: 1 and 2 swap places depending on whether gsm8k is counted — GPTQ-IMATRIX leads by 0.0038 on the five non-gsm8k tasks, GPTQ-AWQ-ultrachat by 0.0011 on all six. They are also from different batches, so read the gap loosely and pick on profile: GPTQ-IMATRIX if nothing may regress badly, GPTQ-AWQ-ultrachat if gsm8k-style CoT arithmetic or mmlu_pro matter. Ranks 2 and 3 are separated by 0.0009, inside the noise on every task that drives it.

Avoid the bottom of the table: NVFP4-SmoothQuant (0.9579), NVFP4-MSE (0.9592, worst or second-worst on four of six tasks) and NVFP4-GPTQ-SmoothQuant (0.9633).

Takeaways

• Nothing recovers BF16. Best 0.9885, mean over 20 is 0.9746. The gap is almost entirely gpqa_diamond (0.9485) and aime25 (0.9503) — the worst task for 19 of 20 variants. ifeval (0.9923) and math_500 (0.9993) are untouched.
• Take GPTQ. The only axis with a consistent, resolvable effect: gsm8k +0.0204 over 10 matched pairs, 9 of 10 positive. Costs a small but well-measured −0.0062 on mmlu_pro.
• UltraChat buys mmlu_pro (+0.0231, 6 of 6 pairs) and costs math_500 (−0.0148). The single most reliable effect in the sweep. Use it unless math is the target.
• The observer is a trade, not an upgrade. GPTQ-IMATRIX vs GPTQ is +0.0618 gpqa / −0.0197 gsm8k, both past their floors, everything else flat. Pick by target metric.
• AWQ is a coin flip; SmoothQuant is a cost. AWQ splits 4/4 on gsm8k, mmlu_pro, math_500 and gpqa — its only resolvable cells are two gpqa losses stacked on GPTQ (−0.0593, −0.0541), so don't stack it there. SmoothQuant is negative on 4 of 4 gsm8k pairs with no upside column.
• Recipe choice matters less than it looks. Best member per family: GPTQ+AWQ 0.9885 > GPTQ 0.9874 > AWQ 0.9831 > RTN 0.9770 > GPTQ+SmoothQuant 0.9743 > SmoothQuant 0.9726 — and within a family, calibration/observer moves things as much (GPTQ+AWQ spans 0.9689–0.9885). Plain RTN at 0.9770 beats 11 of the other 19.
• Read gpqa and aime25 loosely (floors ±0.0490, ±0.0807); ifeval carries no signal at all. mmlu_pro is the clean discriminator — 4.2 pt spread on a ~0.2 seed std. Recovery above 1.0 means "no measurable loss," not a gain.

Knobs to turn:

1. GPTQ — always on. Only axis with a resolvable, consistent effect (gsm8k +0.0204, 9/10).
2. Calibration set → UltraChat vs Perfectblend
3. Observer → imatrix — biggest single resolvable cell in the document (gpqa +0.0618 over plain GPTQ)
4. AWQ — given time
5. SmoothQuant — don't.

Metrics

One metric per task everywhere in this document — the strict / exact-match variant wherever the harness emits more than one. Noise floor = ~95% threshold for a difference between two seed-means, in recovery points; smaller gaps are not evidence.

Recovery vs BF16

Bold = top two quantized variants on that task. Observer-sweep variants (-IMATRIX / -MSE) are marked †.

Per-task detail

gsm8k_platinum_cot_llama

exact_match,strict-match · noise floor ±0.0111

mmlu_pro_chat

exact_match,custom-extract · noise floor ±0.0036

ifeval

inst_level_strict_acc,none · noise floor ±0.0125

math_500

pass@k:k=1&n=1 · noise floor ±0.0177

gpqa_diamond

gpqa_pass@k:k=1 · noise floor ±0.0490

aime25

pass@k:k=1&n=1 · noise floor ±0.0807

Effect of adding GPTQ (actorder: static)

All 10 pairs across both batches. recovery(with GPTQ) − recovery(without), holding the rest of the recipe, the calibration set and the observer fixed; bold = exceeds that task's noise floor. Observer-batch pairs are marked †.

Effect of adding AWQ

All 8 pairs across both batches. recovery(with AWQ) − recovery(without); bold = exceeds that task's noise floor. Observer-batch pairs are marked †.

Effect of adding SmoothQuant

All 4 pairs; the observer batch has no SmoothQuant checkpoint. recovery(with SmoothQuant) − recovery(without); bold = exceeds that task's noise floor.

Effect of calibration set: UltraChat vs perfect-blend

All 6 pairs. Calibration batch only — every observer-batch checkpoint is calibrated on perfect-blend, so this axis cannot be widened to 20. recovery(ultrachat) − recovery(perfect-blend); bold = exceeds that task's noise floor.

Effect of the weight observer: mse vs imatrix †

All 4 pairs. Observer batch only — every calibration-batch checkpoint uses the default observer, so this axis cannot be widened to 20. recovery(mse) − recovery(imatrix); bold = exceeds that task's noise floor.

Algorithm families

All 20 variants grouped by weight-side algorithm, ignoring calibration set and observer. Envelope = best member on that task (so wider families get a higher envelope); Best member = highest 6-task mean of any single member; Spread = range of member 6-task means, i.e. how much the calibration/observer choice moves that family.

Run provenance

Observer batch (†), launched in two concurrent groups of four, one GPU each. The 12 calibration & algorithm runs predate these and their run directories are no longer retained locally; their per-seed numbers are carried forward verbatim from the earlier revision of this document.

Regenerate the raw per-seed comparison for the observer batch with python3 /home/dsikka/compare_runs.py. The merged 20-model recovery, aggregate and per-task tables above are produced by python3 /home/dsikka/merge_tables.py, which recomputes the observer batch from its run directories and re-parses the 12 older variants' per-seed numbers out of this file — so it must be run against a revision that still contains them. The observer-batch delta tables come from python3 /home/dsikka/gen_obs_tables.py.

Reproduction notes

• Runs were launched via /home/dsikka/run_full_generic.sh <MODEL_PATH> <SERVED_NAME> <PORT> <TAG>, which hardcodes every eval-relevant setting so concurrent runs cannot drift apart.
• Concurrent runs need cache isolation. runner._clear_lighteval_caches() rmtree's the CWD-relative .litellm_cache and the entire $HF_HOME/lighteval tree before each lighteval task, so parallel runs sharing either will clobber each other mid-generation. Each run gets its own CWD (wd_<tag>) and HF_HOME (.cache/hf_<tag>), with hub/datasets/xet caches shared. Copy token and stored_tokens into each new HF_HOME or gated GPQA Diamond 401s.
• lm-eval ignores the --output_path filename and appends its own ISO timestamp. eval_agent/summary.py was fixed to glob for the real file; before that fix, every lm-eval score was silently dropped from summary_data.json.

Source

Ported from issue #3001, where this sweep was originally published. The reproduction scripts referenced under run provenance are local to the author's machine and are not published.