NVFP4 Example
Qwen3.8-Flash-Next NVFP4 Example
Qwen3.8-Flash-Next is a sparse MoE model that requires transformers >= 5.17.0. This example quantizes the model to NVFP4 using calibration data and the same single-scheme MoE recipe used for the other Qwen3.8 MoE variants.
Code Walkthrough
- Load model
- Configure quantization recipe
- Apply quantization
- Save to disk
1. Load Model
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor.utils import load_context
# NOTE: This example requires transformers >= 5.17.0
MODEL_ID = "Qwen/Qwen3.8-Flash-Next"
with load_context():
model = AutoModelForCausalLM.from_pretrained(MODEL_ID)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
2. Configure Quantization Recipe
We quantize the linear layers to NVFP4 and keep lm_head, the MoE router (mlp.gate), and shared expert gates at full precision.
from llmcompressor.modifiers.quantization import QuantizationModifier
recipe = QuantizationModifier(
targets="Linear",
scheme="NVFP4",
ignore=[
"re:.*lm_head",
"re:.*mlp.gate$",
"re:.*shared_expert_gate.*",
],
)
3. Apply Quantization
from llmcompressor import oneshot
oneshot(
model=model,
processor=tokenizer,
recipe=recipe,
dataset="perfectblend",
splits="train[:512]",
max_seq_length=2048,
num_calibration_samples=512,
pipeline="sequential",
)
4. Save to Disk
SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-NVFP4"
model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)
Pre-quantized Checkpoint
If you do not need to run quantization yourself, the Red Hat AI team has published the resulting NVFP4 checkpoint as RedHatAI/Qwen3.8-Flash-Next-NVFP4. It can be loaded directly in vLLM: