Skip to content

4-bit Example

Qwen3.8-27B 4-bit Quantization Example

Qwen3.8-27B is a vision-language model. This example produces 4-bit checkpoints with AWQ followed by GPTQ, covering three formats:

  • W4A16 (INT4): weight-only 4-bit quantization with 16-bit activations
  • NVFP4: 4-bit weights and activations with per-group-16 scaling
  • MXFP4: 4-bit weights and activations with a shared E8M0 scale

The full example scripts can be found here:

Code Walkthrough

Let's walk through the main steps of the quantization process: 1. Load model 2. Configure quantization algorithm and scheme 3. Prepare calibration dataset 4. Apply quantization 5. Save to disk in compressed-tensors format

1. Load Model

from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration

from llmcompressor.utils import load_context

MODEL_ID = "Qwen/Qwen3.8-27B"

# Qwen3.8-27B is a vision-language model, so it must be loaded with its
# Qwen3_5ForConditionalGeneration class to keep the vision tower.
with load_context(Qwen3_5ForConditionalGeneration):
    model = Qwen3_5ForConditionalGeneration.from_pretrained(MODEL_ID)
processor = AutoProcessor.from_pretrained(MODEL_ID)

2. Configure Quantization Algorithm and Scheme

All three formats use AWQ to smooth activations before GPTQ performs the weight quantization. The vision tower, lm_head, embeddings, and the Gated DeltaNet projections that are incompatible with 4-bit quantization are skipped.

W4A16 (INT4)

from llmcompressor.modifiers.gptq import GPTQModifier
from llmcompressor.modifiers.transform.awq import AWQModifier

recipe = [
    AWQModifier(duo_scaling="both"),
    GPTQModifier(
        targets="Linear",
        scheme="W4A16",
        ignore=[
            "re:visual.*",
            "re:model.visual.*",
            r"re:.*lm_head",
            "re:.*embed_tokens$",
            r"re:.*linear_attn\.in_proj_a$",
            r"re:.*linear_attn\.in_proj_b$",
        ],
    ),
]

NVFP4

For NVFP4, attention projections, lm_head, and the MLP layers in the last eight decoder blocks are kept in FP8 to preserve accuracy, while the remaining MLP projections are quantized to NVFP4. The KV cache is quantized to FP8 as well.

from compressed_tensors.quantization.quant_scheme import (
    FP8_DYNAMIC,
    NVFP4,
    QuantizationScheme,
)
from llmcompressor.modifiers.gptq import GPTQModifier
from llmcompressor.modifiers.transform.awq import AWQModifier

recipe = [
    AWQModifier(duo_scaling="both"),
    GPTQModifier(
        config_groups={
            "attention": QuantizationScheme(
                targets=[
                    r"re:.*self_attn\.(q|k|v|o)_proj$",
                    r"re:.*linear_attn\.(in_proj_qkv|in_proj_z|out_proj)$",
                    r"re:.*lm_head",
                    r"re:.*layers\.(56|57|58|59|60|61|62|63)\.mlp\..*(gate|up|down)_proj$",
                ],
                **FP8_DYNAMIC,
            ),
            "mlp": QuantizationScheme(
                targets=[r"re:.*mlp\..*(gate|up|down)_proj$"],
                **NVFP4,
            ),
        },
        ignore=[
            "re:visual.*",
            "re:model.visual.*",
        ],
        kv_cache_scheme={
            "num_bits": 8,
            "type": "float",
            "symmetric": True,
            "strategy": "tensor",
            "dynamic": False,
            "observer": "static_minmax",
        },
    ),
]

MXFP4

from llmcompressor.modifiers.gptq import GPTQModifier
from llmcompressor.modifiers.transform.awq import AWQModifier

recipe = [
    AWQModifier(duo_scaling="both"),
    GPTQModifier(
        targets="Linear",
        scheme="MXFP4",
        ignore=[
            "re:visual.*",
            "re:model.visual.*",
            r"re:.*lm_head",
            "re:.*embed_tokens$",
            r"re:.*linear_attn\.in_proj_a$",
            r"re:.*linear_attn\.in_proj_b$",
        ],
        kv_cache_scheme={
            "num_bits": 8,
            "type": "float",
            "symmetric": True,
            "strategy": "tensor",
            "dynamic": False,
            "observer": "static_minmax",
        },
    ),
]

3. Prepare Calibration Dataset

The examples use the prebaked perfectblend calibration dataset with 512 samples and a maximum sequence length of 4096 tokens.

4. Apply Quantization

from llmcompressor import oneshot

oneshot(
    model=model,
    processor=processor,
    recipe=recipe,
    dataset="perfectblend",
    splits="train[:512]",
    max_seq_length=4096,
    num_calibration_samples=512,
)

5. Save to Disk in Compressed-Tensors Format

SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-W4A16-GPTQ-AWQ"
model.save_pretrained(SAVE_DIR)
processor.save_pretrained(SAVE_DIR)

Use the corresponding suffix for the other formats: -NVFP4-GPTQ-AWQ for NVFP4 and -MXFP4 for MXFP4.

Calibration-free MXFP6A8

A calibration-free alternative is also available. It keeps the dense MLP projection weights in MXFP6, quantizes attention, Gated DeltaNet, and MTP projections to MXFP8, and keeps lm_head in FP8 with per-channel weights. The full script is here.

from compressed_tensors.quantization import QuantizationConfig, QuantizationScheme
from compressed_tensors.quantization.quant_scheme import (
    FP8_DYNAMIC,
    MXFP6A8,
    MXFP8,
)

from llmcompressor import model_free_ptq

config = QuantizationConfig(
    config_groups={
        "mtp": QuantizationScheme(targets=[r"re:.*mtp\.layers\..*"], **MXFP8),
        "mlp": QuantizationScheme(
            targets=[r"re:.*mlp\.(gate|up|down)_proj$"], **MXFP6A8
        ),
        "attn": QuantizationScheme(
            targets=[
                r"re:.*self_attn\.[qkvo]_proj$",
                r"re:.*linear_attn\.(in_proj_qkv|in_proj_z|out_proj)$",
            ],
            **MXFP8,
        ),
        "lm_head": QuantizationScheme(targets=[r"re:.*lm_head$"], **FP8_DYNAMIC),
    }
)

model_free_ptq(
    model_stub=MODEL_ID,
    save_directory=MODEL_ID.rstrip("/").split("/")[-1] + "-MXFP6A8",
    config=config,
    ignore=[
        r"re:.*visual\..*",
        r"re:.*in_proj_(a|b|ba)$",
        r"re:.*mtp\.fc$",
        r"re:.*embed_tokens$",
    ],
    device="cuda:0",
)