Skip to content

Methods and Schemes

This page defines the terms the quantization pages use and lists the names vLLM accepts for quantization methods and schemes.

The tables are generated at docs build time by docs/mkdocs/gen_files/generate_quantization_tables.py from the registries in vllm/config/quantization.py and vllm/model_executor/layers/quantization/ and from the platform allow-lists. A name added to a registry appears here on the next build. Names registered by plugins are not listed.

Terms

Term Meaning Where it appears
Method The implementation that loads a quantized model. Most methods read one checkpoint format and are named after the toolkit that writes it, e.g. compressed-tensors, modelopt, quark. quant_method in the model's config.json, or --quantization
Online quantization Quantizing full-precision weights while the model loads. --quantization <shorthand>, --quantization-config
Scheme The numeric format of one tensor: its data format, its scale format, and what shares a scale. The weight and activation fields of --quantization-config
Kernel The code that runs a layer for a given weight and activation scheme. --linear-backend, --moe-backend; see Kernel Backends

A method names how a model is loaded; a scheme names the format of one tensor. One checkpoint can hold layers with different schemes, and kernels are selected per layer from the scheme.

W<n>A<m> gives the bits of weights and activations: W4A16 is 4-bit weights with 16-bit activations, and W4A4 quantizes both. Weight-only schemes save memory; the matrix multiply still runs in 16-bit unless activations are quantized too.

Methods

These are the values of --quantization other than the online shorthands. For a pre-quantized checkpoint vLLM reads quant_method from config.json and selects the method, so the flag is normally omitted. A method that does not correspond to a checkpoint format is used only when it is passed explicitly.

Names Implementation Platforms Guide
awq, awq_marlin, auto_awq AutoAWQConfig CUDA, ROCm, XPU, CPU Guide
fp8 Fp8Config CUDA, ROCm, XPU, CPU –
w4afp8 W4AFP8Config CUDA, CPU –
fbgemm_fp8 (deprecated) FBGEMMFp8Config CUDA, ROCm, CPU –
modelopt ModelOptFp8Config CUDA, ROCm, XPU, CPU Guide
modelopt_fp4 ModelOptNvFp4Config CUDA, ROCm, CPU Guide
modelopt_mxfp8, mxfp8 ModelOptMxFp8Config CUDA, ROCm, XPU, CPU Guide
modelopt_int8 ModelOptInt8Config CUDA, ROCm, CPU Guide
modelopt_mixed ModelOptMixedPrecisionConfig CUDA, ROCm, CPU Guide
auto_gptq, gptq, gptq_marlin AutoGPTQConfig CUDA, ROCm, XPU, CPU Guide
compressed-tensors CompressedTensorsConfig CUDA, ROCm, XPU, CPU Guide
experts_int8 ExpertsInt8Config CUDA, CPU –
quark QuarkConfig CUDA, ROCm, CPU Guide
moe_wna16 MoeWNA16Config CUDA, XPU, CPU –
torchao TorchAOConfig CUDA, ROCm, CPU Guide
inc INCConfig CUDA, ROCm, XPU, CPU Guide
mxfp4 Mxfp4Config CUDA, ROCm, XPU, CPU –
gpt_oss_mxfp4 GptOssMxfp4Config CUDA, ROCm, XPU, CPU –
deepseek_v4_fp8 DeepseekV4FP8Config CUDA, ROCm, XPU, CPU –
humming HummingConfig CUDA, CPU –
online OnlineQuantizationConfig CUDA, ROCm, XPU, CPU Guide
  • Names on one row are aliases of one implementation.
  • Platforms lists where vLLM accepts the name at startup. ROCm and XPU accept only the names on their allow-list; CUDA and CPU accept every name. An accepted method can still have no kernel for a given device; see Kernel Backends.
  • A deprecated method also needs --allow-deprecated-quantization.

Online shorthands

Passed to --quantization, a shorthand quantizes the weights of a checkpoint that is not quantized. The columns show the weight scheme it applies to linear layers and to MoE expert layers; – leaves that kind of layer unquantized.

Name Linear weights MoE weights Platforms
fp8_per_tensor fp8_per_tensor_static fp8_per_tensor_static CUDA, ROCm, XPU, CPU
fp8_per_block fp8_per_block_static fp8_per_block_static CUDA, ROCm, XPU, CPU
fp8_per_channel fp8_per_channel_static fp8_per_channel_static CUDA, ROCm, XPU, CPU
mxfp8 mxfp8 mxfp8 CUDA, ROCm, XPU, CPU
mxfp4 mxfp4 mxfp4 CUDA, ROCm, XPU, CPU
mxfp4_hadamard32 mxfp4 with a 32-wide Hadamard rotation – CUDA, ROCm, CPU
int8_per_channel_weight_only – int8_per_channel_static CUDA, CPU
nvfp4_per_token – FP4 (E2M1), E4M3 per 16 elements, fp32 per tensor CUDA, CPU
nvfp4_weight_only FP4 (E2M1), E4M3 per 16 elements, fp32 per tensor – CUDA, CPU
lut_b_moe – lut_b CUDA, ROCm, CPU

Also a method name: mxfp8, mxfp4. On a checkpoint that is already quantized, these select the method of that name instead.

  • A shorthand is not a scheme, even when the two share a name: read the columns. A cell that is not a name is a scheme that has no name of its own and is reachable only through the shorthand.
  • Shorthands set weights only. The activation scheme is the default of the method that implements the weight scheme, and it can depend on the kernel that is selected.
  • The same names are accepted inside --quantization-config, including on a checkpoint that is already quantized. See Online Quantization.

Scheme names

These names are accepted by the weight and activation fields of --quantization-config:

vllm serve <model> --quantization online \
    --quantization-config '{"moe":{"weight":"mxfp4","activation":"mxfp8"}}'
Name Data Scale Scale computed Online weight for
fp8_per_tensor_static FP8 (E4M3) fp32 per tensor at load linear, MoE
fp8_per_tensor_dynamic FP8 (E4M3) fp32 per tensor per forward pass –
fp8_per_token FP8 (E4M3) fp32 per token per forward pass –
fp8_per_channel_static FP8 (E4M3) fp32 per output channel at load linear, MoE
fp8_per_block_static FP8 (E4M3) fp32 per 128×128 block at load linear, MoE
fp8_per_block_dynamic FP8 (E4M3) fp32 per 128 elements per forward pass –
mxfp8 FP8 (E4M3) E8M0 per 32 elements at load (weight), per forward pass (activation) linear, MoE
mxfp4 FP4 (E2M1) E8M0 per 32 elements at load (weight), per forward pass (activation) linear, MoE
nvfp4_per_token FP4 (E2M1) E4M3 per 16 elements, then fp32 per token (per forward pass) per forward pass –
int8_per_channel_static INT8 fp32 per output channel at load MoE
lut_b uint8 E4M3 per 8×64 block at load MoE
int8_per_group_32 INT8 fp32 per 32 elements per forward pass –
int4_per_group_32 INT4 fp16 per 32 elements at load –
int4_per_group_32_asym UINT4, asymmetric fp16 per 32 elements at load lm_head
  • Data is the element format. ExMy is a float with x exponent and y mantissa bits: E4M3 is 8-bit and E2M1 is 4-bit, each with a sign bit. E8M0 is an unsigned 8-bit exponent, i.e. a power of two.
  • Scale is the format of the scale factor and what shares one value. A second entry is an outer scale applied on top of the first.
  • Scale computed separates static scales, which are stored in the checkpoint or computed once at load, from dynamic ones, which are computed from the tensor on every forward pass. Weights use static scales. Dynamic scales are how activations are quantized without calibration data.
  • Online weight for lists the kinds of layer that online quantization can quantize to this scheme. lm_head marks a scheme that applies only to the lm_head, selected with targets. Every name parses on both fields, but a name with – here is rejected or ignored as an online weight, and only some methods accept an activation override. See Online Quantization.

The table shows each scheme as vLLM's QuantKey describes it. Formats that are not a data format plus scales are approximated: lut_b stores 3-bit indices into a codebook, which appears here as the scale, and an E8M0 scale per 32 elements that the key does not describe.

MXFP8 and MXFP4 are the OCP Microscaling formats. With b bits per element and an s-bit scale shared by g elements, a weight costs about b + s / g bits per parameter: 4.25 for MXFP4 and 4.5 for NVFP4.

See also