Methods and Schemes¶
This page defines the terms the quantization pages use and lists the names vLLM accepts for quantization methods and schemes.
The tables are generated at docs build time by
docs/mkdocs/gen_files/generate_quantization_tables.py from the registries in
vllm/config/quantization.py and vllm/model_executor/layers/quantization/
and from the platform allow-lists. A name added to a registry appears here on
the next build. Names registered by
plugins are not listed.
Terms¶
| Term | Meaning | Where it appears |
|---|---|---|
| Method | The implementation that loads a quantized model. Most methods read one checkpoint format and are named after the toolkit that writes it, e.g. compressed-tensors, modelopt, quark. |
quant_method in the model's config.json, or --quantization |
| Online quantization | Quantizing full-precision weights while the model loads. | --quantization <shorthand>, --quantization-config |
| Scheme | The numeric format of one tensor: its data format, its scale format, and what shares a scale. | The weight and activation fields of --quantization-config |
| Kernel | The code that runs a layer for a given weight and activation scheme. | --linear-backend, --moe-backend; see Kernel Backends |
A method names how a model is loaded; a scheme names the format of one tensor. One checkpoint can hold layers with different schemes, and kernels are selected per layer from the scheme.
W<n>A<m> gives the bits of weights and activations: W4A16 is 4-bit weights
with 16-bit activations, and W4A4 quantizes both. Weight-only schemes save
memory; the matrix multiply still runs in 16-bit unless activations are
quantized too.
Methods¶
These are the values of --quantization other than the
online shorthands. For a pre-quantized checkpoint vLLM
reads quant_method from config.json and selects the method, so the flag is
normally omitted. A method that does not correspond to a checkpoint format is
used only when it is passed explicitly.
| Names | Implementation | Platforms | Guide |
|---|---|---|---|
awq, awq_marlin, auto_awq |
AutoAWQConfig |
CUDA, ROCm, XPU, CPU | Guide |
fp8 |
Fp8Config |
CUDA, ROCm, XPU, CPU | – |
w4afp8 |
W4AFP8Config |
CUDA, CPU | – |
fbgemm_fp8 (deprecated) |
FBGEMMFp8Config |
CUDA, ROCm, CPU | – |
modelopt |
ModelOptFp8Config |
CUDA, ROCm, XPU, CPU | Guide |
modelopt_fp4 |
ModelOptNvFp4Config |
CUDA, ROCm, CPU | Guide |
modelopt_mxfp8, mxfp8 |
ModelOptMxFp8Config |
CUDA, ROCm, XPU, CPU | Guide |
modelopt_int8 |
ModelOptInt8Config |
CUDA, ROCm, CPU | Guide |
modelopt_mixed |
ModelOptMixedPrecisionConfig |
CUDA, ROCm, CPU | Guide |
auto_gptq, gptq, gptq_marlin |
AutoGPTQConfig |
CUDA, ROCm, XPU, CPU | Guide |
compressed-tensors |
CompressedTensorsConfig |
CUDA, ROCm, XPU, CPU | Guide |
experts_int8 |
ExpertsInt8Config |
CUDA, CPU | – |
quark |
QuarkConfig |
CUDA, ROCm, CPU | Guide |
moe_wna16 |
MoeWNA16Config |
CUDA, XPU, CPU | – |
torchao |
TorchAOConfig |
CUDA, ROCm, CPU | Guide |
inc |
INCConfig |
CUDA, ROCm, XPU, CPU | Guide |
mxfp4 |
Mxfp4Config |
CUDA, ROCm, XPU, CPU | – |
gpt_oss_mxfp4 |
GptOssMxfp4Config |
CUDA, ROCm, XPU, CPU | – |
deepseek_v4_fp8 |
DeepseekV4FP8Config |
CUDA, ROCm, XPU, CPU | – |
humming |
HummingConfig |
CUDA, CPU | – |
online |
OnlineQuantizationConfig |
CUDA, ROCm, XPU, CPU | Guide |
- Names on one row are aliases of one implementation.
- Platforms lists where vLLM accepts the name at startup. ROCm and XPU accept only the names on their allow-list; CUDA and CPU accept every name. An accepted method can still have no kernel for a given device; see Kernel Backends.
- A deprecated method also needs
--allow-deprecated-quantization.
Online shorthands¶
Passed to --quantization, a shorthand quantizes the weights of a checkpoint
that is not quantized. The columns show the weight scheme it applies to linear
layers and to MoE expert layers; – leaves that kind of layer unquantized.
| Name | Linear weights | MoE weights | Platforms |
|---|---|---|---|
fp8_per_tensor |
fp8_per_tensor_static |
fp8_per_tensor_static |
CUDA, ROCm, XPU, CPU |
fp8_per_block |
fp8_per_block_static |
fp8_per_block_static |
CUDA, ROCm, XPU, CPU |
fp8_per_channel |
fp8_per_channel_static |
fp8_per_channel_static |
CUDA, ROCm, XPU, CPU |
mxfp8 |
mxfp8 |
mxfp8 |
CUDA, ROCm, XPU, CPU |
mxfp4 |
mxfp4 |
mxfp4 |
CUDA, ROCm, XPU, CPU |
mxfp4_hadamard32 |
mxfp4 with a 32-wide Hadamard rotation |
– | CUDA, ROCm, CPU |
int8_per_channel_weight_only |
– | int8_per_channel_static |
CUDA, CPU |
nvfp4_per_token |
– | FP4 (E2M1), E4M3 per 16 elements, fp32 per tensor | CUDA, CPU |
nvfp4_weight_only |
FP4 (E2M1), E4M3 per 16 elements, fp32 per tensor | – | CUDA, CPU |
lut_b_moe |
– | lut_b |
CUDA, ROCm, CPU |
Also a method name: mxfp8, mxfp4. On a checkpoint that is already quantized, these select the method of that name instead.
- A shorthand is not a scheme, even when the two share a name: read the columns. A cell that is not a name is a scheme that has no name of its own and is reachable only through the shorthand.
- Shorthands set weights only. The activation scheme is the default of the method that implements the weight scheme, and it can depend on the kernel that is selected.
- The same names are accepted inside
--quantization-config, including on a checkpoint that is already quantized. See Online Quantization.
Scheme names¶
These names are accepted by the weight and activation fields of
--quantization-config:
vllm serve <model> --quantization online \
--quantization-config '{"moe":{"weight":"mxfp4","activation":"mxfp8"}}'
| Name | Data | Scale | Scale computed | Online weight for |
|---|---|---|---|---|
fp8_per_tensor_static |
FP8 (E4M3) | fp32 per tensor | at load | linear, MoE |
fp8_per_tensor_dynamic |
FP8 (E4M3) | fp32 per tensor | per forward pass | – |
fp8_per_token |
FP8 (E4M3) | fp32 per token | per forward pass | – |
fp8_per_channel_static |
FP8 (E4M3) | fp32 per output channel | at load | linear, MoE |
fp8_per_block_static |
FP8 (E4M3) | fp32 per 128×128 block | at load | linear, MoE |
fp8_per_block_dynamic |
FP8 (E4M3) | fp32 per 128 elements | per forward pass | – |
mxfp8 |
FP8 (E4M3) | E8M0 per 32 elements | at load (weight), per forward pass (activation) | linear, MoE |
mxfp4 |
FP4 (E2M1) | E8M0 per 32 elements | at load (weight), per forward pass (activation) | linear, MoE |
nvfp4_per_token |
FP4 (E2M1) | E4M3 per 16 elements, then fp32 per token (per forward pass) | per forward pass | – |
int8_per_channel_static |
INT8 | fp32 per output channel | at load | MoE |
lut_b |
uint8 |
E4M3 per 8×64 block | at load | MoE |
int8_per_group_32 |
INT8 | fp32 per 32 elements | per forward pass | – |
int4_per_group_32 |
INT4 | fp16 per 32 elements | at load | – |
int4_per_group_32_asym |
UINT4, asymmetric | fp16 per 32 elements | at load | lm_head |
- Data is the element format. ExMy is a float with x exponent and y mantissa bits: E4M3 is 8-bit and E2M1 is 4-bit, each with a sign bit. E8M0 is an unsigned 8-bit exponent, i.e. a power of two.
- Scale is the format of the scale factor and what shares one value. A second entry is an outer scale applied on top of the first.
- Scale computed separates static scales, which are stored in the checkpoint or computed once at load, from dynamic ones, which are computed from the tensor on every forward pass. Weights use static scales. Dynamic scales are how activations are quantized without calibration data.
- Online weight for lists the kinds of layer that online quantization can
quantize to this scheme.
lm_headmarks a scheme that applies only to the lm_head, selected withtargets. Every name parses on both fields, but a name with–here is rejected or ignored as an online weight, and only some methods accept anactivationoverride. See Online Quantization.
The table shows each scheme as vLLM's QuantKey describes it. Formats that are
not a data format plus scales are approximated: lut_b stores 3-bit indices
into a codebook, which appears here as the scale, and an E8M0 scale per 32
elements that the key does not describe.
MXFP8 and MXFP4 are the
OCP Microscaling
formats. With b bits per element and an s-bit scale shared by g elements,
a weight costs about b + s / g bits per parameter: 4.25 for MXFP4 and 4.5 for
NVFP4.
See also¶
- Kernel Backends: which kernel runs a layer, and how to pin one.
- Online Quantization: the
--quantization-configschema, per-layer targeting, and re-quantizing a quantized checkpoint. - Quantized KV Cache: a separate setting from weight quantization.