Skip to content

Quantization Guide

Model quantization is a technique that reduces model size and computational overhead by lowering the numerical precision of weights and activations, thereby saving memory and improving inference speed.

vLLM Ascend supports multiple quantization methods. This guide provides instructions for using different quantization tools and running quantized models on vLLM Ascend.

Note

You can choose to convert the model yourself or use the quantized model we uploaded. See https://www.modelscope.cn/models/vllm-ascend/Kimi-K2-Instruct-W8A8. Before you quantize a model, ensure sufficient RAM is available.

Quantization Tools

vLLM Ascend supports models quantized by two main tools: ModelSlim and LLM-Compressor.

ModelSlim is an Ascend-friendly compression tool focused on acceleration, using compression techniques, and built for Ascend hardware. It includes a series of inference optimization technologies such as quantization and compression, aiming to accelerate large language dense models, MoE models, multimodal understanding models, multimodal generation models, etc.

Installation

To use ModelSlim for model quantization, install it from its Git repository:

# Install 26.0.0 version, this is currently the latest stable branch
git clone https://gitcode.com/Ascend/msmodelslim.git -b 26.0.0

cd msmodelslim

bash install.sh

Model Quantization

The following example shows how to generate W8A8 quantized weights for the Qwen3-MoE model.

Quantization Script:

cd example/Qwen3-MOE

# Support multi-card quantization
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:False

# Set model and save paths
export MODEL_PATH="/path/to/your/model"
export SAVE_PATH="/path/to/your/quantized_model"

# Run quantization script
python3 quant_qwen_moe_w8a8.py --model_path $MODEL_PATH \
--save_path $SAVE_PATH \
--anti_dataset ../common/qwen3-moe_anti_prompt_50.json \
--calib_dataset ../common/qwen3-moe_calib_prompt_50.json \
--trust_remote_code True

After quantization completes, the output directory will contain the quantized model files.

For more examples, refer to the official examples.

2. LLM-Compressor

LLM-Compressor is a unified compressed model library for faster vLLM inference.

Installation

pip install llmcompressor

Model Quantization

LLM-Compressor provides various quantization scheme examples.

Dense Quantization

An example to generate W8A8 dynamic quantized weights for dense model:

# Navigate to LLM-Compressor examples directory
cd examples/quantization/llm-compressor

# Run quantization script
python3 w8a8_int8_dynamic.py
MoE Quantization

An example to generate W8A8 dynamic quantized weights for MoE model:

# Navigate to LLM-Compressor examples directory
cd examples/quantization/llm-compressor

# Run quantization script
python3 w8a8_int8_dynamic_moe.py

For more content, refer to the official examples.

The quantization types currently supported by LLM-Compressor can be viewed in the vllm_ascend/quantization/compressed_tensors_config.py file.

3. Native FP8 checkpoints (no quantization step needed)

Many models are now published directly in FP8, for example Qwen/Qwen3.8-27B-FP8 and zai-org/GLM-5.3-Flash. Their config.json carries "quant_method": "fp8" together with a weight_block_size, meaning every quantized weight is stored as float8_e4m3fn plus one float32 scale per weight block. vLLM Ascend detects and serves these checkpoints as published, so no offline re-quantization is required:

vllm serve /path/to/Qwen3.8-27B-FP8 --trust-remote-code
vllm serve /path/to/GLM-5.3-Flash --tensor-parallel-size 8 --trust-remote-code

How the weights are executed depends on the hardware:

  • Ascend 950: the block scales are re-grouped into MXFP8 at load time, so weights stay at one byte per element and the native FP8 matmul is used.
  • Other Ascend generations: the block scales are resolved into the model dtype at load time and served by the BF16 matmul. Numerically equivalent to the checkpoint, but plan for roughly twice the weight memory.

Layers the checkpoint left unquantized, such as visual.merger.* on multimodal models, are listed in ignored_layers or modules_to_not_convert and are served unquantized.

Only the block-wise flavour is supported. A native FP8 checkpoint without weight_block_size (per-tensor or per-channel scales) has no Ascend execution path yet; re-quantize it with ModelSlim or LLM-Compressor instead.

Running Quantized Models

Once you have a quantized model which is generated by ModelSlim, you can use vLLM Ascend for inference by specifying the --quantization ascend parameter to enable quantization features, while for models quantized by LLM-Compressor, it is not necessary to add this parameter.

Offline Inference

import torch

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The future of AI is",
]
# Set sampling parameters
sampling_params = SamplingParams(temperature=0.6, top_p=0.95, top_k=40)

llm = LLM(model="/path/to/your/quantized_model",
          max_model_len=4096,
          trust_remote_code=True,
          # Set appropriate TP and DP values
          tensor_parallel_size=2,
          data_parallel_size=1,
          # Set an unused port
          port=8000,
          # Set serving model name
          served_model_name="quantized_model",
          # Specify `quantization="ascend"` to enable quantization for models quantized by ModelSlim
          quantization="ascend")

outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Online Inference

# Corresponding to offline inference
python -m vllm.entrypoints.api_server \
    --model /path/to/your/quantized_model \
    --max-model-len 4096 \
    --port 8000 \
    --tensor-parallel-size 2 \
    --data-parallel-size 1 \
    --served-model-name quantized_model \
    --trust-remote-code

References