32B, W8A8, W4A8, W4A4, W8A8SC-310)¶

1 简介¶

Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models. Built upon extensive training, Qwen3 delivers groundbreaking advancements in reasoning, instruction-following, agent capabilities, and multilingual support. The Dense variants covered in this document include Qwen3-0.6B, 1.7B, 4B, 8B, 14B, and 32B, along with their quantized versions (W8A8, W4A8, W4A4, and W8A8SC-310) optimized for Ascend NPU deployment.

本文档将演示在 vLLM-Ascend 环境中 Qwen3 Dense 模型的主要验证步骤，包括支持的特性、环境准备、模型量化、单节点和多节点部署，以及精度和性能评估。通过定制服务级配置以适应不同的使用场景，您可以确保在各种场景下获得最佳性能。

The Qwen3 Dense models are first supported in v0.8.4rc2. W8A8 quantization was first supported in v0.8.4rc2, W4A8 quantization is supported since v0.9.1rc2, and W4A4 is supported since v0.11.0rc1. Atlas inference products use the W8A8SC-310 quantized weights listed in this tutorial. This document is validated and written based on vLLM-Ascend v0.21.0. All v0.21.0 and later versions can run stably. To use the latest features, it is recommended to use the latest release candidate or official version.

2 支持的特性¶

请参考支持特性列表了解模型支持矩阵。

请参考特性指南了解特性配置信息。

3 前提条件¶

3.1 模型权重¶

以下模型变体可用。建议将模型权重下载到所有节点均可访问的共享目录中。

BF16 版本：

模型	硬件要求	下载地址
Qwen3-0.6B	1 Atlas A3 inference products (64G × 16), 1 Atlas A2 inference products (64G × 8)	Download
Qwen3-1.7B	1 Atlas A3 inference products (64G × 16), 1 Atlas A2 inference products (64G × 8)	Download
Qwen3-4B	1 Atlas A3 inference products (64G × 16), 1 Atlas A2 inference products (64G × 8)	Download
Qwen3-8B	1 Atlas A3 inference products (64G × 16), 1 Atlas A2 inference products (64G × 8)	Download
Qwen3-14B	1 Atlas A3 inference products (64G × 16), 1 Atlas A2 inference products (64G × 8)	Download
Qwen3-32B	1 Atlas A3 inference products (64G × 16), 1 Atlas A2 inference products (64G × 8)	Download

Quantized Versions for Atlas A2/A3 inference products:

模型	量化方式	硬件要求	下载地址
Qwen3-8B-W4A8	W4A8	1 Atlas A3 inference products (64G × 16) or 1 Atlas A2 inference products (64G × 8)	Download
Qwen3-32B-W4A4	W4A4	1 Atlas A3 inference products (64G × 16) or 1 Atlas A2 inference products (64G × 8)	Download
Qwen3-32B-W8A8	W8A8	1 Atlas A3 inference products (64G × 16) or 1 Atlas A2 inference products (64G × 8)	Download

Quantized Versions for Atlas inference products:

模型	量化方式	硬件要求	下载地址
Qwen3-8B-W8A8SC	W8A8SC	Atlas inference products (TP1)	Download
Qwen3-14B-W8A8SC	W8A8SC	Atlas inference products (TP1)	Download
Qwen3-32B-W8A8SC	W8A8SC	Atlas inference products (TP4)	Download

这些是推荐的卡数，可根据实际情况进行调整。

3.2 验证多节点通信¶

如果需要部署多节点环境，请按照验证多节点通信环境验证多节点通信。

4 安装¶

4.1 Docker 镜像安装¶

您可以使用官方提供的 Qwen3 Dense 模型一体化 Docker 镜像。

Docker 拉取：

docker pull quay.io/ascend/vllm-ascend:v0.22.1rc1

Docker 运行：

在每个节点上启动 Docker 镜像。

Atlas A3 inference productsAtlas A2 inference productsAtlas inference products

export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1-a3

docker run --rm \
    --name vllm-ascend-env \
    --shm-size=1g \
    --net=host \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci8 \
    --device /dev/davinci9 \
    --device /dev/davinci10 \
    --device /dev/davinci11 \
    --device /dev/davinci12 \
    --device /dev/davinci13 \
    --device /dev/davinci14 \
    --device /dev/davinci15 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /root/.cache:/root/.cache \
    -it $IMAGE bash

Note

Atlas A3 inference products have 8 NPUs with dual-die design (16 chips total: /dev/davinci[0-15]). If you are on a shared machine, map only the chips you need (e.g., /dev/davinci[0-7] for NPU 0-3).

export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1

docker run --rm \
    --name vllm-ascend-env \
    --shm-size=1g \
    --net=host \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /root/.cache:/root/.cache \
    -it $IMAGE bash

# Use the vllm-ascend image
export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1-310p

docker run --rm \
--name vllm-ascend \
--shm-size=10g \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
--device /dev/davinci3 \
--device /dev/davinci4 \
--device /dev/davinci5 \
--device /dev/davinci6 \
--device /dev/davinci7 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-p 8080:8080 \
-it $IMAGE bash

默认工作目录是 /workspace。vLLM 和 vLLM-Ascend 作为 Python 包安装在 site-packages 中。

安装验证：启动容器后，运行以下命令验证安装：

docker ps | grep vllm-ascend-env

预期结果：容器已列出，状态为 Up。您还可以在容器内验证 vllm-ascend 版本：

pip show vllm-ascend

预期结果：显示版本信息，与拉取的镜像版本匹配。

4.2 源码安装¶

如果您希望从源码构建而不是使用 Docker 镜像，请按照安装指南安装 vLLM-Ascend。

Note

For Atlas inference products, source installation may pull in triton and triton-ascend. Uninstall them before running vLLM-Ascend on Atlas inference products:

pip uninstall -y triton-ascend triton

验证源码安装：

pip show vllm-ascend

预期结果：显示版本信息，确认安装成功。

Note

如果部署多节点环境，请在每个节点上设置环境。

5 在线服务部署¶

5.1 单节点在线部署¶

单节点部署在同一节点内完成 Prefill 和 Decode，适用于开发、测试以及中小规模推理场景。

启动服务：

以下命令为示例配置，请根据实际场景调整参数。

Atlas A2 inference products / Atlas A3 inference productsAtlas inference products

Qwen3-32B-W8A8：

export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1
export HCCL_OP_EXPANSION_MODE="AIV"

vllm serve your_model_path \
    --served-model-name qwen3 \
    --trust-remote-code \
    --async-scheduling \
    --quantization ascend \
    --distributed-executor-backend mp \
    --tensor-parallel-size 4 \
    --max-model-len 5500 \
    --max-num-batched-tokens 40960 \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --port <port> \
    --gpu-memory-utilization 0.9 \
    --additional-config '{"enable_flashcomm1": true}'

Qwen3-32B-W4A4：

export ASCEND_RT_VISIBLE_DEVICES=0,1
export VLLM_USE_V1=1
export TASK_QUEUE_ENABLE=1
export HCCL_BUFFSIZE=1024
vllm serve your_model_path \
    --port 8004 \
    --data-parallel-size 1 \
    --tensor-parallel-size 2 \
    --served-model-name qwen3 \
    --distributed_executor_backend "mp" \
    --max-model-len 40960 \
    --max-num-batched-tokens 16384 \
    --max-num-seqs 64 \
    --trust-remote-code \
    --gpu-memory-utilization 0.9 \
    --quantization ascend \
    --compilation-config '{"cudagraph_capture_sizes": [64]}' \
    --additional-config '{"enable_flashcomm1": true, "ascend_compilation_config": {"fuse_norm_quant": false}}'

Qwen3-8B-W4A8:

export ASCEND_RT_VISIBLE_DEVICES=0,1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
vllm serve your_model_path \
    --served-model-name qwen3 \
    --max-model-len 4096 \
    --port 20001 \
    --additional-config '{"ascend_compilation_config": {"enable_npugraph_ex": false}}' \
    --quantization ascend

Atlas inference products use the W8A8SC-310 quantized weights from the Eco-Tech official ModelScope repository. Keep an explicit --max-model-len for 310P deployment to avoid OOM caused by oversized attention mask allocation.

Qwen3-8B-W8A8SC:

export VLLM_USE_MODELSCOPE=True
export ASCEND_RT_VISIBLE_DEVICES=0

vllm serve Eco-Tech/Qwen3-8B-w8a8sc-310-vllm/TP1/Qwen3-8B-w8a8sc-310-vllm-tp1 \
    --host 127.0.0.1 \
    --port 8080 \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.90 \
    --max-num-seqs 32 \
    --served-model-name qwen3 \
    --dtype float16 \
    --additional-config '{"ascend_compilation_config": {"fuse_norm_quant": false}}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [1,2,4,8,16,32]}' \
    --quantization ascend \
    --max-model-len 16384 \
    --no-enable-prefix-caching \
    --load-format sharded_state

Qwen3-14B-W8A8SC:

export VLLM_USE_MODELSCOPE=True
export ASCEND_RT_VISIBLE_DEVICES=0

vllm serve Eco-Tech/Qwen3-14B-w8a8sc-310-vllm/TP1/Qwen3-14B-w8a8sc-310-vllm-tp1 \
    --host 127.0.0.1 \
    --port 8080 \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.90 \
    --max-num-seqs 16 \
    --served-model-name qwen3 \
    --dtype float16 \
    --additional-config '{"ascend_compilation_config": {"fuse_norm_quant": false}}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [1,2,4,8,16]}' \
    --quantization ascend \
    --max-model-len 16384 \
    --no-enable-prefix-caching \
    --load-format sharded_state

Qwen3-32B-W8A8SC:

export VLLM_USE_MODELSCOPE=True
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3

vllm serve Eco-Tech/Qwen3-32B-w8a8sc-310-vllm/TP4/Qwen3-32B-w8a8sc-310-vllm-tp4 \
    --host 127.0.0.1 \
    --port 8080 \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.90 \
    --max-num-seqs 32 \
    --served-model-name qwen3 \
    --dtype float16 \
    --additional-config '{"ascend_compilation_config": {"fuse_norm_quant": false}}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [16,32]}' \
    --quantization ascend \
    --max-model-len 20480 \
    --no-enable-prefix-caching \
    --load-format sharded_state

Note

vLLM 服务参数文档 — vLLM serve 命令的额外参数详情。
环境变量 — Ascend 特定的环境变量（HCCL_* 等）。

服务验证：

如果服务启动成功，将显示以下启动日志：

(APIServer pid=<pid>) INFO:     Started server process [<pid>]
(APIServer pid=<pid>) INFO:     Waiting for application startup.
(APIServer pid=<pid>) INFO:     Application startup complete.

6 功能验证¶

服务启动后，可以通过发送提示词来调用模型。

聊天补全 API：

curl http://localhost:<port>/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "qwen3",
        "messages": [
            {"role": "user", "content": "Give me a short introduction to large language models."}
        ],
        "temperature": 0.6,
        "top_p": 0.95,
        "top_k": 20,
        "max_completion_tokens": 4096
    }'

预期结果：HTTP 200，返回包含 choices 字段（含生成文本）的 JSON 响应。

7 精度评估¶

使用 AISBench¶

有关安装、数据集下载和配置等设置详情，请参考使用 AISBench。

以下是精度评估配置文件的示例配置：

精度评估配置文件：

# Example configuration: benchmarks/ais_bench/benchmark/configs/models/vllm_api/vllm_api_general_chat.py
from ais_bench.benchmark.models import VLLMCustomAPIChat
from ais_bench.benchmark.utils.model_postprocessors import extract_non_reasoning_content

models = [
    dict(
        attr="service",
        type=VLLMCustomAPIChat,
        abbr='vllm-api-general-chat',
        path="your_model_path",
        model="qwen3",
        request_rate = 0,
        retry = 2,
        host_ip = "127.0.0.1",
        host_port = 2001,
        max_out_len = 32768,
        batch_size = 32,
        trust_remote_code=False,
        generation_kwargs = dict(
            temperature = 0.6,
            top_k = 20,
            top_p = 0.95,
        ),
        pred_postprocessor=dict(type=extract_non_reasoning_content)
    )
]

以 aime2025 数据集为例运行精度评估：

ais_bench --models vllm_api_general_chat --datasets aime2025_gen_0_shot_chat_prompt --debug

--models 参数值对应上述配置文件中的 abbr 字段。请根据实际场景调整 max_out_len、batch_size 和数据集任务。

8 性能评估¶

使用 AISBench¶

有关安装、数据集下载和配置等设置详情，请参考使用 AISBench 了解详细信息。

以下是精度评估配置文件的示例配置：

# Example configuration: benchmarks/ais_bench/benchmark/configs/models/vllm_api/vllm_api_stream_chat.py
from ais_bench.benchmark.models import VLLMCustomAPIChat
from ais_bench.benchmark.utils.postprocess.model_postprocessors import extract_non_reasoning_content

models = [
    dict(
        attr="service",
        type=VLLMCustomAPIChat,
        abbr="vllm-api-stream-chat",
        path="your_model_path",
        model="qwen3",
        stream=True,
        request_rate=0,
        use_timestamp=False,
        retry=2,
        host_ip="127.0.0.1",
        host_port=8004,
        max_out_len=1500,
        batch_size=90,
        trust_remote_code=False,
        generation_kwargs=dict(
            temperature=0,
            ignore_eos = True
        ),
    )
]

以 GSM8K 数据集为例运行性能评估：

ais_bench --models vllm_api_stream_chat --datasets gsm8k_gen_0_shot_cot_str_perf --debug --summarizer default_perf --mode perf --num-prompts 360

使用 vLLM 基准测试¶

更多详情请参考 vLLM 基准测试。

有三个 vllm bench 子命令：

latency：对单批请求的延迟进行基准测试。
serve：对在线服务吞吐量进行基准测试。
throughput：对离线推理吞吐量进行基准测试。

以 serve 为例：

vllm bench serve \
    --model your_model_path \
    --served-model-name qwen3 \
    --port <port> \
    --dataset-name random \
    --random-input 200 \
    --num-prompts 200 \
    --request-rate 1 \
    --save-result \
    --result-dir ./

几分钟后，您将获得性能评估结果。

9 性能调优¶

9.1 推荐配置¶

注意：以下配置在特定测试环境中验证，仅供参考。最佳配置取决于最大输入/输出长度、前缀缓存命中率、精度要求和部署机器比例等因素。建议参考第 9.2 节根据实际情况进行调优。

表 1：场景概览¶

场景	部署模式	*NPU总数	权重版本	关键考量
High Throughput	Single-Node (TP4)	4 (Atlas A3 inference products)	W8A8	4-card TP maximizes concurrent request processing
Long Context	Single-Node (TP4)	4 (Atlas A3 inference products)	W8A8	4-card TP extends context window for long sequences
Low Latency	Single-Node (TP8)	8 (Atlas A3 inference products)	W8A8	8-card TP reduces per-token latency for interactive responses

*Total NPUs 表示所有节点使用的 NPU 总数。

表 2：详细节点配置¶

场景	配置	NPU数量	TP	DP	FUSED_MC2	EP开关	异步调度
High Throughput	Single-Node	4	4	1	Off	Off	On
Long Context	Single-Node	4	4	1	Off	Off	On
Low Latency	Single-Node	8	8	1	Off	Off	On

有关详细的参数说明，请参考第 5 节中的部署示例

高吞吐配置：

export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1
export HCCL_OP_EXPANSION_MODE="AIV"

vllm serve your_model_path \
    --served-model-name qwen3 \
    --trust-remote-code \
    --distributed-executor-backend mp \
    --tensor-parallel-size 4 \
    --max-model-len 5500 \
    --max-num-batched-tokens 40960 \
    --no-enable-prefix-caching \
    --async-scheduling \
    --quantization ascend \
    --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[4,8,64,72,76,80,96,100,120,140,144,160,192,216,240,252,288,320,336,360,384,400,408,416,420,432,480,540,576,600]}' \
    --additional-config '{"weight_prefetch_config":{"enabled":true}, "enable_flashcomm1": true}' \
    --host <host_ip> \
    --port <port> \
    --block-size 128 \
    --gpu-memory-utilization 0.9

长上下文配置：

export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1
export HCCL_OP_EXPANSION_MODE="AIV"

vllm serve your_model_path \
    --host <host_ip> \
    --port <port> \
    --served-model-name qwen3 \
    --trust-remote-code \
    --seed 1024 \
    --max-model-len 135000 \
    --max-num-batched-tokens 40960 \
    --tensor-parallel-size 4 \
    --distributed-executor-backend "mp" \
    --async-scheduling \
    --no-enable-prefix-caching \
    --speculative-config '{"method": "eagle3", "model":"your_eagle3_model_path", "num_speculative_tokens": 3}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --hf-overrides '{"rope_parameters": {"rope_type":"yarn","rope_theta":1000000,"factor":4,"original_max_position_embeddings":131072}}' \
    --gpu-memory-utilization 0.9 \
    --quantization ascend \
    --additional-config '{"enable_flashcomm1": true}'

低延迟配置：

export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1
export HCCL_OP_EXPANSION_MODE="AIV"

vllm serve your_model_path \
    --served-model-name qwen3 \
    --trust-remote-code \
    --distributed-executor-backend mp \
    --tensor-parallel-size 8 \
    --max-model-len 5500 \
    --max-num-batched-tokens 40960 \
    --no-enable-prefix-caching \
    --async-scheduling \
    --quantization ascend \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY","cudagraph_capture_sizes":[1,2,4,8,16,32,64,72,76,80,96,100,120,140,144,160,192,216,240,252,288,320,336,360,384,400,408,416,420,432,480,540,576,600]}' \
    --speculative-config '{"method": "eagle3", "model":"your_eagle3_model_path", "enforce_eager": true, "num_speculative_tokens": 3}' \
    --port <port> \
    --block-size 128 \
    --gpu-memory-utilization 0.9

9.2 调优指南¶

9.2.1 通用调优参考¶

请参考公共性能调优文档了解调优方法。请参考特性指南了解详细的特性说明。

10 常见问题¶

有关常见的环境、安装和通用参数问题，请参考 vLLM-Ascend 常见问题。本节仅涵盖 Qwen3 Dense 模型的特定问题。