跳转至

Qwen3.5-27B 与 Qwen3.6-27B

1 简介

Note

本文档中提到的“PD 分离部署”仅作为示例/参考设计提供，用于演示以分离模式部署 Prefill 和 Decode 的基本流程。不建议在生产环境中使用。

对于客户侧的生产环境，推荐使用 MindIE Motor + vLLM-Ascend。

Qwen3.5-27B 和 Qwen3.6-27B 是 Qwen3.5/Qwen3.6 系列中的密集混合 Mamba-Transformer 语言模型，融合了架构效率、强化学习规模和全球可访问性方面的突破。它们共享相同的混合注意力设计（GDN + 全注意力），因此在 Ascend NPU 上的部署对两个模型遵循相同的模式。它们适用于在 Ascend NPU 上运行的通用文本生成任务，例如对话、内容创作和代码生成。

本文档将演示模型的主要验证步骤，包括支持的特性、特性配置、环境准备、单节点和多节点部署，以及精度和性能评估。

强烈建议使用 vllm-ascend 的最新候选发布（rc）版本或最新正式版本。作为最低版本要求，Qwen3.5-27B 首次在 vllm-ascend:v0.17.0rc1 中支持，Qwen3.6-27B 首次在 vllm-ascend:v0.18.0rc1 中支持。对 Atlas 300I DUO 和 Ascend 950DT系列产品的支持从 vllm-ascend:v0.23.0rc1 开始。

2 支持的特性

请参考支持的特性列表了解模型支持矩阵。

请参考特性指南了解特性配置信息。

3 前提条件

3.1 模型权重

权重版本 硬件要求 下载链接
Qwen3.5-27B (BF16 version) 1 Atlas 800I A3 (64GB × 16) node or 1 Atlas 800I A2 (64GB × 8) node or Atlas 300I DUO ModelScope | Hugging Face
Qwen3.5-27B-w8a8 (Quantized version) 1 Atlas 800I A3 (64GB × 16) node or 1 Atlas 800I A2 (64GB × 8) node or Atlas 300I DUO ModelScope
Qwen3.6-27B (BF16 version) 1 950DT Products(96GB × 8) node or 1 Atlas 800I A3 (64GB × 16) node or 1 Atlas 800I A2 (64GB × 8) node or Atlas 300I DUO ModelScope | Hugging Face
Qwen3.6-27B-w8a8 (Quantized version) 1 Atlas 800I A3 (64GB × 16) node or 1 Atlas 800I A2 (64GB × 8) node ModelScope
Qwen3.6-27B-w8a8-mxfp8（量化版本） 1 个 Ascend 950DT系列产品（96GB × 8）节点 ModelScope
Qwen3.6-27B-w8a8c8-mxfp8 (Quantized version) 1 950DT Products (96GB × 8) node ModelScope
Qwen3.6-27B-w8a8-310p（量化版本） 1 个 Atlas 300I DUO ModelScope
  • Qwen3.6-27B-w8a8c8-mxfp8 (W8A8 MXFP8 quantization with C8 MXFP8 KV cache): serve it with --kv-cache-dtype mxfp8; recommended for long-sequence scenarios.

建议将模型权重下载到多节点的共享目录中，例如 /root/.cache/。

路径说明：将模型权重下载到您选择的目录并记录该目录。确保后续部署命令中的模型路径与该目录一致。

3.2 验证多节点通信

如果您想部署多节点环境，需要按照验证多节点通信环境来验证多节点通信。

4 安装

4.1 Docker 镜像安装

根据您的机器类型选择镜像并在节点上启动docker镜像，请参考使用docker。

It is recommended to use the latest release candidate (rc) version or the latest official version of the vllm-ascend image to ensure the best compatibility and access to the latest features. As a minimum-version requirement, use vllm-ascend:v0.17.0rc1 (or a later version) for Qwen3.5-27B, and vllm-ascend:v0.18.0rc1 (or a later version) for Qwen3.6-27B. For Qwen3.6-27B on Atlas 800I A3, please use the matching v0.18.0rc1-a3 (or a later -a3) image. For Atlas 300I DUO, use vllm-ascend:v0.23.0rc1-310p (or a later -310p) image.

在每个节点上启动 docker 镜像。

export IMAGE=quay.io/ascend/vllm-ascend:v0.23.0-a3
docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --net=host \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci8 \
    --device /dev/davinci9 \
    --device /dev/davinci10 \
    --device /dev/davinci11 \
    --device /dev/davinci12 \
    --device /dev/davinci13 \
    --device /dev/davinci14 \
    --device /dev/davinci15 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /root/.cache:/root/.cache \
    -it $IMAGE bash

在每个节点上启动 docker 镜像。

export IMAGE=quay.io/ascend/vllm-ascend:v0.23.0
docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --net=host \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /root/.cache:/root/.cache \
    -it $IMAGE bash

在每个节点上启动 docker 镜像。

export IMAGE=quay.io/ascend/vllm-ascend:v0.23.0-310p
docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /root/.cache:/root/.cache \
    -p 8000:8000 \
    -it $IMAGE bash

在每个节点上启动 docker 镜像。

export IMAGE=quay.io/ascend/vllm-ascend:v0.23.0-a5
export NAME=vllm-ascend

docker run --rm \
    --name $NAME \
    --net=host \
    --shm-size=1g \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci_manager \
    --device /dev/hisi_hdc \
    --device /dev/ummu \
    --device /dev/uburma \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /etc/hccl_rootinfo.json:/etc/hccl_rootinfo.json \
    -v /etc/hixlep/:/etc/hixlep/ \
    -v /root/.cache:/root/.cache \
    -v /usr/local/sbin:/usr/local/sbin \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
    -v /usr/lib64:/usr/lib64 \
    -it $IMAGE bash

docker 运行成功后，可以通过执行 docker ps 命令验证正在运行的容器服务。预期结果是容器 vllm-ascend 的状态为 Up，确认 docker 安装成功。

4.2 源码安装

如果您不想像上面那样使用 docker 镜像，也可以从源码构建所有内容：

  1. 克隆仓库并从源码安装 vllm-ascend：

    git clone https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
pip install -e .

    完整的安装步骤请参考安装。

    Note

    在 Atlas 300I DUO 上，您可能需要卸载 triton-ascend 和 triton 以避免依赖冲突：

    pip uninstall -y triton-ascend triton

  2. 如果要部署多节点环境，需要在每个节点上设置环境。

要验证源码安装，请运行以下命令并确认显示的版本与您安装的版本一致：

pip show vllm-ascend

预期结果：显示 vllm-ascend 的版本信息，确认安装成功。

5 在线服务部署

5.1 单节点在线部署

Single-node deployment completes both Prefill and Decode within the same node, suitable for development, testing, and medium-scale inference scenarios. The Qwen3.5-27B, Qwen3.5-27B-w8a8, Qwen3.6-27B, and Qwen3.6-27B-w8a8 models can all be deployed on 1 Atlas 800I A3 (64GB × 16), 1 Atlas 800I A2 (64GB × 8). On Atlas 300I DUO, at least 2 devices are required. The Qwen3.6-27B, Qwen3.6-27B-w8a8-mxfp8 and Qwen3.6-27B-w8a8c8-mxfp8 models can all be deployed on 1 950DT Products (96GB × 8). The quantized versions need to start with the --quantization ascend parameter.

Qwen3.5-27B 和 Qwen3.6-27B 共享相同的 MTP 头部设计，因此 qwen3_5_mtp 推测解码方法可以用于两者。

The following examples are for Atlas 800I A3 / Atlas 800I A2. Quantized versions need --quantization ascend.

启动命令：

#!/bin/sh
# Load model from ModelScope to speed up download
export MODEL_PATH=Eco-Tech/Qwen3.5-27B-w8a8-mtp
export VLLM_USE_MODELSCOPE=True
export HCCL_BUFFSIZE=512
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl kernel.sched_migration_cost_ns=50000
# To reduce memory fragmentation and avoid out of memory
# Size of the shared buffer (in MB) used by HCCL for NPU-to-NPU collective communication

# Model weight path; can be a ModelScope model id (e.g., Eco-Tech/Qwen3.5-27B-w8a8-mtp) or a local directory path
# Ensure the model path matches the directory recorded during download

vllm serve $MODEL_PATH \
    --host 0.0.0.0 \
    --port 8000 \
    --data-parallel-size 1 \
    --tensor-parallel-size 2 \
    --quantization ascend \
    --served-model-name qwen3.5 \
    --max-num-seqs 32 \
    --max-model-len 133120 \
    --max-num-batched-tokens 16384 \
    --trust-remote-code \
    --allowed-local-media-path / \
    --dtype bfloat16 \
    --reasoning-parser qwen3 \
    --mm-processor-cache-gb 0 \
    --mm-encoder-tp-mode data \
    --mm-processor-cache-type shm \
    --gpu-memory-utilization 0.85 \
    --speculative-config '{"method": "qwen3_5_mtp", "num_speculative_tokens": 3}' \
    --compilation-config '{"cudagraph_capture_sizes": [1,4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,100,104,108,112,116,120,124,128],"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --additional-config '{"enable_cpu_binding":true}'

启动命令：

#!/bin/sh
# Load model from ModelScope to speed up download
export MODEL_PATH=Eco-Tech/Qwen3.6-27B-w8a8
export VLLM_USE_MODELSCOPE=True
export HCCL_BUFFSIZE=512
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl kernel.sched_migration_cost_ns=50000
# To reduce memory fragmentation and avoid out of memory
# Size of the shared buffer (in MB) used by HCCL for NPU-to-NPU collective communication

# Model weight path; can be a ModelScope model id (e.g., Eco-Tech/Qwen3.6-27B-w8a8) or a local directory path
# Ensure the model path matches the directory recorded during download

vllm serve $MODEL_PATH \
    --host 0.0.0.0 \
    --port 8000 \
    --data-parallel-size 1 \
    --tensor-parallel-size 2 \
    --quantization ascend \
    --served-model-name qwen3.6 \
    --max-num-seqs 32 \
    --max-model-len 133120 \
    --max-num-batched-tokens 16384 \
    --trust-remote-code \
    --allowed-local-media-path / \
    --dtype bfloat16 \
    --reasoning-parser qwen3 \
    --mm-processor-cache-gb 0 \
    --mm-encoder-tp-mode data \
    --mm-processor-cache-type shm \
    --gpu-memory-utilization 0.85 \
    --speculative-config '{"method": "qwen3_5_mtp", "num_speculative_tokens": 3}' \
    --compilation-config '{"cudagraph_capture_sizes": [1,4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,100,104,108,112,116,120,124,128],"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --additional-config '{"enable_cpu_binding":true}'

关键参数说明：

  • --data-parallel-size 1 and --tensor-parallel-size 2 are common settings for data parallelism (DP) and tensor parallelism (TP) sizes.
  • --max-model-len represents the context length, which is the maximum value of the input plus output for a single request. The Qwen3.6-27B model supports up to 262144.
  • --max-num-seqs indicates the maximum number of requests that each DP group is allowed to process. If the number of requests sent to the service exceeds this limit, the excess requests will remain in a waiting state and will not be scheduled. Note that the time spent in the waiting state is also counted in metrics such as TTFT and TPOT. Therefore, when testing performance, it is generally recommended that --max-num-seqs * --data-parallel-size >= the actual total concurrency.
  • --max-num-batched-tokens represents the maximum number of tokens that the model can process in a single step. Currently, vLLM v1 scheduling enables ChunkPrefill/SplitFuse by default, which means:
    • (1) If the input length of a request is greater than --max-num-batched-tokens, it will be divided into multiple rounds of computation according to --max-num-batched-tokens;
    • (2) Decode requests are prioritized for scheduling, and prefill requests are scheduled only if there is available capacity.
    • Generally, if --max-num-batched-tokens is set to a larger value, the overall latency will be lower, but the pressure on HBM memory (activation value usage) will be greater.
  • --gpu-memory-utilization represents the proportion of HBM that vLLM will use for actual inference. Its essential function is to calculate the available kv_cache size. During the warm-up phase (referred to as profile run in vLLM), vLLM records the peak HBM memory usage during an inference process with an input size of --max-num-batched-tokens. The available kv_cache size is then calculated as: --gpu-memory-utilization * HBM size - peak HBM memory usage. Therefore, the larger the value of --gpu-memory-utilization, the more kv_cache can be used. However, since the HBM memory usage during the warm-up phase may differ from that during actual inference (e.g., due to uneven EP load), setting --gpu-memory-utilization too high may lead to OOM (Out of Memory) issues during actual inference. The default value is 0.9.
  • Prefix caching is enabled by default, and the example above uses the default. Add --no-enable-prefix-caching to disable it if needed. For workloads with long shared prefixes, finer-grained hybrid prefix matching can be enabled with --enable-prefix-caching --prefix-match-unit 16. The match unit must divide every KV cache group's block size. A smaller unit can reuse more of a partial physical page, at the cost of additional hash metadata, copy-on-write operations, and an extra prefill split at the final matching boundary.
  • --quantization ascend indicates that quantization is used. To disable quantization, remove this option.
  • --speculative-config uses qwen3_5_mtp for both Qwen3.5-27B and Qwen3.6-27B because they share the same MTP head design.
  • --compilation-config contains configurations related to the aclgraph graph mode. The most significant configurations are "cudagraph_mode" and "cudagraph_capture_sizes", which have the following meanings:
    • "cudagraph_mode": represents the specific graph mode. Currently, "PIECEWISE" and "FULL_DECODE_ONLY" are supported. The graph mode is mainly used to reduce the cost of operator dispatch. Currently, "FULL_DECODE_ONLY" is recommended.
    • "cudagraph_capture_sizes": specifies the batch size tiers at which graphs are captured. The default value is [1, 2, 4, 8, 16, 24, 32, 40,..., --max-num-seqs]. In the graph mode, the batch size of each captured graph is fixed, and an actual batch size between two adjacent capture sizes is automatically padded up to the next capture size. Currently, the default setting is recommended. Only in some scenarios is it necessary to set this separately to achieve optimal performance.
  • --allowed-local-media-path allows the API server to read local images or videos from the specified directory when processing multimodal inputs. In this example, / whitelists the entire file system. This is security-sensitive and should only be enabled in trusted environments.
  • --dtype bfloat16 sets the data type for model weights and activations. bfloat16 is used on Atlas 800I A3 / Atlas 800I A2; float16 is required on Atlas 300I DUO instead.
  • --reasoning-parser qwen3 enables the reasoning output parser for the OpenAI-compatible API. It splits the model's thinking content from the final answer and returns it in the reasoning_content field of chat.completions responses.
  • --mm-processor-cache-gb sets the size (in GiB) of the multimodal processor cache, which is used to avoid re-processing past multimodal inputs. The cache is duplicated in each API server process and engine core process, so the total host memory usage is the cache size multiplied by the number of these processes. A value of 0 disables this cache completely.
  • --mm-encoder-tp-mode indicates how to optimize multimodal encoder inference using tensor parallelism (TP). data splits the batched encoder input data across TP ranks while each rank hosts the full encoder weights, and is recommended when multimodal inputs are used. The default mode weights splits the encoder weights across TP ranks instead.
  • --mm-processor-cache-type sets the cache type used by the multimodal preprocessor and mapper. shm uses a shared-memory FIFO cache, while the default lru uses a mirrored LRU cache.
  • --additional-config contains Ascend-specific optimization options:
    • "enable_cpu_binding": true enables Ascend-native CPU binding, which pins the worker process, key runtime threads, memory pages, and NPU IRQs to CPU ranges close to the driven NPU, reducing cross-NUMA traffic and thread preemption and improving latency stability. It is enabled by default starting from vllm-ascend v0.18.0rc1.

目前仅支持 TP 场景。请根据可用设备选择 TP=2 或 TP=4。将 MODEL_PATH 替换为 ModelScope 模型 ID 或本地目录路径。量化版本需要使用 --quantization ascend 参数启动。

启动命令：

#!/bin/sh
# Load model from ModelScope to speed up download
export MODEL_PATH=Eco-Tech/Qwen3.5-27B-w8a8-mtp
export VLLM_USE_MODELSCOPE=True

# Model weight path; can be a ModelScope model id (e.g., Eco-Tech/Qwen3.5-27B-w8a8-mtp) or a local directory path
# Ensure the model path matches the directory recorded during download

vllm serve $MODEL_PATH \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 4 \
    --served-model-name qwen3.5 \
    --max-num-seqs 128 \
    --max-model-len 16384 \
    --trust-remote-code \
    --gpu-memory-utilization 0.90 \
    --mamba-ssm-cache-dtype float16 \
    --dtype float16 \
    --speculative-config '{"method": "qwen3_5_mtp","num_speculative_tokens":1}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [2,16]}' \
    --additional-config '{"ascend_compilation_config": {"enable_npugraph_ex": false}}'

启动命令：

#!/bin/sh
# Load model from ModelScope to speed up download
export MODEL_PATH=Eco-Tech/Qwen3.6-27B-w8a8-310p
export VLLM_USE_MODELSCOPE=True

# Model weight path; can be a ModelScope model id (e.g., Eco-Tech/Qwen3.6-27B-w8a8) or a local directory path
# Ensure the model path matches the directory recorded during download

vllm serve $MODEL_PATH \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 4 \
    --served-model-name qwen3.6 \
    --max-num-seqs 128 \
    --max-model-len 16384 \
    --trust-remote-code \
    --gpu-memory-utilization 0.90 \
    --mamba-ssm-cache-dtype float16 \
    --dtype float16 \
    --speculative-config '{"method": "qwen3_5_mtp","num_speculative_tokens":1}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [2,16]}' \
    --additional-config '{"ascend_compilation_config": {"enable_npugraph_ex": false}}'

关键参数说明：

  • --tensor-parallel-size sets the tensor parallel size. Choose TP=2 or TP=4 according to the available devices.
  • --max-model-len represents the context length, which is the maximum value of the input plus output for a single request. Configure it based on the actual workload and available memory; The Qwen3.6-27B model supports up to 262144.
  • --max-num-seqs indicates the maximum number of concurrent requests.
  • --gpu-memory-utilization represents the proportion of HBM that vLLM will use for actual inference. Configure this value according to the actual device memory; setting it too high may cause OOM. The default value is 0.9.
  • --mamba-ssm-cache-dtype sets the data type of the Mamba SSM cache. On Atlas 300I DUO, only float16 is supported.
  • --dtype float16 must be set on Atlas 300I DUO. These devices only support the FP16 data type.
  • --speculative-config uses qwen3_5_mtp for both Qwen3.5-27B and Qwen3.6-27B because they share the same MTP head design. On Atlas 300I DUO, it is recommended to set num_speculative_tokens to 1.
  • --compilation-config contains configurations related to the aclgraph graph mode. The most significant configurations are "cudagraph_mode" and "cudagraph_capture_sizes", which have the following meanings:
    • "cudagraph_mode": represents the specific graph mode. Currently, "PIECEWISE" and "FULL_DECODE_ONLY" are supported. The graph mode is mainly used to reduce the cost of operator dispatch. Currently, "FULL_DECODE_ONLY" is recommended.
    • "cudagraph_capture_sizes": specifies the batch size tiers at which graphs are captured. When tensor parallelism (TP) is enabled, hardware event-id constraints allow at most two capture sizes (for example, [1, 8]). With MTP enabled, calculate each capture size as n * (num_speculative_tokens + 1), where n is a capture size for the deployment without MTP. For example, when num_speculative_tokens is 1, the non-MTP sizes [1,2,4,8] become [2,4,8,16].
  • --additional-config with "ascend_compilation_config": {"enable_npugraph_ex": false} is required on Atlas 300I DUO because enable_npugraph_ex is not supported on this platform.

以下示例适用于 Ascend 950DT系列产品。量化版本需要 --quantization ascend。

启动命令：

#!/bin/sh
# Load model from ModelScope to speed up download
export MODEL_PATH=Eco-Tech/Qwen3.6-27B-w8a8c8-mxfp8
export VLLM_USE_MODELSCOPE=True
export HCCL_BUFFSIZE=1024
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl kernel.sched_migration_cost_ns=50000
# To reduce memory fragmentation and avoid out of memory
# Size of the shared buffer (in MB) used by HCCL for NPU-to-NPU collective communication

# Model weight path; can be a ModelScope model id (e.g., Eco-Tech/Qwen3.6-27B-w8a8c8-mxfp8) or a local directory path
# Ensure the model path matches the directory recorded during download

vllm serve $MODEL_PATH \
    --host 0.0.0.0 \
    --port 8000 \
    --data-parallel-size 1 \
    --tensor-parallel-size 1 \
    --quantization ascend \
    --kv-cache-dtype mxfp8 \
    --served-model-name qwen3.6 \
    --max-num-seqs 32 \
    --max-model-len 153600 \
    --max-num-batched-tokens 16384 \
    --trust-remote-code \
    --allowed-local-media-path / \
    --mm-processor-cache-gb 0 \
    --mm-encoder-tp-mode data \
    --mm-processor-cache-type shm \
    --gpu-memory-utilization 0.90 \
    --speculative-config '{"method": "qwen3_5_mtp", "num_speculative_tokens": 3}' \
    --compilation-config '{"cudagraph_capture_sizes": [1,4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64,68,72,76,80,84,88,92,96,100,104,108,112,116,120,124,128],"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --additional-config '{"enable_cpu_binding": true, "enable_flashcomm1": true}'

关键参数说明：

  • --data-parallel-size 1 and --tensor-parallel-size 1 are common settings for data parallelism (DP) and tensor parallelism (TP) sizes.
  • --max-model-len represents the context length, which is the maximum value of the input plus output for a single request. The Qwen3.6-27B model supports up to 262144.
  • --max-num-seqs indicates the maximum number of requests that each DP group is allowed to process. If the number of requests sent to the service exceeds this limit, the excess requests will remain in a waiting state and will not be scheduled. Note that the time spent in the waiting state is also counted in metrics such as TTFT and TPOT. Therefore, when testing performance, it is generally recommended that --max-num-seqs * --data-parallel-size >= the actual total concurrency.
  • --max-num-batched-tokens represents the maximum number of tokens that the model can process in a single step. Currently, vLLM v1 scheduling enables ChunkPrefill/SplitFuse by default, which means:
    • (1) If the input length of a request is greater than --max-num-batched-tokens, it will be divided into multiple rounds of computation according to --max-num-batched-tokens;
    • (2) Decode requests are prioritized for scheduling, and prefill requests are scheduled only if there is available capacity.
    • Generally, if --max-num-batched-tokens is set to a larger value, the overall latency will be lower, but the pressure on HBM memory (activation value usage) will be greater.
  • --gpu-memory-utilization represents the proportion of HBM that vLLM will use for actual inference. Its essential function is to calculate the available kv_cache size. During the warm-up phase (referred to as profile run in vLLM), vLLM records the peak HBM memory usage during an inference process with an input size of --max-num-batched-tokens. The available kv_cache size is then calculated as: --gpu-memory-utilization * HBM size - peak HBM memory usage. Therefore, the larger the value of --gpu-memory-utilization, the more kv_cache can be used. However, since the HBM memory usage during the warm-up phase may differ from that during actual inference (e.g., due to uneven EP load), setting --gpu-memory-utilization too high may lead to OOM (Out of Memory) issues during actual inference. The default value is 0.9.
  • Prefix caching is enabled by default, and the example above uses the default. Add --no-enable-prefix-caching to disable it if needed. For workloads with long shared prefixes, finer-grained hybrid prefix matching can be enabled with --enable-prefix-caching --prefix-match-unit 16. The match unit must divide every KV cache group's block size. A smaller unit can reuse more of a partial physical page, at the cost of additional hash metadata, copy-on-write operations, and an extra prefill split at the final matching boundary.
  • --quantization ascend indicates that quantization is used. To disable quantization, remove this option.
  • --kv-cache-dtype mxfp8 enables the C8 (MXFP8 KV cache) and is required by the Qwen3.6-27B-w8a8c8-mxfp8 weights. It stores the KV cache of the full-attention layers as FP8 (E4M3) with E8M0 scales, and runs the full-attention layers on the dedicated QFA (quant flash attention) kernel. The W8A8C8 weights bring performance gains in long-sequence scenarios. With this option, --block-size must be a multiple of 512 (it defaults to 512 if unset).
  • For short-sequence scenarios (e.g., 16K input length), the W8A8 weights (e.g., Qwen3.6-27B-w8a8-mxfp8) are recommended instead. When serving a W8A8 weight, remove --kv-cache-dtype mxfp8 from the startup command. Note that C8 quantization is only supported on 950PR&950DT Products.
  • --speculative-config uses qwen3_5_mtp for Qwen3.6-27B because it shares the same MTP head design as Qwen3.5-27B.
  • --compilation-config contains configurations related to the aclgraph graph mode. The most significant configurations are "cudagraph_mode" and "cudagraph_capture_sizes", which have the following meanings:
    • "cudagraph_mode": represents the specific graph mode. Currently, "PIECEWISE" and "FULL_DECODE_ONLY" are supported. The graph mode is mainly used to reduce the cost of operator dispatch. Currently, "FULL_DECODE_ONLY" is recommended.
    • "cudagraph_capture_sizes": specifies the batch size tiers at which graphs are captured. The default value is [1, 2, 4, 8, 16, 24, 32, 40,..., --max-num-seqs]. In the graph mode, the batch size of each captured graph is fixed, and an actual batch size between two adjacent capture sizes is automatically padded up to the next capture size. Currently, the default setting is recommended. Only in some scenarios is it necessary to set this separately to achieve optimal performance.
  • --allowed-local-media-path allows the API server to read local images or videos from the specified directory when processing multimodal inputs. In this example, / whitelists the entire file system. This is security-sensitive and should only be enabled in trusted environments.
  • --mm-processor-cache-gb sets the size (in GiB) of the multimodal processor cache, which is used to avoid re-processing past multimodal inputs. The cache is duplicated in each API server process and engine core process, so the total host memory usage is the cache size multiplied by the number of these processes. A value of 0 disables this cache completely.
  • --mm-encoder-tp-mode indicates how to optimize multimodal encoder inference using tensor parallelism (TP). data splits the batched encoder input data across TP ranks while each rank hosts the full encoder weights, and is recommended when multimodal inputs are used. The default mode weights splits the encoder weights across TP ranks instead.
  • --mm-processor-cache-type sets the cache type used by the multimodal preprocessor and mapper. shm uses a shared-memory FIFO cache, while the default lru uses a mirrored LRU cache.
  • --additional-config contains Ascend-specific optimization options:
    • "enable_cpu_binding": true enables Ascend-native CPU binding, which pins the worker process, key runtime threads, memory pages, and NPU IRQs to CPU ranges close to the driven NPU, reducing cross-NUMA traffic and thread preemption and improving latency stability. It is enabled by default starting from vllm-ascend v0.18.0rc1.
    • "enable_flashcomm1": true enables the FlashComm communication optimization, which splits the tensor-parallel AllReduce into Reduce-Scatter and All-Gather to reduce communication overhead and improve prefill throughput.

常见问题提示：如果遇到问题，请参阅公共常见问题进行故障排查。

服务验证：

如果服务启动成功，将显示以下启动日志：

(APIServer pid=<pid>) INFO:     Started server process [<pid>]
(APIServer pid=<pid>) INFO:     Waiting for application startup.
(APIServer pid=<pid>) INFO:     Application startup complete.

功能测试（例如，包含预期响应的 completions 和 chat.completions curl 示例）请参考第 6 节。

6 功能验证

服务启动后，可通过发送提示词来调用模型。支持两种 API 接口：completions 和 chat.completions。使用您配置的 --served-model-name（Qwen3.5-27B 对应 qwen3.5，Qwen3.6-27B 对应 qwen3.6）。

Completions API：

curl http://127.0.0.1:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "qwen3.5",
        "prompt": "The future of AI is",
        "max_tokens": 50,
        "temperature": 0
    }'

注意：对于 Qwen3.6-27B，请在请求体中设置 "model": "qwen3.6"。

Chat Completions API：

curl http://127.0.0.1:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "qwen3.5",
        "messages": [
            {"role": "user", "content": "The future of AI is"}
        ],
        "max_completion_tokens": 1024,
        "temperature": 0.7,
        "top_p": 0.95
    }'

注意：对于 Qwen3.6-27B，请在请求体中设置 "model": "qwen3.6"。

预期结果：服务返回 HTTP 200 OK。JSON 响应中包含 choices 字段及生成的文本。以下是 completions API 的示例输出（内容已截断以保持简洁）：

{
    "id": "cmpl-xxxxxxxxxxxxx",
    "object": "text_completion",
    "created": 1780971952,
    "model": "qwen3.5",
    "choices": [
        {
            "index": 0,
            "text": "The future of AI is a rapidly evolving landscape with breakthroughs in natural language understanding, multimodal reasoning, and autonomous agents. As models grow more capable and efficient...",
            "logprobs": null,
            "finish_reason": "length"
        }
    ],
    "usage": {
        "prompt_tokens": 4,
        "total_tokens": 54,
        "completion_tokens": 50
    }
}

7 精度评估

以下是两种精度评估方法。

7.1 使用 AISBench

  1. For detailed instructions, refer to Using AISBench for accuracy evaluation.

  2. After execution, you can get the result. For reference only, here is the result of Qwen3.5-27B-w8a8 obtained with vllm-ascend:v0.17.0rc1 on an Atlas 800I A3 node. The accuracy result of Qwen3.6-27B-w8a8 can be obtained in the same way and is not listed here.

对于 Qwen3.6-27B-w8a8，请将 model 改为 qwen3.6，并将 path 改为对应的模型权重路径。

数据集 版本 指标 模式 vllm-api-general-chat
gsm8k - accuracy gen 96.74

7.2 使用 Language Model Evaluation Harness

以 gsm8k 数据集为例，在在线模式下对 Qwen3.5-27B-w8a8 进行精度评估。

  1. lm_eval 的安装请参考使用 lm_eval。
  2. 运行 lm_eval 执行精度评估。
# Run lm_eval in another terminal
lm_eval \
  --model local-completions \
  --model_args model=qwen3.5,base_url=http://127.0.0.1:8000/v1/completions,tokenized_requests=False,trust_remote_code=True \
  --tasks gsm8k \
  --output_path ./

8 性能评估

8.1 使用 AISBench

详情请参考使用 AISBench 进行性能评估。

8.2 使用 vLLM Benchmark

更多详情请参考 vllm benchmark。

9 性能调优

9.1 推荐配置

Note: The following configurations are validated in specific test environments and are for reference only. The optimal configuration depends on factors such as maximum input/output length, prefix cache hit rate, precision requirements, and deployment machine ratios. It is recommended to refer to Section 9.2 for tuning based on actual conditions.

Parallelism Strategy: Qwen3.5-27B-w8a8 and Qwen3.6-27B-w8a8 are only ~30 GB and easily fit in a single NPU (64 GB HBM per NPU). Following the DP-first principle, TP=2 is the recommended default for most scenarios, and the remaining NPUs should be allocated to DP for parallel request batches. TP=8 is only recommended for ultra-long context (256k+) scenarios, where it shards the KV cache across 8 NPUs to maximize the available context window per rank. For Qwen3.6-27B-w8a8, you can also raise --max-model-len up to 262144 in the same TP/DP layout.

Atlas 300I DUO: Currently only the TP scenario is supported. Choose TP=2 or TP=4 according to the available devices. With TP=4, --max-model-len can support 128k and 256k long-sequence scenarios.

950DT Products: The Qwen3.6-27B-w8a8c8-mxfp8 model weight easily fits in a single NPU (96 GB HBM per NPU). Following the DP-first principle, TP=1 is the recommended default for most scenarios, and the remaining NPUs should be allocated to DP for parallel request batches. The C8 weights are recommended for long-sequence scenarios and are served with --kv-cache-dtype mxfp8; for short-sequence scenarios (e.g., 16K input), use Qwen3.6-27B-w8a8-mxfp8 without --kv-cache-dtype mxfp8. --max-model-len can support up to 262144 in the same TP=1 + DP=8 layout.

表 1：场景概览

场景 部署模式 *NPU总数 权重版本 关键考量
高吞吐
(128k上下文)		 单节点 (A2) 8 (A2) Qwen3.5-27B-w8a8 / Qwen3.6-27B-w8a8 TP=2 + DP=4 充分利用全部8个NPU并行处理请求批次
高吞吐
(128k上下文)		 单节点 (A3) 16 (A3) Qwen3.5-27B-w8a8 / Qwen3.6-27B-w8a8 TP=2 + DP=8 充分利用全部16个NPU并行处理请求批次
低延迟
(128k上下文)		 单节点 (A3) 16 (A3) Qwen3.5-27B-w8a8 / Qwen3.6-27B-w8a8 TP=2 + DP=8 减少小型交互式批次的逐层Allreduce开销
长上下文
(256k+上下文)		 单节点 (A3) 16 (A3) Qwen3.5-27B-w8a8 / Qwen3.6-27B-w8a8 TP=8 + DP=2 将KV缓存分布在8个NPU上，以最大化可用上下文窗口
High Throughput
(128k context)		 Single-Node (950DT Products) 8 (950DT Products) Qwen3.6-27B-w8a8c8-mxfp8 TP=1 + DP=8 fully utilizes all 8 NPUs for parallel request batches
Long Context
(256k+ context)		 Single-Node (950DT Products) 8 (950DT Products) Qwen3.6-27B-w8a8c8-mxfp8 TP=1 + DP=8 maximizes the available context window while keeping all 8 NPUs busy

*Total NPUs indicates the total number of NPUs used across all nodes. 1 Atlas 800I A3 node = 16 NPUs, 1 Atlas 800I A2 node = 8 NPUs, 1 950DT Products node = 8 NPUs.

表 2：详细节点配置

场景 配置 NPU数量 TP DP 最大序列数 最大批处理Token数 最大模型长度 MTP推测数量 异步调度
High Throughput (128k) Single-Node (A2) 8 2 4 32 16384 133000 3 On
High Throughput (128k) Single-Node (A3) 16 2 8 32 16384 133000 3 On
Low Latency (128k) Single-Node (A3) 16 2 8 4 4096 133000 3 On
Long Context (256k+) Single-Node (A3) 16 8 2 8 8192 262144 3 On
High Throughput (128k) Single-Node (950DT Products) 8 1 8 32 16384 133000 3 On
Long Context (256k+) Single-Node (950DT Products) 8 1 8 32 16384 262144 3 On

For complete startup commands and parameter descriptions, please refer to the deployment examples in Section 5.

9.2 调优指南

9.2.1 通用调优参考

调优方法请参阅公共性能调优文档。 详细功能描述请参阅功能矩阵。

10 常见问题

对于常见的环境、安装和一般参数问题，请参阅公共常见问题。