Qwen3.5-27B/Qwen3.6-27B

1 简介

Qwen3.5-27B 和 Qwen3.6-27B 是 Qwen3.5/Qwen3.6 系列中的密集混合 Mamba-Transformer 语言模型，融合了架构效率、强化学习规模和全球可访问性方面的突破。它们共享相同的混合注意力设计（GDN + 全注意力），因此在 Ascend NPU 上的部署对两个模型遵循相同的模式。它们适用于在 Ascend NPU 上运行的通用文本生成任务，例如对话、内容创作和代码生成。

本文档将演示模型的主要验证步骤，包括支持的特性、特性配置、环境准备、单节点和多节点部署，以及精度和性能评估。

强烈建议使用 vllm-ascend 的最新候选发布（rc）版本或最新正式版本。作为最低版本要求，Qwen3.5-27B 首次在 vllm-ascend:v0.17.0rc1 中得到支持，Qwen3.6-27B 首次在 vllm-ascend:v0.18.0rc1 中得到支持。 Support for Atlas inference products starts from vllm-ascend:v0.23.0rc1.

2 支持的特性

请参考支持的特性列表了解模型支持矩阵。

请参考特性指南了解特性配置信息。

3 前提条件

3.1 模型权重

Qwen3.5-27B

• Qwen3.5-27B (BF16 version): requires 1 Atlas 800 A3 (64G × 16) node or 1 Atlas 800 A2 (64G × 8) node or Atlas inference products. Download model weight
• Qwen3.5-27B-w8a8 (Quantized version): requires 1 Atlas 800 A3 (64G × 16) node or 1 Atlas 800 A2 (64G × 8) node or Atlas inference products. Download model weight

Qwen3.6-27B

• Qwen3.6-27B (BF16 version): requires 1 Atlas 800 A3 (64G × 16) node or 1 Atlas 800 A2 (64G × 8) node or Atlas inference products. Download model weight
• Qwen3.6-27B-w8a8 (Quantized version): requires 1 Atlas 800 A3 (64G × 16) node or 1 Atlas 800 A2 (64G × 8) node or Atlas inference products. Download model weight

建议将模型权重下载到多节点的共享目录中，例如 /root/.cache/。

3.2 验证多节点通信

如果要部署多节点环境，需要按照验证多节点通信环境中的说明验证多节点通信。

4 安装

4.1 Docker 镜像安装

根据您的机器类型选择镜像，并在节点上启动 docker 镜像，请参考使用 docker。

建议使用 vllm-ascend 镜像的最新候选发布（rc）版本或最新正式版本，以确保最佳兼容性并访问最新特性。作为最低版本要求，对于 Qwen3.5-27B，请使用 vllm-ascend:v0.17.0rc1（或更高版本）；对于 Qwen3.6-27B，请使用 vllm-ascend:v0.18.0rc1（或更高版本）。对于 Atlas 800 A3 上的 Qwen3.6-27B，请使用匹配的 v0.18.0rc1-a3（或更高版本的 -a3）镜像。 For Atlas inference products, use vllm-ascend:v0.23.0rc1-310p (or a later -310p) image.

A3 系列A2 系列Atlas inference products

在每个节点上启动 docker 镜像。

export IMAGE=m.daocloud.io/quay.io/ascend/vllm-ascend:v0.22.1rc1-a3
docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --net=host \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci8 \
    --device /dev/davinci9 \
    --device /dev/davinci10 \
    --device /dev/davinci11 \
    --device /dev/davinci12 \
    --device /dev/davinci13 \
    --device /dev/davinci14 \
    --device /dev/davinci15 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /root/.cache:/root/.cache \
    -it $IMAGE bash

在每个节点上启动 docker 镜像。

export IMAGE=m.daocloud.io/quay.io/ascend/vllm-ascend:v0.22.1rc1
docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --net=host \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /root/.cache:/root/.cache \
    -it $IMAGE bash

在每个节点上启动 docker 镜像。

export IMAGE=m.daocloud.io/quay.io/ascend/vllm-ascend:v0.22.1rc1-310p
docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /root/.cache:/root/.cache \
    -p 8080:8080 \
    -it $IMAGE bash

docker 运行成功后，可以通过执行 docker ps 命令验证正在运行的容器服务。预期结果是容器 vllm-ascend 的状态为 Up，确认 docker 安装成功。

4.2 源码安装

如果您不想像上面那样使用 docker 镜像，也可以从源码构建所有内容：

1. 克隆仓库并从源码安装 vllm-ascend：

   git clone https://github.com/vllm-project/vllm-ascend.git
   cd vllm-ascend
   pip install -e .
   

   完整的安装步骤请参考安装指南。

   Note

   On Atlas inference products, you may need to uninstall triton-ascend and triton to avoid dependency conflicts:

   pip uninstall -y triton-ascend triton
   

2. 如果要部署多节点环境，需要在每个节点上设置环境。

要验证源码安装，请运行以下命令并确认显示的版本与您安装的版本一致：

pip show vllm-ascend

预期结果：显示 vllm-ascend 的版本信息，确认安装成功。

5 在线服务部署

5.1 单节点在线部署

Single-node deployment completes both Prefill and Decode within the same node, suitable for development, testing, and medium-scale inference scenarios. The Qwen3.5-27B, Qwen3.5-27B-w8a8, Qwen3.6-27B, and Qwen3.6-27B-w8a8 models can all be deployed on 1 Atlas 800 A3 (64G × 16) or 1 Atlas 800 A2 (64G × 8). On Atlas inference products, at least 2 devices are required. The quantized versions need to start with the --quantization ascend parameter.

Qwen3.5-27B 和 Qwen3.6-27B 共享相同的 MTP 头部设计，因此 qwen3_5_mtp 推测解码方法可以用于两者。

Atlas 800 A3 / Atlas 800 A2Atlas inference products

The following examples are for Atlas 800 A3 / Atlas 800 A2. Quantized versions need --quantization ascend.

Qwen3.5-27B-w8a8Qwen3.6-27B-w8a8

启动命令：

#!/bin/sh
# Load model from ModelScope to speed up download
export VLLM_USE_MODELSCOPE=True
# To reduce memory fragmentation and avoid out of memory
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
# Size of the shared buffer (in MB) used by HCCL for NPU-to-NPU collective communication
export HCCL_BUFFSIZE=512
# Whether OpenMP threads are bound to specific CPU cores
export OMP_PROC_BIND=false
# Number of OpenMP threads available for parallel regions
export OMP_NUM_THREADS=1
# Enables the Ascend task queue for asynchronous operator dispatch
export TASK_QUEUE_ENABLE=1

# Model weight path; can be a ModelScope model id (e.g., Eco-Tech/Qwen3.5-27B-w8a8-mtp) or a local directory path
export MODEL_PATH=Eco-Tech/Qwen3.5-27B-w8a8-mtp

vllm serve $MODEL_PATH \
--host 0.0.0.0 \
--port 8000 \
--data-parallel-size 1 \
--tensor-parallel-size 2 \
--seed 1024 \
--quantization ascend \
--served-model-name qwen3.5 \
--max-num-seqs 32 \
--max-model-len 133000 \
--max-num-batched-tokens 8096 \
--trust-remote-code \
--gpu-memory-utilization 0.90 \
--no-enable-prefix-caching \
--speculative-config '{"method": "qwen3_5_mtp", "num_speculative_tokens": 3, "enforce_eager": true}' \
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
--additional-config '{"enable_cpu_binding":true}' \
--async-scheduling

启动命令（支持最长 262144 上下文长度）：

#!/bin/sh
# Load model from ModelScope to speed up download
export VLLM_USE_MODELSCOPE=True
# To reduce memory fragmentation and avoid out of memory
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
# Size of the shared buffer (in MB) used by HCCL for NPU-to-NPU collective communication
export HCCL_BUFFSIZE=512
# Whether OpenMP threads are bound to specific CPU cores
export OMP_PROC_BIND=false
# Number of OpenMP threads available for parallel regions
export OMP_NUM_THREADS=1
# Enables the Ascend task queue for asynchronous operator dispatch
export TASK_QUEUE_ENABLE=1

# Model weight path; can be a ModelScope model id (e.g., Eco-Tech/Qwen3.6-27B-w8a8) or a local directory path
export MODEL_PATH=Eco-Tech/Qwen3.6-27B-w8a8

vllm serve $MODEL_PATH \
--host 0.0.0.0 \
--port 8000 \
--data-parallel-size 1 \
--tensor-parallel-size 2 \
--seed 1024 \
--quantization ascend \
--served-model-name qwen3.6 \
--max-num-seqs 32 \
--max-model-len 262144 \
--max-num-batched-tokens 8096 \
--trust-remote-code \
--gpu-memory-utilization 0.90 \
--no-enable-prefix-caching \
--speculative-config '{"method": "qwen3_5_mtp", "num_speculative_tokens": 3, "enforce_eager": true}' \
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
--additional-config '{"enable_cpu_binding":true}' \
--async-scheduling

关键参数说明：

• --data-parallel-size 1 and --tensor-parallel-size 2 are common settings for data parallelism (DP) and tensor parallelism (TP) sizes.
• --max-model-len represents the context length, which is the maximum value of the input plus output for a single request. The Qwen3.6-27B model supports up to 262144.
• --max-num-seqs indicates the maximum number of requests that each DP group is allowed to process. If the number of requests sent to the service exceeds this limit, the excess requests will remain in a waiting state and will not be scheduled. Note that the time spent in the waiting state is also counted in metrics such as TTFT and TPOT. Therefore, when testing performance, it is generally recommended that --max-num-seqs * --data-parallel-size >= the actual total concurrency.
• --max-num-batched-tokens represents the maximum number of tokens that the model can process in a single step. Currently, vLLM v1 scheduling enables ChunkPrefill/SplitFuse by default, which means:
  • (1) If the input length of a request is greater than --max-num-batched-tokens, it will be divided into multiple rounds of computation according to --max-num-batched-tokens;
  • (2) Decode requests are prioritized for scheduling, and prefill requests are scheduled only if there is available capacity.
  • Generally, if --max-num-batched-tokens is set to a larger value, the overall latency will be lower, but the pressure on HBM memory (activation value usage) will be greater.
• --gpu-memory-utilization represents the proportion of HBM that vLLM will use for actual inference. Its essential function is to calculate the available kv_cache size. During the warm-up phase (referred to as profile run in vLLM), vLLM records the peak HBM memory usage during an inference process with an input size of --max-num-batched-tokens. The available kv_cache size is then calculated as: --gpu-memory-utilization * HBM size - peak HBM memory usage. Therefore, the larger the value of --gpu-memory-utilization, the more kv_cache can be used. However, since the HBM memory usage during the warm-up phase may differ from that during actual inference (e.g., due to uneven EP load), setting --gpu-memory-utilization too high may lead to OOM (Out of Memory) issues during actual inference. The default value is 0.9.
• --no-enable-prefix-caching indicates that prefix caching is disabled. The current implementation of hybrid kv cache for Qwen3.5-27B / Qwen3.6-27B may result in a very large effective block_size when prefix caching is enabled (e.g., 2048), which means any prefix shorter than block_size will never be cached. If your workload has many short repeated prefixes, consider keeping prefix caching disabled. For related issues, see the Public FAQ.
• --quantization ascend indicates that quantization is used. To disable quantization, remove this option.
• --speculative-config uses qwen3_5_mtp for both Qwen3.5-27B and Qwen3.6-27B because they share the same MTP head design.
• --compilation-config contains configurations related to the aclgraph graph mode. The most significant configurations are "cudagraph_mode" and "cudagraph_capture_sizes", which have the following meanings:
  • "cudagraph_mode": represents the specific graph mode. Currently, "PIECEWISE" and "FULL_DECODE_ONLY" are supported. The graph mode is mainly used to reduce the cost of operator dispatch. Currently, "FULL_DECODE_ONLY" is recommended.
  • "cudagraph_capture_sizes": represents different levels of graph modes. The default value is [1, 2, 4, 8, 16, 24, 32, 40,..., --max-num-seqs]. In the graph mode, the input for graphs at different levels is fixed, and inputs between levels are automatically padded to the next level. Currently, the default setting is recommended. Only in some scenarios is it necessary to set this separately to achieve optimal performance.

Currently only the TP scenario is supported. Choose TP=2 or TP=4 according to the available devices. Replace MODEL_PATH with a ModelScope model id or a local directory path.The quantized versions need to start with the --quantization ascend parameter.

Qwen3.5-27B-w8a8Qwen3.6-27B-w8a8

启动命令：

#!/bin/sh
# Load model from ModelScope to speed up download
export VLLM_USE_MODELSCOPE=True

# Model weight path; can be a ModelScope model id (e.g., Eco-Tech/Qwen3.5-27B-w8a8-mtp) or a local directory path
export MODEL_PATH=Eco-Tech/Qwen3.5-27B-w8a8-mtp

vllm serve $MODEL_PATH \
--host 127.0.0.1 \
--port 1025 \
--tensor-parallel-size 4 \
--served-model-name qwen3.5 \
--max-num-seqs 128 \
--max-model-len 16384 \
--trust-remote-code \
--gpu-memory-utilization 0.90 \
--mamba-ssm-cache-dtype float16 \
--dtype float16 \
--speculative-config '{"method": "qwen3_5_mtp","num_speculative_tokens":1}' \
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [1,8]}' \
--additional-config '{"ascend_compilation_config": {"enable_npugraph_ex": false}}'

启动命令：

#!/bin/sh
# Load model from ModelScope to speed up download
export VLLM_USE_MODELSCOPE=True

# Model weight path; can be a ModelScope model id (e.g., Eco-Tech/Qwen3.6-27B-w8a8) or a local directory path
export MODEL_PATH=Eco-Tech/Qwen3.6-27B-w8a8

vllm serve $MODEL_PATH \
--host 127.0.0.1 \
--port 1025 \
--tensor-parallel-size 4 \
--served-model-name qwen3.6 \
--max-num-seqs 128 \
--max-model-len 16384 \
--trust-remote-code \
--gpu-memory-utilization 0.90 \
--mamba-ssm-cache-dtype float16 \
--dtype float16 \
--speculative-config '{"method": "qwen3_5_mtp","num_speculative_tokens":1}' \
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [1,8]}' \
--additional-config '{"ascend_compilation_config": {"enable_npugraph_ex": false}}'

关键参数说明：

• --tensor-parallel-size sets the tensor parallel size. Choose TP=2 or TP=4 according to the available devices.
• --max-model-len represents the context length, which is the maximum value of the input plus output for a single request. Configure it based on the actual workload and available memory; The Qwen3.6-27B model supports up to 262144.
• --max-num-seqs indicates the maximum number of concurrent requests. Configure it as needed—setting it too high may cause OOM.
• --gpu-memory-utilization represents the proportion of HBM that vLLM will use for actual inference. Configure this value according to the actual device memory; setting it too high may cause OOM. The default value is 0.9.
• --mamba-ssm-cache-dtype sets the data type of the Mamba SSM cache. On Atlas inference products, only float16 is supported.
• --dtype float16 must be set on Atlas inference products. These devices only support the FP16 data type.
• --speculative-config uses qwen3_5_mtp for both Qwen3.5-27B and Qwen3.6-27B because they share the same MTP head design. On Atlas inference products, it is recommended to set num_speculative_tokens to 1.
• --compilation-config contains configurations related to the aclgraph graph mode. The most significant configurations are "cudagraph_mode" and "cudagraph_capture_sizes", which have the following meanings:
  • "cudagraph_mode": represents the specific graph mode. Currently, "PIECEWISE" and "FULL_DECODE_ONLY" are supported. The graph mode is mainly used to reduce the cost of operator dispatch. Currently, "FULL_DECODE_ONLY" is recommended.
  • "cudagraph_capture_sizes": represents different levels of graph modes. When tensor parallelism (TP) is enabled, hardware event-id constraints allow at most two capture sizes (for example, [1, 8]).
• --additional-config with "ascend_compilation_config": {"enable_npugraph_ex": false} is required on Atlas inference products because enable_npugraph_ex is not supported on this platform.

常见问题提示：如果遇到问题，请参考公共 FAQ 进行故障排除。

服务验证：

如果服务启动成功，将显示以下启动日志：

(APIServer pid=<pid>) INFO:     Started server process [<pid>]
(APIServer pid=<pid>) INFO:     Waiting for application startup.
(APIServer pid=<pid>) INFO:     Application startup complete.

功能测试（例如，包含预期响应的 completions 和 chat.completions curl 示例）请参考第 6 节。

5.2 多节点 PD 分离部署

Note

Multi-node PD separation deployment is not supported on Atlas inference products.

对于高并发生产场景，可以使用多节点 PD（Prefill-Decode）分离来扩展服务。推荐使用 Mooncake 进行部署：Mooncake 多节点 PD 分离指南。

在标准单节点部署模式下，Prefill（提示处理）和 Decode（令牌生成）任务运行在同一组 NPU 上。这可能导致两个问题：

1. Prefill 抢占中断 Decode：Prefill 是计算密集型任务，一次性处理整个输入上下文，而 Decode 逐个生成令牌。当新的用户请求到达时，其 Prefill 阶段可能会抢占并中断正在进行的 Decode 任务，导致抖动和更高的每输出令牌时间（TPOT）延迟。
2. 资源分配不灵活：Prefill 和 Decode 具有根本不同的计算特性——Prefill 是计算受限且内存带宽密集型的，而 Decode 是内存带宽受限的。在相同硬件上运行它们会迫使做出折衷，无法使两者都达到最优。

PD（Prefill-Decode）分离通过将 Prefill 和 Decode 运行在专用的节点组上来解决这些问题，每个节点组独立配置：

• **Prefill节点**专注于高吞吐量的提示处理，针对计算和通信进行了优化（例如，启用FlashComm以加速Allreduce）。
• **Decode节点**专注于低延迟的令牌生成，针对内存带宽进行了优化（例如，启用异步调度和全解码aclgraph）。

对于Qwen3.5-27B-w8a8和Qwen3.6-27B-w8a8，典型的**1P1D**配置需要**2个Atlas 800 A3（64G × 16）节点**（1个Prefill节点 + 1个Decode节点），每个节点上**TP=2**和**DP=8**，这充分利用了Atlas A3的所有16个NPU。下面的示例使用Qwen3.5-27B-w8a8；对于Qwen3.6-27B-w8a8，将模型路径替换为Eco-Tech/Qwen3.6-27B-w8a8，并将--served-model-name调整为qwen3.6（如果需要，将--max-model-len调整为262144）。

为什么是TP=2 + DP=8（DP优先策略）？ Qwen3.5-27B-w8a8（和Qwen3.6-27B-w8a8）模型仅约30 GB，可以轻松放入单个NPU（每个NPU有64 GB HBM）。TP > 1主要用于无法放入一个NPU的模型。 对于27B模型，TP=2足以平衡NPU之间的算子分发开销，而**最大化DP**则让Atlas A3的所有16个NPU都忙于处理独立的请求批次，充分利用硬件。这种**DP优先的并行策略**是小型密集模型（例如Qwen3.5-27B、Qwen3.6-27B、Llama-3-8B）的标准实践，并已通过Qwen3.5-27B B200基准测试验证，其中从TP=8切换到DP=8将单节点吞吐量从9.5k提升到95k tokens/s。

注意：由于Qwen3.5-27B和Qwen3.6-27B适合单个节点，多节点PD分离仅推荐用于高并发生产部署。关于Mooncake部署的具体细节，请参考Mooncake多节点PD分离指南。

要运行vllm-ascend的Prefill-Decode分离服务，您需要：

• 在每个节点上部署launch_online_dp.py脚本和run_dp_template.sh脚本；

• 在prefill主节点上部署load_balance_proxy_server_example.py脚本以转发请求。

• launch_online_dp.py用于启动外部dp vllm服务。 launch_online_dp.py

  参数说明：

  参数	类型	必填	默认值	描述
  --dp-size	int	是	-	数据并行大小（所有节点上的DP rank总数）。
  --tp-size	int	否	1	每个DP rank内的张量并行大小。
  --dp-size-local	int	否	（与--dp-size相同）	当前节点上的DP rank数量。如果未设置，默认为--dp-size。
  --dp-rank-start	int	否	0	此节点上数据并行rank的起始偏移量。
  --dp-address	str	是	-	数据并行主节点（节点0）的IP地址。
  --dp-rpc-port	str	否	12345	数据并行主节点通信的RPC端口。
  --vllm-start-port	int	否	9000	此节点上每个vLLM引擎实例的起始端口。每个DP rank的引擎端口 = vllm_start_port + 本地rank索引。

• Prefill节点0的run_dp_template.sh脚本。您可以在仓库的示例中获取模板：run_dp_template.sh。

  # nic_name is the network interface name corresponding to local_ip of the current node
  nic_name="xxx"
  local_ip="141.xx.xx.1"
  
  export HCCL_IF_IP=$local_ip
  export GLOO_SOCKET_IFNAME=$nic_name
  export TP_SOCKET_IFNAME=$nic_name
  export HCCL_SOCKET_IFNAME=$nic_name
  
  # [Optional] jemalloc
  # jemalloc is for better performance, if `libjemalloc.so` is installed on your machine, you can turn it on.
  # export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
  
  export HCCL_OP_EXPANSION_MODE="AIV"
  export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
  export OMP_PROC_BIND=false
  export OMP_NUM_THREADS=1
  export TASK_QUEUE_ENABLE=1
  export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/mooncake:$LD_LIBRARY_PATH
  
  export HCCL_BUFFSIZE=1024
  export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
  export ASCEND_RT_VISIBLE_DEVICES=$1
  
  vllm serve Eco-Tech/Qwen3.5-27B-w8a8-mtp \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --seed 1024 \
    --quantization ascend \
    --served-model-name qwen3.5 \
    --trust-remote-code \
    --max-num-seqs 4 \
    --max-model-len 32768 \
    --max-num-batched-tokens 16384 \
    --no-enable-prefix-caching \
    --gpu-memory-utilization 0.95 \
    --enforce-eager \
    --speculative-config '{"method": "qwen3_5_mtp", "num_speculative_tokens": 3, "enforce_eager": true}' \
    --additional-config '{"enable_cpu_binding":true}' \
    --kv-transfer-config \
    '{"kv_connector": "MooncakeConnectorV1",
    "kv_role": "kv_producer",
    "kv_port": "30000",
    "engine_id": "0",
    "kv_connector_extra_config": {
              "prefill": {
                      "dp_size": 8,
                      "tp_size": 2
              },
              "decode": {
                      "dp_size": 8,
                      "tp_size": 2
          }
      }
    }'
  

• Decode节点0的run_dp_template.sh脚本。您可以在仓库的示例中获取模板：run_dp_template.sh。

  # nic_name is the network interface name corresponding to local_ip of the current node
  nic_name="xxx"
  local_ip="141.xx.xx.2"
  
  export HCCL_IF_IP=$local_ip
  export GLOO_SOCKET_IFNAME=$nic_name
  export TP_SOCKET_IFNAME=$nic_name
  export HCCL_SOCKET_IFNAME=$nic_name
  
  # [Optional] jemalloc
  # jemalloc is for better performance, if `libjemalloc.so` is installed on your machine, you can turn it on.
  # export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
  
  export HCCL_OP_EXPANSION_MODE="AIV"
  export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
  export OMP_PROC_BIND=false
  export OMP_NUM_THREADS=1
  export TASK_QUEUE_ENABLE=1
  export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/mooncake:$LD_LIBRARY_PATH
  
  export HCCL_BUFFSIZE=1024
  export ASCEND_RT_VISIBLE_DEVICES=$1
  
  vllm serve Eco-Tech/Qwen3.5-27B-w8a8-mtp \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --seed 1024 \
    --quantization ascend \
    --served-model-name qwen3.5 \
    --trust-remote-code \
    --max-num-seqs 16 \
    --max-model-len 32768 \
    --max-num-batched-tokens 2048 \
    --no-enable-prefix-caching \
    --gpu-memory-utilization 0.91 \
    --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
    --additional-config '{"recompute_scheduler_enable":true,"enable_cpu_binding":true}' \
    --async-scheduling \
    --speculative-config '{"method": "qwen3_5_mtp", "num_speculative_tokens": 3, "enforce_eager": true}' \
    --kv-transfer-config \
    '{"kv_connector": "MooncakeConnectorV1",
    "kv_role": "kv_consumer",
    "kv_port": "30200",
    "engine_id": "1",
    "kv_connector_extra_config": {
              "prefill": {
                      "dp_size": 8,
                      "tp_size": 2
              },
              "decode": {
                      "dp_size": 8,
                      "tp_size": 2
          }
      }
    }'
  

关键参数说明：

• VLLM_ASCEND_ENABLE_FLASHCOMM1=1：在prefill节点上启用Allreduce通信优化，减少长上下文prefill的通信开销。
• recompute_scheduler_enable: true：启用重计算调度器。当decode节点的KV缓存不足时，请求将被发送到prefill节点以重新计算KV缓存。在PD分离场景中，仅在decode节点上启用此配置。
• --async-scheduling（在decode节点上）：启用异步调度，可以减少高并发解码工作负载的TPOT。

• --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'（在decode节点上）：启用全解码aclgraph模式，显著降低解码端的调度延迟。

• 为每个节点运行服务器：

  # p0 (Prefill node 0)
  python launch_online_dp.py --dp-size 8 --tp-size 2 --dp-size-local 8 --dp-rank-start 0 --dp-address 141.xx.xx.1 --dp-rpc-port 12321 --vllm-start-port 7100
  # d0 (Decode node 0)
  python launch_online_dp.py --dp-size 8 --tp-size 2 --dp-size-local 8 --dp-rank-start 0 --dp-address 141.xx.xx.2 --dp-rpc-port 12321 --vllm-start-port 7100
  

• 在prefill主节点上运行代理服务器。

  您可以在仓库的示例中获取代理程序：load_balance_proxy_server_example.py。

  注意：由于每个节点有8个DP rank（使用--vllm-start-port 7100 + 本地rank索引，占用端口7100-7107），您需要在代理命令中列出每个节点的所有8个端口：

  python load_balance_proxy_server_example.py \
    --port 1999 \
    --host 141.xx.xx.1 \
    --prefiller-hosts \
      141.xx.xx.1 \
      141.xx.xx.1 \
      141.xx.xx.1 \
      141.xx.xx.1 \
      141.xx.xx.1 \
      141.xx.xx.1 \
      141.xx.xx.1 \
      141.xx.xx.1 \
    --prefiller-ports \
      7100 7101 7102 7103 7104 7105 7106 7107 \
    --decoder-hosts \
      141.xx.xx.2 \
      141.xx.xx.2 \
      141.xx.xx.2 \
      141.xx.xx.2 \
      141.xx.xx.2 \
      141.xx.xx.2 \
      141.xx.xx.2 \
      141.xx.xx.2 \
    --decoder-ports \
      7100 7101 7102 7103 7104 7105 7106 7107 \
  

部署验证：

PD分离服务完全启动后，通过prefill主节点上的代理端口发送请求，验证Prefill和Decode节点是否协同工作正常：

curl http://<proxy_node0_ip>:1999/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "qwen3.5",
        "messages": [
            {"role": "user", "content": "The future of AI is"}
        ],
        "max_tokens": 1024,
        "temperature": 1.0,
        "top_p": 0.95
    }'

注意：对于Qwen3.6-27B-w8a8，将上面的model字段改为"qwen3.6"，并将Prefill/Decode节点的--served-model-name改为qwen3.6。

预期结果：代理返回 HTTP 200 OK。JSON 响应中包含 choices 字段及生成的文本，确认 Prefill 节点已成功处理提示，Decode 节点已生成响应。

常见问题提示：如果在 PD 分离部署中遇到问题，请参考公共 FAQ 进行故障排除。

6 功能验证

服务启动后，可通过发送提示词来调用模型。支持两种 API 接口：completions 和 chat.completions。使用您配置的 --served-model-name（Qwen3.5-27B 对应 qwen3.5，Qwen3.6-27B 对应 qwen3.6）。

Completions API：

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "qwen3.5",
        "prompt": "The future of AI is",
        "max_tokens": 50,
        "temperature": 0
    }'

注意：对于 Qwen3.6-27B，请在请求体中设置 "model": "qwen3.6"。

Chat Completions API：

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "qwen3.5",
        "messages": [
            {"role": "user", "content": "The future of AI is"}
        ],
        "max_completion_tokens": 1024,
        "temperature": 0.7,
        "top_p": 0.95
    }'

注意：对于 Qwen3.6-27B，请在请求体中设置 "model": "qwen3.6"。

预期结果：服务返回 HTTP 200 OK。JSON 响应中包含 choices 字段及生成的文本。以下是 completions API 的示例输出（内容已截断以保持简洁）：

{
    "id": "cmpl-xxxxxxxxxxxxx",
    "object": "text_completion",
    "created": 1780971952,
    "model": "qwen3.5",
    "choices": [
        {
            "index": 0,
            "text": "The future of AI is a rapidly evolving landscape with breakthroughs in natural language understanding, multimodal reasoning, and autonomous agents. As models grow more capable and efficient...",
            "logprobs": null,
            "finish_reason": "length"
        }
    ],
    "usage": {
        "prompt_tokens": 4,
        "total_tokens": 54,
        "completion_tokens": 50
    }
}

7 精度评估

以下是两种精度评估方法。

使用 AISBench

1. 详情请参考使用 AISBench。

2. 执行后可获取结果。以下为 vllm-ascend:v0.17.0rc1 中 Qwen3.5-27B-w8a8 的结果，仅供参考。Qwen3.6-27B-w8a8 的精度结果可通过相同方式获取，此处不再列出。

精度评估配置文件：

# Example configuration: benchmarks/ais_bench/benchmark/configs/models/vllm_api/vllm_api_general_chat.py
from ais_bench.benchmark.models import VLLMCustomAPIChat
from ais_bench.benchmark.utils.model_postprocessors import extract_non_reasoning_content

models = [
    dict(
        attr="service",
        type=VLLMCustomAPIChat,
        abbr="vllm-api-general-chat",
        path="your_model_path",
        model="qwen3.5",
        request_rate=0,
        retry=2,
        host_ip="127.0.0.1",
        host_port=8000,
        max_out_len=32768,
        batch_size=32,
        trust_remote_code=False,
        generation_kwargs=dict(
            temperature=1.0,
            top_p=0.95,
            top_k=20,
            min_p=0.0,
            presence_penalty=1.5,
            repetition_penalty=1.0,
            ignore_eos=False,
        ),
        pred_postprocessor=dict(type=extract_non_reasoning_content)
    )
]

对于 Qwen3.6-27B-w8a8，请将 model 改为 qwen3.6，并将 path 改为对应的模型权重路径。

数据集	版本	指标	模式	vllm-api-general-chat
gsm8k	-	accuracy	gen	96.74

使用 Language Model Evaluation Harness

以 gsm8k 数据集为例，在在线模式下对 Qwen3.5-27B-w8a8 进行精度评估。

1. lm_eval 的安装请参考使用 lm_eval。
2. 运行 lm_eval 执行精度评估。

# For Qwen3.5-27B-w8a8
export VLLM_USE_MODELSCOPE=True
vllm serve Eco-Tech/Qwen3.5-27B-w8a8-mtp \
    --served-model-name qwen3.5 \
    --trust-remote-code \
    --quantization ascend \
    --tensor-parallel-size 2 \
    --max-model-len 133000 \
    --max-num-seqs 32 \
    --gpu-memory-utilization 0.90 \
    --no-enable-prefix-caching

# Run lm_eval in another terminal
lm_eval \
  --model local-completions \
  --model_args model=qwen3.5,base_url=http://127.0.0.1:8000/v1/completions,tokenized_requests=False,trust_remote_code=True \
  --tasks gsm8k \
  --output_path ./

8 性能评估

使用 AISBench

详情请参考使用 AISBench 进行性能评估。

使用 vLLM Benchmark

以 Qwen3.5-27B-w8a8 或 Qwen3.6-27B-w8a8 为例进行性能评估。

更多详情请参考 vllm benchmark。

vllm bench 包含三个子命令：

• latency：对单批次请求的延迟进行基准测试。
• serve：对在线服务吞吐量进行基准测试。
• throughput：对离线推理吞吐量进行基准测试。

以 serve 为例，运行如下代码。

export VLLM_USE_MODELSCOPE=True
# For Qwen3.5-27B-w8a8:
vllm bench serve --model Eco-Tech/Qwen3.5-27B-w8a8-mtp --dataset-name random --random-input 200 --num-prompts 200 --request-rate 1 --save-result --result-dir ./
# For Qwen3.6-27B-w8a8:
vllm bench serve --model Eco-Tech/Qwen3.6-27B-w8a8 --dataset-name random --random-input 200 --num-prompts 200 --request-rate 1 --save-result --result-dir ./

大约几分钟后，即可获得性能评估结果。

9 性能调优

9.1 推荐配置

注意：以下配置在特定测试环境中验证，仅供参考。最佳配置取决于最大输入/输出长度、前缀缓存命中率、精度要求及部署机器配比等因素。建议根据实际情况参考第 9.2 节进行调优。

并行策略：Qwen3.5-27B-w8a8 和 Qwen3.6-27B-w8a8 仅约 30 GB，可轻松放入单个 NPU（每 NPU 64 GB HBM）。遵循 DP优先 原则，TP=2 是大多数场景的推荐默认值，其余 NPU 应分配给 DP 以并行处理请求批次。TP=8 仅推荐用于超长上下文（256K+）场景，此时将 KV 缓存分片到 8 个 NPU 上，以最大化每个 rank 的可用上下文窗口。对于 Qwen3.6-27B-w8a8，在相同的 TP/DP 布局下，还可以将 --max-model-len 提升至 262144。

Atlas inference products: Currently only the TP scenario is supported. Choose TP=2 or TP=4 according to the available devices. With TP=4, --max-model-len can support 128K and 256K long-sequence scenarios; configure --max-num-seqs as needed—setting it too high may cause OOM.

表 1：场景概览

场景	部署模式	*NPU总数	权重版本	关键考量
高吞吐量 （128K上下文）	单节点（A2）	8（A2）	Qwen3.5-27B-w8a8 / Qwen3.6-27B-w8a8	TP=2 + DP=4 充分利用所有8个NPU处理并行请求批次
高吞吐量 （128K上下文）	单节点（A3）	16（A3）	Qwen3.5-27B-w8a8 / Qwen3.6-27B-w8a8	TP=2 + DP=8 充分利用所有16个NPU处理并行请求批次
低延迟 （128K上下文）	单节点（A3）	16（A3）	Qwen3.5-27B-w8a8 / Qwen3.6-27B-w8a8	TP=2 + DP=8 减少小交互批次中每层Allreduce的开销
长上下文 （256K+上下文）	单节点（A3）	16（A3）	Qwen3.5-27B-w8a8 / Qwen3.6-27B-w8a8	TP=8 + DP=2 将KV缓存分片到8个NPU上，以最大化可用上下文窗口

*Total NPUs 表示所有节点使用的 NPU 总数。1 个 Atlas 800 A3 节点 = 16 个 NPU，1 个 Atlas 800 A2 节点 = 8 个 NPU。

表 2：详细节点配置

场景	配置	NPU数量	TP	DP	最大序列数	最大批处理Token数	最大模型长度	MTP推测数量	异步调度
High Throughput (128K)	Single-Node (A2)	8	2	4	32	16384	133000	3	On
High Throughput (128K)	Single-Node (A3)	16	2	8	32	16384	133000	3	On
Low Latency (128K)	Single-Node (A3)	16	2	8	4	4096	133000	3	On
Long Context (256K+)	Single-Node (A3)	16	8	2	8	8192	266000	3	On

完整的启动命令和参数说明请参考第5章中的部署示例。

9.2 调优指南

9.2.1 通用调优参考

调优方法请参考优化与调优。 详细特性描述请参考特性指南。

10 常见问题

常见环境、安装及通用参数问题，请参考vLLM-Ascend 公共常见问题。