DeepSeek-V4.1-Flash

1 简介

DeepSeek-V4.1-Flash 是一个原生多模态混合专家（MoE）模型，主干参数量为 552B，上下文长度最高可达 一百万个 token。它采用 40 层因果编码器-解码器（CED）架构，包含 20 层因果编码器 层和 20 层解码器层，在 Prefill 阶段每个 token 激活 8B 参数，在 Decode 阶段激活 16B 参数。

该模型引入了压缩稀疏注意力 2（CSA2）、FP4 主 KV 缓存、SWA 有界重放、 单遍 mHC、Engram 条件记忆以及 DSpark 投机解码。这些设计将全局 KV 缓存占用 降低至每个 token 890 字节，并将持久化 KV 缓存占用降低至 DeepSeek-V4-Flash 的 约八分之一。该模型接受文本和图像输入，并支持从 1 到 100 连续可调的推理强度。

vLLM Ascend supports W8A8 deployment on Atlas 800 A3 and A2 servers. This guide provides a single-node colocated A3 configuration and a two-node A3 Prefill-Decode (PD) disaggregated configuration.

DeepSeek-V4.1-Flash support requires the main branch of vLLM Ascend and an upstream vLLM revision dated September 11, 2026 or later. vLLM v0.30.0 is recommended.

2 支持的特性

完整支持矩阵请参阅支持的模型， 特性配置请参阅特性指南。

The A3 configurations in this guide use W8A8 weights and INT8 Engram storage. The single-node colocated configuration uses DP4/TP4. The PD configuration uses DP4/TP4 on the Prefill node, DP8/TP2 on the Decode node, DSpark speculative decoding, and FULL_DECODE_ONLY ACL Graph on Decode. The DeepSeek-V4.1-Flash model currently supports only model runner V1 on Ascend, so all A3 scripts set VLLM_USE_V2_MODEL_RUNNER=0 explicitly.

3 前提条件

3.1 模型权重与硬件

官方 DeepSeek-V4.1-Flash 权重可从 Hugging Face 和 ModelScope 获取。

The Ascend W8A8 checkpoint used by this guide will be published as Eco-Tech/DeepSeek-V4.1-Flash-w8a8 on ModelScope. It includes the DSpark draft parameters and INT8 Engram tables. After the checkpoint is available, download it to the same absolute path on every server and replace the checkpoint-path placeholder in each serving command.

或者，使用 ModelSlim 从官方权重 准备兼容 ModelSlim 的 W8A8 权重。

使用以下硬件配置之一：

• A3 single-node colocated: one Atlas 800 A3 server with 8 NPUs and 128GB memory per NPU. The server exposes 16 logical devices to the container.
• A3 1P1D: two Atlas 800 A3 servers with the same device configuration.
• A2 series: four Atlas 800 A2 servers. Each server has 8 NPUs with 64GB memory per NPU and exposes 8 devices to the container. Its deployment configuration is retained unchanged in this update.

将权重存储在共享目录中，或将其复制到每台服务器上的相同绝对路径。

3.2 验证多节点通信

部署前，请遵循 验证多节点通信。 所有服务器必须能够通过所选的网络接口进行通信，且服务端口不得被阻塞。

4 安装

4.1 Docker 镜像安装

选择目标硬件对应的标签页。A2 和 A3 使用不同的验证镜像。

A3 系列A2 系列

An A3 server exposes 16 logical devices. Run this command on every A3 server.

export IMAGE=quay.io/ascend/vllm-ascend:deepseek-v4.1-flash-a3
export MODEL_ROOT="/data/weights"

docker pull "$IMAGE"

docker run --rm -it \
  --name deepseek-v41 \
  --net=host \
  --ipc=host \
  --privileged=true \
  --device /dev/davinci0 \
  --device /dev/davinci1 \
  --device /dev/davinci2 \
  --device /dev/davinci3 \
  --device /dev/davinci4 \
  --device /dev/davinci5 \
  --device /dev/davinci6 \
  --device /dev/davinci7 \
  --device /dev/davinci8 \
  --device /dev/davinci9 \
  --device /dev/davinci10 \
  --device /dev/davinci11 \
  --device /dev/davinci12 \
  --device /dev/davinci13 \
  --device /dev/davinci14 \
  --device /dev/davinci15 \
  --device /dev/davinci_manager \
  --device /dev/devmm_svm \
  --device /dev/hisi_hdc \
  -v /usr/local/dcmi:/usr/local/dcmi \
  -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
  -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
  -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
  -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
  -v /etc/ascend_install.info:/etc/ascend_install.info \
  -v /etc/hccn.conf:/etc/hccn.conf \
  -v "$MODEL_ROOT:$MODEL_ROOT" \
  "$IMAGE" bash

一台 A2 服务器暴露 8 个设备。在全部四台 A2 服务器上运行此命令。

export IMAGE=quay.io/ascend/vllm-ascend:deepseek-v4.1-flash
export MODEL_ROOT="/data/weights"

docker pull "$IMAGE"

docker run --rm -it \
  --name deepseek-v41 \
  --net=host \
  --shm-size=512g \
  --privileged=true \
  --device /dev/davinci0 \
  --device /dev/davinci1 \
  --device /dev/davinci2 \
  --device /dev/davinci3 \
  --device /dev/davinci4 \
  --device /dev/davinci5 \
  --device /dev/davinci6 \
  --device /dev/davinci7 \
  --device /dev/davinci_manager \
  --device /dev/devmm_svm \
  --device /dev/hisi_hdc \
  -v /usr/local/dcmi:/usr/local/dcmi \
  -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
  -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
  -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
  -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
  -v /etc/ascend_install.info:/etc/ascend_install.info \
  -v /etc/hccn.conf:/etc/hccn.conf \
  -v "$MODEL_ROOT:$MODEL_ROOT" \
  "$IMAGE" bash

如果权重存储在其他位置，请修改 MODEL_ROOT。请确保每个容器内外使用相同的 绝对路径。

4.2 源码安装

如需从源码构建，请遵循 软件环境安装指南， 并使用 main 分支以及记录在 .github/vllm-main-verified.commit 中的匹配 vLLM 版本。

5 在线服务部署

5.1 Single-Node Colocated Deployment

A3 系列

This configuration runs Prefill and Decode on one Atlas 800 A3 server. It uses DP4/TP4 across all 16 logical devices, expert parallelism, Engram host offload, asynchronous scheduling, and FULL_DECODE_ONLY ACL Graph.

Set MODEL_PATH to the local checkpoint path.

#!/usr/bin/env bash

MODEL_PATH="<YOUR_MODEL_PATH>"

export VLLM_USE_V2_MODEL_RUNNER=0
export VLLM_ENGINE_READY_TIMEOUT_S=36000

export HCCL_BUFFSIZE=1024
export HCCL_OP_EXPANSION_MODE="AIV"

# Ensure the model path matches the directory recorded during download
vllm serve "$MODEL_PATH" \
    --host 0.0.0.0 \
    --port 8900 \
    --max-model-len 150000 \
    --max-num-batched-tokens 8192 \
    --served-model-name dsv41 \
    --gpu-memory-utilization 0.9 \
    --max-num-seqs 32 \
    --data-parallel-size 4 \
    --tensor-parallel-size 4 \
    --enable-expert-parallel \
    --tokenizer-mode deepseek_v41 \
    --reasoning-parser deepseek_v41 \
    --tool-call-parser deepseek_v41 \
    --enable-auto-tool-choice \
    --safetensors-load-strategy lazy \
    --model-loader-extra-config '{"enable_multithread_load":true,"num_threads":128}' \
    --quantization ascend \
    --block-size 128 \
    --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
    --async-scheduling \
    --engram-config '{"cpu_offload":true,"dp_shared_memory":true}' \
    --additional-config '{
        "ascend_compilation_config":{
            "enable_npugraph_ex":true,
            "enable_static_kernel":false
        },
        "enable_cpu_binding":true,
        "enable_fused_mc2":1,
        "enable_dsa_cp":true,
        "enable_flashcomm1":true,
        "enable_shared_expert_dp":true
    }'

Key parameters:

• --max-model-len 150000 limits the total input and output length of one request. --max-num-batched-tokens 8192 limits the tokens scheduled in one iteration, while --max-num-seqs 32 limits the sequences scheduled by each DP engine. Increasing either scheduler limit can improve throughput but also increases memory usage.
• VLLM_ENGINE_READY_TIMEOUT_S=36000 allows up to 36,000 seconds for engine processes to finish initialization, including weight loading and graph preparation.
• VLLM_USE_V2_MODEL_RUNNER=0 explicitly selects model runner V1. Keep this value at 0 because DeepSeek-V4.1-Flash currently does not support model runner V2 on Ascend.
• --engram-config '{"cpu_offload":true,"dp_shared_memory":true}' keeps the Engram table in host memory and lets local DP ranks share the host-memory allocation. This is required on A3 because the model's 24 Engram heads cannot be evenly divided across 16 logical devices. It also reduces duplicate host-memory copies. The A3 container example uses --ipc=host so all four local DP ranks share the host IPC namespace.
• --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' captures the Decode path while Prefill remains outside the captured graph.
• --safetensors-load-strategy lazy avoids eagerly materializing the whole checkpoint during loading.
• enable_fused_mc2, enable_dsa_cp, enable_flashcomm1, and enable_shared_expert_dp enable the A3 MoE and communication optimizations used by this configuration. enable_cpu_binding pins worker processes to CPUs, while enable_npugraph_ex enables the enhanced ACL Graph path.

5.2 1P1D PD Disaggregated Deployment

A3 系列

This example uses two Atlas 800 A3 servers. The Prefill node runs four DP ranks with TP4 (DP4/TP4), and the Decode node runs eight DP ranks with TP2 (DP8/TP2). Both layouts consume all 16 logical devices on their respective servers. Mooncake transfers KV cache from the Prefill engines to the Decode engines.

5.2.1 Prepare the DP Launcher

Save the following script as launch_online_dp.py on both nodes. It divides the node's visible devices among local DP ranks and starts one vLLM process per rank.

import argparse
import multiprocessing
import os
import subprocess
import sys


def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument("--dp-size", type=int, required=True)
    parser.add_argument("--tp-size", type=int, default=1)
    parser.add_argument("--dp-size-local", type=int, default=-1)
    parser.add_argument("--dp-rank-start", type=int, default=0)
    parser.add_argument("--dp-address", type=str, required=True)
    parser.add_argument("--dp-rpc-port", type=str, default="12345")
    parser.add_argument("--vllm-start-port", type=int, default=9000)
    return parser.parse_args()


args = parse_args()
dp_size = args.dp_size
tp_size = args.tp_size
dp_size_local = args.dp_size if args.dp_size_local == -1 else args.dp_size_local


def run_command(visible_devices, dp_rank, vllm_engine_port):
    command = [
        "bash",
        "./run_dp_template.sh",
        visible_devices,
        str(vllm_engine_port),
        str(dp_size),
        str(dp_rank),
        args.dp_address,
        args.dp_rpc_port,
        str(tp_size),
    ]
    subprocess.run(command, check=True)


if __name__ == "__main__":
    if not os.path.exists("./run_dp_template.sh"):
        print("Template file ./run_dp_template.sh does not exist.")
        sys.exit(1)

    processes = []
    for i in range(dp_size_local):
        dp_rank = args.dp_rank_start + i
        vllm_engine_port = args.vllm_start_port + i
        visible_devices = ",".join(
            str(device) for device in range(i * tp_size, (i + 1) * tp_size)
        )
        process = multiprocessing.Process(
            target=run_command,
            args=(visible_devices, dp_rank, vllm_engine_port),
        )
        processes.append(process)
        process.start()

    for process in processes:
        process.join()

The launcher arguments are:

Parameter	Description
--dp-size	Global DP size within the Prefill or Decode node group.
--tp-size	Number of logical devices used by each DP rank.
--dp-size-local	Number of DP ranks started on the current node.
--dp-rank-start	First DP rank assigned to this node. It is 0 for both single-node groups in this 1P1D example.
--dp-address	IP address of the node that coordinates the corresponding DP group. Use the Prefill IP on the Prefill node and the Decode IP on the Decode node.
--dp-rpc-port	DP coordination port. It must be unused and reachable within the node group.
--vllm-start-port	First API port; the launcher increments it for each local DP rank.

5.2.2 Start the Prefill Node

On the Prefill node, save the following script as run_dp_template.sh. Set LOCAL_IP, NIC_NAME, and MODEL_PATH to the Prefill node's service IP, network interface, and local checkpoint path.

#!/usr/bin/env bash

unset https_proxy
unset http_proxy
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH:+$LD_LIBRARY_PATH:}/usr/local/lib/"

NIC_NAME="<NETWORK_INTERFACE>"
LOCAL_IP="<PREFILL_NODE_IP>"
MODEL_PATH="<YOUR_MODEL_PATH>"

export HCCL_IF_IP="$LOCAL_IP"
export GLOO_SOCKET_IFNAME="$NIC_NAME"
export TP_SOCKET_IFNAME="$NIC_NAME"
export HCCL_SOCKET_IFNAME="$NIC_NAME"
export VLLM_USE_V2_MODEL_RUNNER=0
export VLLM_ENGINE_READY_TIMEOUT_S=36000
export HCCL_BUFFSIZE=1024
export HCCL_OP_EXPANSION_MODE="AIV"
export ASCEND_RT_VISIBLE_DEVICES=$1

# Ensure the model path matches the directory recorded during download
vllm serve "$MODEL_PATH" \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name dsv41 \
    --max-model-len 1048576 \
    --max-num-batched-tokens 8192 \
    --max-num-seqs 16 \
    --speculative-config '{"num_speculative_tokens":5,"method":"dspark"}' \
    --trust-remote-code \
    --block-size 128 \
    --tokenizer-mode deepseek_v41 \
    --reasoning-parser deepseek_v41 \
    --tool-call-parser deepseek_v41 \
    --enable-auto-tool-choice \
    --model-loader-extra-config '{"enable_multithread_load":true,"num_threads":128}' \
    --safetensors-load-strategy lazy \
    --gpu-memory-utilization 0.9 \
    --quantization ascend \
    --enforce-eager \
    --enable-prefix-caching \
    --engram-config '{"cpu_offload":true,"dp_shared_memory":true}' \
    --additional-config '{
        "enable_cpu_binding":true,
        "enable_fused_mc2":1,
        "enable_dsa_cp":true,
        "enable_flashcomm1":true,
        "enable_shared_expert_dp":true
    }' \
    --kv-transfer-config '{
        "kv_connector":"MooncakeHybridConnector",
        "kv_role":"kv_producer",
        "kv_port":"30000",
        "engine_id":"0",
        "kv_connector_extra_config":{
            "prefill":{"dp_size":4,"tp_size":4},
            "decode":{"dp_size":8,"tp_size":2}
        }
    }'

Start four DP4/TP4 Prefill engines. --dp-address uses the Prefill node IP.

PREFILL_NODE_IP="<PREFILL_NODE_IP>"

python launch_online_dp.py \
    --dp-size 4 \
    --tp-size 4 \
    --dp-size-local 4 \
    --dp-rank-start 0 \
    --dp-address "$PREFILL_NODE_IP" \
    --dp-rpc-port 12321 \
    --vllm-start-port 7100

5.2.3 Start the Decode Node

On the Decode node, save the following script as run_dp_template.sh. Set LOCAL_IP, NIC_NAME, and MODEL_PATH to the Decode node's service IP, network interface, and the same local checkpoint path used on the Prefill node.

#!/usr/bin/env bash

unset https_proxy
unset http_proxy
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH:+$LD_LIBRARY_PATH:}/usr/local/lib/"

NIC_NAME="<NETWORK_INTERFACE>"
LOCAL_IP="<DECODE_NODE_IP>"
MODEL_PATH="<YOUR_MODEL_PATH>"

export VLLM_USE_V2_MODEL_RUNNER=0
export VLLM_ENGINE_READY_TIMEOUT_S=36000
export HCCL_IF_IP="$LOCAL_IP"
export GLOO_SOCKET_IFNAME="$NIC_NAME"
export TP_SOCKET_IFNAME="$NIC_NAME"
export HCCL_SOCKET_IFNAME="$NIC_NAME"
export HCCL_BUFFSIZE=1800
export ASCEND_RT_VISIBLE_DEVICES=$1

# Ensure the model path matches the directory recorded during download
vllm serve "$MODEL_PATH" \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name dsv41 \
    --max-model-len 1048576 \
    --max-num-batched-tokens 400 \
    --max-num-seqs 32 \
    --async-scheduling \
    --block-size 128 \
    --no-enable-prefix-caching \
    --trust-remote-code \
    --tokenizer-mode deepseek_v41 \
    --reasoning-parser deepseek_v41 \
    --tool-call-parser deepseek_v41 \
    --enable-auto-tool-choice \
    --model-loader-extra-config '{"enable_multithread_load":true,"num_threads":128}' \
    --safetensors-load-strategy lazy \
    --gpu-memory-utilization 0.95 \
    --quantization ascend \
    --speculative-config '{"num_speculative_tokens":5,"method":"dspark","enforce_eager":true}' \
    --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
    --kv-transfer-config '{
        "kv_connector":"MooncakeHybridConnector",
        "kv_role":"kv_consumer",
        "kv_port":"30100",
        "engine_id":"1",
        "kv_connector_extra_config":{
            "prefill":{"dp_size":4,"tp_size":4},
            "decode":{"dp_size":8,"tp_size":2}
        }
    }' \
    --engram-config '{"cpu_offload":true,"dp_shared_memory":true}' \
    --additional-config '{
        "ascend_compilation_config":{
            "enable_npugraph_ex":true,
            "enable_static_kernel":false
        },
        "enable_cpu_binding":true,
        "multistream_overlap_shared_expert":true,
        "recompute_scheduler_enable":true
    }'

Start eight DP8/TP2 Decode engines. --dp-address uses the Decode node IP.

DECODE_NODE_IP="<DECODE_NODE_IP>"

python launch_online_dp.py \
    --dp-size 8 \
    --tp-size 2 \
    --dp-size-local 8 \
    --dp-rank-start 0 \
    --dp-address "$DECODE_NODE_IP" \
    --dp-rpc-port 12321 \
    --vllm-start-port 7100

5.2.4 Deploy the PD Proxy

After all Prefill and Decode engines are ready, deploy the proxy as described in Prefill-Decode Disaggregation (DeepSeek). Configure the proxy with Prefill endpoints <PREFILL_NODE_IP>:7100 through <PREFILL_NODE_IP>:7103 and Decode endpoints <DECODE_NODE_IP>:7100 through <DECODE_NODE_IP>:7107.

5.2.5 Key Parameter Descriptions

• VLLM_ENGINE_READY_TIMEOUT_S=36000 gives every Prefill and Decode engine up to 36,000 seconds to finish startup. This includes weight loading and Decode graph preparation.
• VLLM_USE_V2_MODEL_RUNNER=0 explicitly selects model runner V1 on both Prefill and Decode. Keep this value at 0 because DeepSeek-V4.1-Flash currently does not support model runner V2 on Ascend.
• --data-parallel-size and --tensor-parallel-size define DP4/TP4 on Prefill and DP8/TP2 on Decode. Their product must be 16 on each A3 node.
• --data-parallel-address and --data-parallel-rpc-port coordinate DP ranks within one role. Prefill and Decode use their respective node IPs; port 12321 can be reused because the roles run on different hosts.
• --vllm-start-port 7100 assigns API ports 7100-7103 on Prefill and 7100-7107 on Decode. These endpoints must be reachable by the PD proxy.
• Both roles use --max-model-len 1048576. Prefill uses --max-num-batched-tokens 8192 and --max-num-seqs 16 to favor prompt throughput, while Decode uses 400 and 32 respectively to favor decode concurrency. Tune these role-specific scheduler limits independently.
• MooncakeHybridConnector transfers KV cache between the two roles. kv_role must be kv_producer on Prefill and kv_consumer on Decode; kv_port must be reachable and must not conflict with another service.
• kv_connector_extra_config must match the actual global layouts on both sides: Prefill DP4/TP4 and Decode DP8/TP2. Keep the values identical in the Prefill and Decode commands.
• Both roles use DSpark with five speculative tokens. Prefill runs the target model in eager mode. Decode uses FULL_DECODE_ONLY for the target model, while enforce_eager inside --speculative-config applies to the Decode draft model.
• --engram-config '{"cpu_offload":true,"dp_shared_memory":true}' keeps the Engram table in host memory and lets local DP ranks share the host-memory allocation. dp_shared_memory is required on A3 because the model's 24 Engram heads cannot be evenly divided across 16 logical devices. Both A3 containers use --ipc=host so the local DP ranks can share the allocation.
• --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' captures the Decode path while Prefill remains outside the captured graph.
• Prefix caching is enabled only on Prefill. Decode disables it and enables recompute_scheduler_enable so Prefill can recompute unavailable KV cache.

• Prefill uses the following --additional-config options:

  • enable_cpu_binding: true enables Ascend-native CPU affinity for worker processes and runtime threads.
  • enable_fused_mc2: 1 enables the fused MoE communication and computation path to reduce expert dispatch, FFN, and combine overhead.
  • enable_dsa_cp: true enables context parallelism for DeepSeek Sparse Attention. This option applies to models with an indexer and requires the sequence-parallel MoE path used by this topology.
  • enable_flashcomm1: true enables the FlashComm communication backend used by the sequence-parallel MoE and DSA context-parallel paths.
  • enable_shared_expert_dp: true uses data parallelism for shared experts. It takes effect only when expert parallelism is enabled and TP is greater than one; both conditions are met by Prefill DP4/TP4.

• Decode uses the following --additional-config options:

  • ascend_compilation_config.enable_npugraph_ex: true enables the enhanced ACL Graph compilation and execution path used by FULL_DECODE_ONLY.
  • ascend_compilation_config.enable_static_kernel: false disables static kernel generation while retaining the enhanced ACL Graph path.
  • enable_cpu_binding: true enables Ascend-native CPU affinity for worker processes and runtime threads.
  • multistream_overlap_shared_expert: true overlaps shared-expert execution with routed-expert computation on separate streams to improve Decode MoE throughput.
  • recompute_scheduler_enable: true enables the PD Decode recomputation scheduler. When the Decode-side KV cache is unavailable, the request can be sent back to Prefill to recompute it. Enable this option only on Decode.

The Prefill and Decode configurations are role-specific. Do not copy the Prefill fused-MC2 and DSA options to Decode, or the Decode graph, multistream, and recomputation options to Prefill, without validating the resulting topology and performance.

HCCL_IF_IP and the socket interface variables must select the same high-speed service network used by the configured node IPs. HCCL_BUFFSIZE is 1024 on Prefill and 1800 on Decode; retain these role-specific values unless another setting has been validated. The DP RPC, engine, Mooncake, and proxy ports must be allowed by the host firewall.

Wait until every engine finishes loading weights and Decode finishes graph capture. A successful startup includes output similar to:

INFO:     Started server process
INFO:     Waiting for application startup.
INFO:     Application startup complete.

5.3 Multi-Node Colocated Deployment

A2 系列

The existing A2 configuration is retained unchanged in this update. It uses four Atlas 800 A2 servers with one local DP rank per server and a global DP4/TP8/EP32 topology.

在全部四台 A2 服务器上运行此脚本。在对应节点上将 NODE_RANK 设置为 0、1、2 或 3。 Set NODE0_IP to the IP address of Node 0 and set LOCAL_IP, NIC_NAME, and MODEL_PATH for each node.

#!/usr/bin/env bash
set -euo pipefail

NODE_RANK=0
NODE0_IP="<NODE0_IP>"
LOCAL_IP="<LOCAL_IP>"
NIC_NAME="<NETWORK_INTERFACE>"
MODEL_PATH="<YOUR_MODEL_PATH>"

# Allow time for weight loading and graph capture on large models.
export VLLM_ENGINE_READY_TIMEOUT_S="${VLLM_ENGINE_READY_TIMEOUT_S:-3600}"
export HCCL_IF_IP="$LOCAL_IP"
export GLOO_SOCKET_IFNAME="$NIC_NAME"
export TP_SOCKET_IFNAME="$NIC_NAME"
export HCCL_SOCKET_IFNAME="$NIC_NAME"
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

HEADLESS_ARGS=()
if [[ "$NODE_RANK" != "0" ]]; then
  HEADLESS_ARGS+=(--headless --data-parallel-start-rank "$NODE_RANK")
fi

vllm serve "$MODEL_PATH" \
  --host 0.0.0.0 \
  --port 8000 \
  "${HEADLESS_ARGS[@]}" \
  --data-parallel-address "$NODE0_IP" \
  --data-parallel-rpc-port 13399 \
  --data-parallel-size 4 \
  --data-parallel-size-local 1 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --served-model-name deepseek-v41 \
  --max-model-len 1048576 \
  --max-num-batched-tokens 4096 \
  --max-num-seqs 32 \
  --gpu-memory-utilization 0.90 \
  --block-size 128 \
  --tokenizer-mode deepseek_v41 \
  --reasoning-parser deepseek_v41 \
  --tool-call-parser deepseek_v41 \
  --enable-auto-tool-choice \
  --trust-remote-code \
  --model-loader-extra-config '{"enable_multithread_load":true,"num_threads":128}' \
  --safetensors-load-strategy lazy \
  --quantization ascend \
  --additional-config '{"enable_cpu_binding":true,"ascend_compilation_config":{"enable_npugraph_ex":false,"enable_static_kernel":false}}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":5,"enforce_eager":true}' \
  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'

Omit --data-parallel-start-rank on Node 0. Start Node 0 first, followed by Nodes 1 through 3. Only Node 0 exposes the API endpoint. Each server contributes one TP8 rank to the global DP4 topology.

5.4 Service Verification

Set the endpoint for the selected deployment, then verify the health endpoint. Use <A3_IP> with port 8900 for A3 single-node colocated deployment, <PROXY_IP> with <PROXY_PORT> for A3 PD deployment, or <A2_NODE0_IP> with port 8000 for the retained A2 deployment.

export SERVICE_URL="http://<SERVICE_IP>:<SERVICE_PORT>"

curl -sS -o /dev/null -w 'HTTP %{http_code}\n' \
  "$SERVICE_URL/health"

预期输出：

HTTP 200

然后验证已配置的模型是否可用：

curl -sS "$SERVICE_URL/v1/models" | \
  jq '{object, models: [.data[] | {id, object}]}'

The response must contain a model entry whose id matches the configured --served-model-name: dsv41 for the A3 examples and deepseek-v41 for the retained A2 example.

6 功能验证

6.1 文本请求

export SERVICE_URL="http://<SERVICE_IP>:<SERVICE_PORT>"

curl -sS "$SERVICE_URL/v1/chat/completions" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "dsv41",
    "messages": [{"role": "user", "content": "Who are you?"}],
    "temperature": 0,
    "max_completion_tokens": 256
  }' | jq -e '.choices[0].message.content | length > 0'

预期输出：

true

6.2 图像请求

Set IMAGE_URL to an HTTP(S) image URL reachable from the node that executes Prefill, and send a multimodal request:

export IMAGE_URL="<YOUR_IMAGE_URL>"
export SERVICE_URL="http://<SERVICE_IP>:<SERVICE_PORT>"

curl -sS "$SERVICE_URL/v1/chat/completions" \
  -H 'Content-Type: application/json' \
  -d "{
    \"model\": \"dsv41\",
    \"messages\": [{
      \"role\": \"user\",
      \"content\": [
        {\"type\": \"image_url\", \"image_url\": {\"url\": \"${IMAGE_URL}\"}},
        {\"type\": \"text\", \"text\": \"Describe this image.\"}
      ]
    }],
    \"temperature\": 0,
    \"max_completion_tokens\": 256
  }" | jq -e '.choices[0].message.content | length > 0'

预期输出：

true

7 精度评估

请参考 使用 AISBench 来评估已部署的服务。目前尚未针对此配置发布 vLLM Ascend 任务级精度结果。 在报告结果时，请记录 checkpoint、prompt encoder、reasoning effort、采样参数、数据集 版本以及是否启用了 DSpark。

8 性能评估

请参考 AISBench 性能评估指南 或 vLLM 基准测试指南。 目前尚未针对此配置发布生产性能基线。

9 性能调优

The values in Sections 5.1 and 5.2 are starting points rather than globally optimal settings. Tune --max-num-seqs, --max-num-batched-tokens, and --gpu-memory-utilization together for the target input length, image sizes, output length, and concurrency. Keep DP4/TP4 for A3 single-node deployment, or Prefill DP4/TP4 and Decode DP8/TP2 for A3 PD deployment, until an alternative configuration has been validated.

10 常见问题

如何启用工具调用和推理解析？

请在服务命令中保留以下选项：

--tokenizer-mode deepseek_v41 \
--tool-call-parser deepseek_v41 \
--reasoning-parser deepseek_v41 \
--enable-auto-tool-choice

有关常见的环境、安装和参数问题，请参考 公共常见问题。

11 限制

• The documented A3 deployments use either one server in colocated mode or two servers in 1P1D mode, with an Ascend W8A8 checkpoint and INT8 Engram storage.
• The A2 configuration is retained unchanged and is not revalidated by this update. Its revised configuration will be documented separately.
• DeepSeek-V4.1-Flash currently supports only model runner V1 on Ascend. Keep VLLM_USE_V2_MODEL_RUNNER=0 in every A3 serving script. Pipeline parallelism is not covered by this guide.
• In the A3 PD example, Prefill runs in eager mode. Decode uses FULL_DECODE_ONLY ACL Graph for the target model and eager execution for the DSpark draft model.
• Production performance qualification and task-level accuracy evaluation are not complete.

12 Bounded Replay Compatibility for vLLM 0.30.0

For vLLM 0.30.0 images without the upstream replay protocol, the Ascend plugin backports configuration, request payloads, and local prefix-hit admission logic. Installed vLLM files and compiled operators remain unchanged. The compatibility path is disabled by default. Enable it explicitly with:

--enable-prefix-caching --swa-bounded-replay

Use --no-swa-bounded-replay to disable replay explicitly. vLLM builds with native replay use the upstream implementation. Compatibility is detected from the replay interfaces rather than a version string. Configuration and wire fields are registered before use; scheduler and cache behavior is patched only when replay is enabled. Unsupported scheduler source bodies fail explicitly. The backport supports local prefix caching only; enabling replay with a KV connector raises an error. Existing replay restrictions still apply to PCP and Ascend modes that use independent scheduler copies.

SWA caches do not participate in prefix reuse. After a hit in another cache group, the scheduler recomputes the final SWA window of the cached prefix. The worker masks writes to shared cache groups during replay and clamps the visible SWA range. Hits no longer than the window are recomputed in full. After preemption, readmission recalculates and forwards the replay boundary.

12.1 Real-Weight Validation

The following A5 regression configuration uses TP1/DP4/EP4. Use the existing CANN environment and compiled Ascend operators before running it:

cd /workspace
VLLM_USE_V2_MODEL_RUNNER=1 vllm serve /home/weights/deepseek-ai--DeepSeek-V4.1-Flash \
  --tokenizer-mode deepseek_v41 --served-model-name deepseek-v41-accuracy \
  --host 127.0.0.1 --port 8011 \
  --tensor-parallel-size 1 --data-parallel-size 4 --data-parallel-size-local 4 \
  --enable-expert-parallel --quantization deepseek_v4_fp8 \
  --max-model-len 16384 --max-num-batched-tokens 2048 --max-num-seqs 1 \
  --block-size 128 --enforce-eager --enable-prefix-caching --swa-bounded-replay \
  --limit-mm-per-prompt '{"image":0}' \
  --engram-config '{"cpu_offload":true,"dp_shared_memory":false}' \
  --model-loader-extra-config '{"enable_multithread_load":true,"num_threads":16}' \
  --safetensors-load-strategy lazy --additional-config '{"enable_cpu_binding":false}'

First save a baseline from a replay-disabled service, then send the same requests to a replay-enabled service:

cd /path/to/vllm-ascend
python tests/e2e/models/swa_bounded_replay_probe.py \
  --base-url http://127.0.0.1:8011 --model deepseek-v41-accuracy \
  --output /tmp/replay-disabled.json

# Restart the service with --swa-bounded-replay before running this command.
python tests/e2e/models/swa_bounded_replay_probe.py \
  --base-url http://127.0.0.1:8011 --model deepseek-v41-accuracy \
  --baseline /tmp/replay-disabled.json --output /tmp/replay-enabled.json

Validation requires successful HTTP requests, outputs matching the baseline, and an actual SWA bounded replay is active message in the service log. Startup success or a true configuration field alone does not prove replay was exercised. Cross-chunk regression should also use a token budget smaller than the SWA window.