跳转至

DeepSeek-V4-Pro

1.简介

DeepSeek-V4在DeepSeek-V3基础上引入了多项关键升级：

  • 流形约束超连接(mHC)增强传统残差连接。
  • 混合注意力架构，通过Compress-4-Attention和Compress-128-Attention大幅提升长上下文效率。对于混合专家(MoE)组件，仍采用DeepSeekMoE架构，仅做少量调整。

DeepSeek-V4-Pro作为DeepSeek-V4的最大推理努力模式，显著提升了开源模型的知识能力，牢固确立了其作为当今最佳开源模型的地位。它在编码基准测试中达到顶级性能，并在推理和智能体任务上显著缩小了与领先闭源模型的差距。

本文档将展示该模型的主要验证步骤，包括支持特性、特性配置、环境准备、单节点和多节点部署、精度及性能评估。

注意：请将上述版本占位符替换为您实际的验证版本。

2.支持特性

参考支持特性获取模型的支持特性矩阵。

参考特性指南获取特性的配置。

3.前提条件

3.1模型权重

  • DeepSeek-V4-Pro-w4a8-mtp(量化版本)：需要2个Atlas 800 A3(128G × 8)节点或4个Atlas 800 A2(64G × 8)节点。下载模型权重

建议将模型权重下载到多节点共享目录，例如/root/.cache/。

3.2验证多节点通信(可选)

如需部署多节点环境，需按照验证多节点通信环境验证多节点通信。

4.安装

4.1Docker镜像安装

根据机器类型选择镜像并在节点上启动docker镜像，参考使用docker。

在每个节点上启动docker镜像。

export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1-a3
docker run --rm \
    --name vllm-ascend \
    --shm-size=512g \
    --net=host \
    --privileged=true \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci8 \
    --device /dev/davinci9 \
    --device /dev/davinci10 \
    --device /dev/davinci11 \
    --device /dev/davinci12 \
    --device /dev/davinci13 \
    --device /dev/davinci14 \
    --device /dev/davinci15 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /etc/hccn.conf:/etc/hccn.conf \
    -v /root/.cache:/root/.cache \
    -it $IMAGE bash

Start the docker image on each node.

export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1
docker run --rm \
    --name vllm-ascend \
    --shm-size=512g \
    --net=host \
    --privileged=true \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /etc/hccn.conf:/etc/hccn.conf \
    -v /root/.cache:/root/.cache \
    -it $IMAGE bash

After a successful docker run, you can verify the running container service by executing the docker ps command.

4.2 Source Code Installation

If you don't want to use the docker image as above, you can also build all from source:

If you want to deploy a multi-node environment, you need to set up the environment on each node.

5 Online Service Deployment

Note

In this tutorial, we suppose you downloaded the model weight to /root/.cache/模式lscope/hub/模式ls/vllm-ascend/. Feel free to change it to your own path.

It is recommended that the following service code be encapsulated in a .sh script file and executed in Bash mode.

5.1 Multi-Node Online Deployment

The quantized model DeepSeek-V4-Pro-w4a8-mtp requires at least 2 Atlas 800 A3 (128G × 8) nodes or 4 Atlas 800 A2 (64G × 8) nodes. Run the following scripts on each node respectively.

Node0

local_ip="xxx"
node0_ip="xxxx"

export HCCL_IF_IP=$local_ip
export IFNAME="xxx"
export GLOO_SOCKET_IFNAME="$IFNAME"
export TP_SOCKET_IFNAME="$IFNAME"
export HCCL_SOCKET_IFNAME="$IFNAME"
export HCCL_BUFFSIZE=512
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export ACL_OP_INIT_MODE=1
export VLLM_ENGINE_READY_TIMEOUT_S=3600
export HCCL_OP_EXPANSION_MODE="AIV"

export TASK_QUEUE_ENABLE=1
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1

export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD

export HCCL_CONNECT_TIMEOUT=7200
export ASCEND_CONNECT_TIMEOUT=10000
export ASCEND_TRANSFER_TIMEOUT=10000
export VLLM_RPC_TIMEOUT=1800000

vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Pro-w4a8-mtp \
  --host 0.0.0.0 \
  --port 10010 \
  --max-model-len 135000 \
  --max-num-batched-tokens 4096 \
  --served-model-name dsv4 \
  --gpu-memory-utilization 0.9 \
  --max-num-seqs 16 \
  --data-parallel-size 4 \
  --tensor-parallel-size 8 \
  --data-parallel-size-local 1 \
  --data-parallel-start-rank 0 \
  --data-parallel-address $node0_ip  \
  --enable-expert-parallel \
  --quantization ascend \
  --no-enable-prefix-caching \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4 \
  --async-scheduling \
  --safetensors-load-strategy 'prefetch' \
  --block-size 128 \
  --speculative-config '{
     "num_speculative_tokens": 1,
     "method": "mtp",
     "enforce_eager": true
  }' \
  --additional-config '{
     "ascend_compilation_config":{
        "enable_npugraph_ex":true,
        "enable_static_kernel":false
     },
     "enable_cpu_binding": true,
     "enable_shared_expert_dp": true,
     "multistream_overlap_shared_expert":true
  }' \
  --compilation-config '{
     "cudagraph_mode":"FULL_DECODE_ONLY"
  }' \
  --model-loader-extra-config '{
     "enable_multithread_load": "true",
     "num_threads": 128
  }'

Node1-Node3

local_ip="xxx"
node0_ip="xxxx"

export HCCL_IF_IP=$local_ip
export IFNAME="xxx"
export GLOO_SOCKET_IFNAME="$IFNAME"
export TP_SOCKET_IFNAME="$IFNAME"
export HCCL_SOCKET_IFNAME="$IFNAME"
export HCCL_BUFFSIZE=512
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export ACL_OP_INIT_MODE=1
export VLLM_ENGINE_READY_TIMEOUT_S=3600
export HCCL_OP_EXPANSION_MODE="AIV"

export TASK_QUEUE_ENABLE=1
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1

export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD

export HCCL_CONNECT_TIMEOUT=7200
export ASCEND_CONNECT_TIMEOUT=10000
export ASCEND_TRANSFER_TIMEOUT=10000
export VLLM_RPC_TIMEOUT=1800000

vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Pro-w4a8-mtp \
  --host 0.0.0.0 \
  --port 10010 \
  --max-model-len 135000 \
  --max-num-batched-tokens 4096 \
  --served-model-name dsv4 \
  --gpu-memory-utilization 0.9 \
  --max-num-seqs 16 \
  --data-parallel-size 4 \
  --tensor-parallel-size 8 \
  --data-parallel-size-local 1 \
  --data-parallel-start-rank 1 \
  --data-parallel-address $node0_ip  \
  --enable-expert-parallel \
  --quantization ascend \
  --no-enable-prefix-caching \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4 \
  --async-scheduling \
  --safetensors-load-strategy 'prefetch' \
  --block-size 128 \
  --headless \
  --speculative-config '{
     "num_speculative_tokens": 1,
     "method": "mtp",
     "enforce_eager": true
  }' \
  --additional-config '{
     "ascend_compilation_config":{
        "enable_npugraph_ex":true,
        "enable_static_kernel":false
     },
     "enable_cpu_binding": true,
     "enable_shared_expert_dp": true,
     "multistream_overlap_shared_expert":true
  }' \
  --compilation-config '{
     "cudagraph_mode":"FULL_DECODE_ONLY"
  }' \
  --model-loader-extra-config '{
     "enable_multithread_load": "true",
     "num_threads": 128
  }'

Node0

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="xxx"
local_ip="xxx"

# The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)
node0_ip="xxxx"

export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export HCCL_BUFFSIZE=2048
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export TASK_QUEUE_ENABLE=1
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1

vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Pro-w4a8-mtp \
  --safetensors-load-strategy 'prefetch' \
  --max-model-len 135000  \
  --max-num-batched-tokens 4096 \
  --served-model-name dsv4 \
  --gpu-memory-utilization 0.9 \
  --max-num-seqs 32 \
  --data-parallel-size 2 \
  --data-parallel-size-local 1 \
  --data-parallel-start-rank 0 \
  --data-parallel-address $node0_ip \
  --data-parallel-rpc-port 13399 \
  --tensor-parallel-size 16 \
  --enable-expert-parallel \
  --quantization ascend \
  --port 8900 \
  --host 0.0.0.0 \
  --block-size 128 \
  --async-scheduling \
  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4 \
  --speculative-config '{"num_speculative_tokens": 1,"method": "mtp","enforce_eager": true}' \
  --additional-config '
    {"ascend_compilation_config":{
        "enable_npugraph_ex":true,
        "enable_static_kernel":false
        },
    "enable_cpu_binding": true,
    "multistream_overlap_shared_expert":true}'

Node1

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="xxx"
local_ip="xxx"

# The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)
node0_ip="xxxx"

export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export HCCL_BUFFSIZE=2048
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export TASK_QUEUE_ENABLE=1
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1

vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Pro-w4a8-mtp \
  --safetensors-load-strategy 'prefetch' \
  --max-model-len 135000  \
  --max-num-batched-tokens 4096 \
  --served-model-name dsv4 \
  --gpu-memory-utilization 0.9 \
  --max-num-seqs 32 \
  --data-parallel-size 2 \
  --data-parallel-size-local 1 \
  --data-parallel-start-rank 1 \
  --data-parallel-address $node0_ip \
  --data-parallel-rpc-port 13399 \
  --headless \
  --tensor-parallel-size 16 \
  --enable-expert-parallel \
  --quantization ascend \
  --port 8900 \
  --host 0.0.0.0 \
  --block-size 128 \
  --async-scheduling \
  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4 \
  --speculative-config '{"num_speculative_tokens": 1,"method": "mtp","enforce_eager": true}' \
  --additional-config '
    {"ascend_compilation_config":{
        "enable_npugraph_ex":true,
        "enable_static_kernel":false
        },
    "enable_cpu_binding": true,
    "multistream_overlap_shared_expert":true}'

Key Parameter Descriptions:

  • --data-parallel-start-rank specifies the starting data parallel rank of the current node. Each node must be set to a unique value (e.g., Node0 = 0, Node1 = 1).
  • --data-parallel-address specifies the IP address of the data parallel master node (Node0). It must be consistent across all nodes.
  • --headless (used on non-master nodes) disables the API server on the node, since only the master node serves requests.
  • --max-模式l-len specifies the maximum context length. Adjust it according to your actual scenario.
  • --speculative-config configures the MTP (Multi-Token Prediction) speculative decoding to accelerate inference.
  • --compilation-config '{"cudagraph_模式":"FULL_DECODE_ONLY"}' enables full ACL graph execution in the decode phase to reduce scheduling latency.
  • --async-scheduling enables asynchronous scheduling to overlap CPU scheduling with NPU computation.
  • VLLM_ASCEND_ENABLE_FLASHCOMM1=1启用FlashComm通信优化。

Common Issues Tip: If you encounter issues, please refer to the Public FAQ for troubleshooting.

服务验证：

curl http://<node0_ip>:8900/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "dsv4",
        "messages": [
            {
                "role": "user",
                "content": "Who are you?"
            }
        ],
        "max_tokens": 256,
        "temperature": 0
    }'

预期结果：

服务返回HTTP 200 OK，JSON响应中包含choices字段。

5.2多节点PD分离部署

我们推荐使用Mooncake进行部署：Mooncake。

在标准部署模式下，Prefill(提示处理)和Decode(令牌生成)任务在同一组NPU上运行。PD(Prefill-Decode)分离通过在专用节点组上分别运行Prefill和Decode来解决此问题，每个节点组独立配置。此架构推荐用于具有并发多用户工作负载的生产部署，其中需要稳定的延迟和高吞吐量。

以下章节描述在Atlas 800 A3(128G × 8)和Atlas 800 A2(64G × 8)多节点环境上的PD分离部署。

5.2.1A3系列PD分离部署

本节展示在Atlas 800 A3(128G × 8)多节点环境中使用1P1D部署DeepSeek-V4-Pro的指南，以获得更佳性能。

开始前，请：

  1. 在每个节点上准备脚本launch_online_dp.py。

    import argparse
import multiprocessing
import os
import subprocess
import sys

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--dp-size",
        type=int,
        required=True,
        help="Data parallel size."
    )
    parser.add_argument(
        "--tp-size",
        type=int,
        default=1,
        help="Tensor parallel size."
    )
    parser.add_argument(
        "--dp-size-local",
        type=int,
        default=-1,
        help="Local data parallel size."
    )
    parser.add_argument(
        "--dp-rank-start",
        type=int,
        default=0,
        help="Starting rank for data parallel."
    )
    parser.add_argument(
        "--dp-address",
        type=str,
        required=True,
        help="IP address for data parallel master node."
    )
    parser.add_argument(
        "--dp-rpc-port",
        type=str,
        default=12345,
        help="Port for data parallel master node."
    )
    parser.add_argument(
        "--vllm-start-port",
        type=int,
        default=9000,
        help="Starting port for the engine."
    )
    return parser.parse_args()

args = parse_args()
dp_size = args.dp_size
tp_size = args.tp_size
dp_size_local = args.dp_size_local
if dp_size_local == -1:
    dp_size_local = dp_size
dp_rank_start = args.dp_rank_start
dp_address = args.dp_address
dp_rpc_port = args.dp_rpc_port
vllm_start_port = args.vllm_start_port

def run_command(visible_devices, dp_rank, vllm_engine_port):
    command = [
        "bash",
        "./run_dp_template.sh",
        visible_devices,
        str(vllm_engine_port),
        str(dp_size),
        str(dp_rank),
        dp_address,
        dp_rpc_port,
        str(tp_size),
    ]
    subprocess.run(command, check=True)

if __name__ == "__main__":
    template_path = "./run_dp_template.sh"
    if not os.path.exists(template_path):
        print(f"Template file {template_path} does not exist.")
        sys.exit(1)

    processes = []
    num_cards = dp_size_local * tp_size
    for i in range(dp_size_local):
        dp_rank = dp_rank_start + i
        vllm_engine_port = vllm_start_port + i
        visible_devices = ",".join(str(x) for x in range(i * tp_size, (i + 1) * tp_size))
        process = multiprocessing.Process(target=run_command,
                                        args=(visible_devices, dp_rank,
                                                vllm_engine_port))
        processes.append(process)
        process.start()

    for process in processes:
        process.join()

    Parameter descriptions:

    Parameter Type Required Default Description
    --dp-size int Yes - Data parallel size (total number of DP ranks across all nodes).
    --tp-size int No 1 Tensor parallel size within each DP rank.
    --dp-size-local int No (same as --dp-size) Number of DP ranks on the current node. If not set, defaults to --dp-size.
    --dp-rank-start int No 0 Starting rank offset for data parallel ranks on this node.
    --dp-address str Yes - IP address of the data parallel master node.
    --dp-rpc-port str No 12345 RPC port for data parallel master communication.
    --vllm-start-port int No 9000 Starting port for each vLLM engine instance on this node.

  2. Prepare the script run_dp_template.sh on each node.

    1. Prefill node 0

      nic_name="xxxx" # change to your own nic name
local_ip=xx.xx.xx.1 # change to your own ip

export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export VLLM_RPC_TIMEOUT=3600000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=204
export HCCL_CONNECT_TIMEOUT=120
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=1024
export TASK_QUEUE_ENABLE=1
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
export ASCEND_RT_VISIBLE_DEVICES=$1
export VLLM_ASCEND_ENABLE_FUSED_MC2=1
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1

vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Pro-w4a8-mtp \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name auto \
    --max-model-len 131072 \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 16 \
    --no-disable-hybrid-kv-cache-manager \
    --tokenizer-mode deepseek_v4 \
    --tool-call-parser deepseek_v4 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v4 \
    --safetensors-load-strategy 'prefetch' \
    --model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 128}' \
    --trust-remote-code \
    --gpu-memory-utilization 0.92 \
    --quantization ascend \
    --block-size 128 \
    --enforce-eager \
    --speculative-config '{"num_speculative_tokens": 1,"method": "mtp","enforce_eager": true}' \
    --additional-config '{"enable_cpu_binding": true, "enable_dsa_cp": true}' \
    --kv-transfer-config \
    '{"kv_connector": "MooncakeHybridConnector",
    "kv_role": "kv_producer",
    "kv_port": "30200",
    "engine_id": "1",
    "kv_connector_extra_config": {
                "prefill": {
                        "dp_size": 2,
                        "tp_size": 16
                },
                "decode": {
                        "dp_size": 16,
                        "tp_size": 2
                }
        }
    }'

    2. Prefill node 1

      nic_name="xxxx" # change to your own nic name
local_ip=xx.xx.xx.2 # change to your own ip

export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export VLLM_RPC_TIMEOUT=3600000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=204
export HCCL_CONNECT_TIMEOUT=120
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=1024
export TASK_QUEUE_ENABLE=1
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
export ASCEND_RT_VISIBLE_DEVICES=$1
export VLLM_ASCEND_ENABLE_FUSED_MC2=1
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1

vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Pro-w4a8-mtp \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name auto \
    --max-model-len 131072 \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 16 \
    --no-disable-hybrid-kv-cache-manager \
    --tokenizer-mode deepseek_v4 \
    --tool-call-parser deepseek_v4 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v4 \
    --safetensors-load-strategy 'prefetch' \
    --model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 128}' \
    --trust-remote-code \
    --gpu-memory-utilization 0.92 \
    --quantization ascend \
    --block-size 128 \
    --enforce-eager \
    --speculative-config '{"num_speculative_tokens": 1,"method": "mtp","enforce_eager": true}' \
    --additional-config '{"enable_cpu_binding": true, "enable_dsa_cp": true}' \
    --kv-transfer-config \
    '{"kv_connector": "MooncakeHybridConnector",
    "kv_role": "kv_producer",
    "kv_port": "30200",
    "engine_id": "1",
    "kv_connector_extra_config": {
                "prefill": {
                        "dp_size": 2,
                        "tp_size": 16
                },
                "decode": {
                        "dp_size": 16,
                        "tp_size": 2
                }
        }
    }'

    3. Decode node (Same as another D node)

      nic_name="xxxx" # change to your own nic name
local_ip=xx.xx.xx.3/4 # change to your own ip

export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
export HCCL_OP_EXPANSION_MODE="AIV"
export TASK_QUEUE_ENABLE=1
export VLLM_RPC_TIMEOUT=3600000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=2000
export HCCL_CONNECT_TIMEOUT=1200
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=1024
export ASCEND_RT_VISIBLE_DEVICES=$1

vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Pro-w4a8-mtp \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name auto \
    --max-model-len 131072 \
    --max-num-batched-tokens 120 \
    --max-num-seqs 60 \
    --async-scheduling \
    --block-size 128 \
    --no-enable-prefix-caching \
    --tokenizer-mode deepseek_v4 \
    --tool-call-parser deepseek_v4 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v4 \
    --no-disable-hybrid-kv-cache-manager \
    --safetensors-load-strategy 'prefetch' \
    --model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 128}' \
    --trust-remote-code \
    --gpu-memory-utilization 0.9 \
    --quantization ascend \
    --speculative-config '{"num_speculative_tokens": 1, "method":"mtp", "enforce_eager": true}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --kv-transfer-config \
    '{"kv_connector": "MooncakeHybridConnector",
    "kv_role": "kv_consumer",
    "kv_port": "30800",
    "engine_id": "8",
    "kv_connector_extra_config": {
                "prefill": {
                        "dp_size": 2,
                        "tp_size": 16
                },
                "decode": {
                        "dp_size": 16,
                        "tp_size": 2
                }
        }
    }' \
    --additional-config '{
        "ascend_compilation_config":{
            "enable_npugraph_ex":true,
            "enable_static_kernel":false
        },
    "enable_cpu_binding":true,
    "recompute_scheduler_enable":true
    }'

  3. Start the server with the following command on each node.

    1. Prefill node 0

      # change ip to your own
python launch_online_dp.py --dp-size 2 --tp-size 16 --dp-size-local 1 --dp-rank-start 0 --dp-address xx.xx.xx.1 --dp-rpc-port 12321 --vllm-start-port 7100

    2. Prefill node 1

      # change ip to your own
python launch_online_dp.py --dp-size 2 --tp-size 16 --dp-size-local 1 --dp-rank-start 1 --dp-address xx.xx.xx.1 --dp-rpc-port 12321 --vllm-start-port 7100

    3. Decode node 0

      # change ip to your own
python launch_online_dp.py --dp-size 16 --tp-size 2 --dp-size-local 8 --dp-rank-start 0 --dp-address xx.xx.xx.3 --dp-rpc-port 12321 --vllm-start-port 7100

    4. Decode node 1

      # change ip to your own
python launch_online_dp.py --dp-size 16 --tp-size 2 --dp-size-local 8 --dp-rank-start 8 --dp-address xx.xx.xx.3 --dp-rpc-port 12321 --vllm-start-port 7100

  4. Deploy the P-D disaggregation proxy.

    Refer to Prefill-Decode Disaggregation (Deepseek) to deploy the P-D disaggregation proxy.

5.2.2 A2 Series PD Separation Deployment

This section shows the deployment guide of DeepSeek-V4-Pro on Atlas 800 A2 (64G × 8) multi-node environment with 1P1D for better performance.

Before you start, please:

  1. Prepare the script launch_online_dp.py on each node.

    import argparse
import multiprocessing
import os
import subprocess
import sys

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--dp-size",
        type=int,
        required=True,
        help="Data parallel size."
    )
    parser.add_argument(
        "--tp-size",
        type=int,
        default=1,
        help="Tensor parallel size."
    )
    parser.add_argument(
        "--dp-size-local",
        type=int,
        default=-1,
        help="Local data parallel size."
    )
    parser.add_argument(
        "--dp-rank-start",
        type=int,
        default=0,
        help="Starting rank for data parallel."
    )
    parser.add_argument(
        "--dp-address",
        type=str,
        required=True,
        help="IP address for data parallel master node."
    )
    parser.add_argument(
        "--dp-rpc-port",
        type=str,
        default=12345,
        help="Port for data parallel master node."
    )
    parser.add_argument(
        "--vllm-start-port",
        type=int,
        default=9000,
        help="Starting port for the engine."
    )
    return parser.parse_args()

args = parse_args()
dp_size = args.dp_size
tp_size = args.tp_size
dp_size_local = args.dp_size_local
if dp_size_local == -1:
    dp_size_local = dp_size
dp_rank_start = args.dp_rank_start
dp_address = args.dp_address
dp_rpc_port = args.dp_rpc_port
vllm_start_port = args.vllm_start_port

def run_command(visible_devices, dp_rank, vllm_engine_port):
    command = [
        "bash",
        "./run_dp_template.sh",
        visible_devices,
        str(vllm_engine_port),
        str(dp_size),
        str(dp_rank),
        dp_address,
        dp_rpc_port,
        str(tp_size),
    ]
    subprocess.run(command, check=True)

if __name__ == "__main__":
    template_path = "./run_dp_template.sh"
    if not os.path.exists(template_path):
        print(f"Template file {template_path} does not exist.")
        sys.exit(1)

    processes = []
    num_cards = dp_size_local * tp_size
    for i in range(dp_size_local):
        dp_rank = dp_rank_start + i
        vllm_engine_port = vllm_start_port + i
        visible_devices = ",".join(str(x) for x in range(i * tp_size, (i + 1) * tp_size))
        process = multiprocessing.Process(target=run_command,
                                        args=(visible_devices, dp_rank,
                                                vllm_engine_port))
        processes.append(process)
        process.start()

    for process in processes:
        process.join()

  2. Prepare the script run_dp_template.sh on each node.

    1. Prefill node (4 P nodes share the same script)

      nic_name="xxxx" # change to your own nic name
local_ip=xx.xx.xx.1/2/3/4 # change to your own ip

export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
export HCCL_OP_EXPANSION_MODE="AIV"
export TASK_QUEUE_ENABLE=1

export VLLM_RPC_TIMEOUT=3600000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=204
export HCCL_CONNECT_TIMEOUT=1200

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=1024

sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl kernel.sched_migration_cost_ns=50000

export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
export ASCEND_RT_VISIBLE_DEVICES=$1

vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Pro-w4a8-mtp \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name dsv4 \
    --max-model-len 133072 \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 16 \
    --no-disable-hybrid-kv-cache-manager \
    --trust-remote-code \
    --gpu-memory-utilization 0.9 \
    --quantization ascend \
    --safetensors-load-strategy 'prefetch' \
    --model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 128}' \
    --tokenizer-mode deepseek_v4 \
    --tool-call-parser deepseek_v4 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v4 \
    --enforce-eager \
    --no-enable-prefix-caching \
    --speculative-config '{"num_speculative_tokens": 1, "method":"mtp", "enforce_eager": true}' \
    --additional-config '{"enable_cpu_binding": true, "enable_shared_expert_dp": true, "enable_dsa_cp": true}' \
    --kv-transfer-config \
    '{"kv_connector": "MooncakeHybridConnector",
    "kv_role": "kv_producer",
    "kv_port": "30000",
    "engine_id": "0",
    "kv_connector_extra_config": {
            "prefill": {
                "dp_size": 4,
                "tp_size": 8
            },
            "decode": {
                "dp_size": 8,
                "tp_size": 4
            }
      }
  }'

    2. Decode node (4 D nodes share the same script)

      nic_name="xxxx" # change to your own nic name
local_ip=xx.xx.xx.5/6/7/8 # change to your own ip

export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
export HCCL_OP_EXPANSION_MODE="AIV"
export TASK_QUEUE_ENABLE=1

export VLLM_RPC_TIMEOUT=3600000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=204
export HCCL_CONNECT_TIMEOUT=1200

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=1024

sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl kernel.sched_migration_cost_ns=50000

export ASCEND_RT_VISIBLE_DEVICES=$1

vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Pro-w4a8-mtp \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name dsv4 \
    --max-model-len 133072 \
    --max-num-batched-tokens 120 \
    --max-num-seqs 60 \
    --async-scheduling \
    --block-size 128 \
    --no-disable-hybrid-kv-cache-manager \
    --trust-remote-code \
    --gpu-memory-utilization 0.9 \
    --quantization ascend \
    --tokenizer-mode deepseek_v4 \
    --tool-call-parser deepseek_v4 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v4 \
    --safetensors-load-strategy 'prefetch' \
    --model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 128}' \
    --no-enable-prefix-caching \
    --speculative-config '{"num_speculative_tokens": 1, "method":"mtp", "enforce_eager": true}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --kv-transfer-config \
    '{"kv_connector": "MooncakeHybridConnector",
    "kv_role": "kv_consumer",
    "kv_port": "30100",
    "engine_id": "1",
    "kv_connector_extra_config": {
            "prefill": {
                "dp_size": 4,
                "tp_size": 8
            },
            "decode": {
                "dp_size": 8,
                "tp_size": 4
            }
      }
  }' \
    --additional-config '{"ascend_compilation_config":{"enable_npugraph_ex":true,"enable_static_kernel":false}, "enable_cpu_binding":true, "recompute_scheduler_enable":true}'

  3. Start the server with the following command on each node.

    1. Prefill node 0

      # change ip to your own
python launch_online_dp.py --dp-size 4 --tp-size 8 --dp-size-local 1 --dp-rank-start 0 --dp-address xx.xx.xx.1 --dp-rpc-port 12321 --vllm-start-port 7100

    2. Prefill node 1

      # change ip to your own
python launch_online_dp.py --dp-size 4 --tp-size 8 --dp-size-local 1 --dp-rank-start 1 --dp-address xx.xx.xx.1 --dp-rpc-port 12321 --vllm-start-port 7100

    3. Prefill node 2

      # change ip to your own
python launch_online_dp.py --dp-size 4 --tp-size 8 --dp-size-local 1 --dp-rank-start 2 --dp-address xx.xx.xx.1 --dp-rpc-port 12321 --vllm-start-port 7100

    4. Prefill node 3

      # change ip to your own
python launch_online_dp.py --dp-size 4 --tp-size 8 --dp-size-local 1 --dp-rank-start 3 --dp-address xx.xx.xx.1 --dp-rpc-port 12321 --vllm-start-port 7100

    5. Decode node 0

      # change ip to your own
python launch_online_dp.py --dp-size 8 --tp-size 4 --dp-size-local 2 --dp-rank-start 0 --dp-address xx.xx.xx.2 --dp-rpc-port 12321 --vllm-start-port 7100

    6. Decode node 1

      # change ip to your own
python launch_online_dp.py --dp-size 8 --tp-size 4 --dp-size-local 2 --dp-rank-start 2 --dp-address xx.xx.xx.2 --dp-rpc-port 12321 --vllm-start-port 7100

    7. Decode node 2

      # change ip to your own
python launch_online_dp.py --dp-size 8 --tp-size 4 --dp-size-local 2 --dp-rank-start 4 --dp-address xx.xx.xx.2 --dp-rpc-port 12321 --vllm-start-port 7100

    8. Decode node 3

      # change ip to your own
python launch_online_dp.py --dp-size 8 --tp-size 4 --dp-size-local 2 --dp-rank-start 6 --dp-address xx.xx.xx.2 --dp-rpc-port 12321 --vllm-start-port 7100

  4. Deploy the P-D disaggregation proxy.

    Refer to Prefill-Decode Disaggregation (Deepseek) to deploy the P-D disaggregation proxy.

Key Parameter Descriptions:

  • VLLM_ASCEND_ENABLE_FLASHCOMM1=1: enables the communication optimization function on the prefill nodes.
  • VLLM_ASCEND_ENABLE_FUSED_MC2=1: enables the Fused MC2 fusion operator to accelerate communication on prefill nodes (A3 series).
  • recompute_scheduler_enable: true: enables the recomputation scheduler. When the KV Cache of the decode node is insufficient, requests will be sent to the prefill node to recompute the KV Cache. In the PD separation scenario, enable this configuration only on decode nodes.
  • MooncakeHybridConnector: the KV transfer connector used for PD separation, transferring KV Cache between prefill and decode nodes.
  • enable_shared_expert_dp: true：启用共享专家的数据并行，适用于MoE模型。

部署验证：

PD分离服务完全启动后，通过预填充主节点上的代理端口发送请求，验证预填充和解码节点是否协同正常工作。有关代理验证方法，请参考预填充-解码分离(Deepseek)。

Common Issues Tip: If you encounter issues with PD separation deployment, please refer to the Public FAQ for troubleshooting.

6 功能验证

服务器启动后，您可以使用输入提示查询模型：

在:中，使用主节点的IP地址和端口号。如果主备节点分离，请使用代理节点的IP地址和端口号。

curl http://<node0_ip>:<port>/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "dsv4",
        "messages": [
            {
                "role": "user",
                "content": "Who are you?"
            }
        ],
        "max_tokens": 256,
        "temperature": 0
    }'

预期结果：

服务返回HTTP 200 OK，JSON响应中包含choices字段。

7 精度评估

以下是两种精度评估方法。

使用AISBench

  1. 详情请参考使用AISBench。

  2. 执行后即可获取结果。

数据集 版本 指标 模式 vllm-api-general-chat 备注
GPQA - 准确率 gen 89.90 1 Atlas 800 A3 (128G × 8)
GSM8K - 准确率 gen 96.21 1 Atlas 800 A3 (128G × 8)

8 性能评估

使用AISBench

详情请参考使用AISBench进行性能评估。

使用vLLM Benchmark

更多详情请参考vllm benchmark。

9性能调优

9.1推荐配置

注意：以下配置在特定测试环境中验证，仅供参考。最佳配置取决于最大输入/输出长度、前缀缓存命中率、精度要求和部署机器比例等因素。建议参考第9.2节根据实际情况进行调优。

表1：场景概览

*Total NPUs表示所有节点使用的NPU总数。

场景 部署模式 *NPU总数 权重版本 关键考虑因素
高吞吐 单节点混合 32 (A3) DeepSeek-V4-Pro-w4a8-mtp 使用dp2 tp16平衡内存容量和计算效率
高吞吐 1P1D部署 64 (A3) DeepSeek-V4-Pro-w4a8-mtp dp16 tp2或dp2 tp16，取决于内存和并发度
长上下文(1M) 单节点混合 32 (A3) DeepSeek-V4-Pro-w4a8-mtp 使用dp2 tp16平衡内存容量和计算效率
长上下文(1M) 1P1D部署 64 (A3) DeepSeek-V4-Pro-w4a8-mtp P和D节点均使用dp2 tp16；平衡延迟和吞吐

表2：详细节点配置

场景 配置 NPU数 TP DP 最大序列数 最大批处理Token数 最大模型长度 MTP推测数
多节点(A3) 否de0 / 否de1 8 16 2 32 4096 135000 1
PD分离(A3) 预填充节点 8 16 2 16 4096 131072 1
PD分离(A3) 解码节点 8 2 16 60 120 131072 1

完整的启动命令和参数描述请参考第5章中的部署示例。

注意：

max-model-len和max-num-seqs需要根据实际使用场景设置。其他设置请参考部署章节。

目前，我们以实验性方式支持4K前缀缓存命中。您只需将服务中的--block-size值从128改为32即可。

9.2调优指南

9.2.1通用调优参考

调优方法请参考公共性能调优文档。

详细功能描述请参考功能指南。

10常见问题

For common environment, installation, and general parameter issues, please refer to the Public FAQ; this chapter only covers 模式l-specific issues.