大规模专家并行下的分布式 DP 服务端

快速开始

vLLM-Ascend now supports prefill-decode (PD) disaggregation in the large-scale Expert Parallelism (EP) scenario. To achieve better performance, the distributed DP server is applied in vLLM-Ascend. In the PD separation scenario, different optimization strategies can be implemented based on the distinct characteristics of PD 节点类型s, thereby enabling more flexible model deployment. \ Taking the DeepSeek model as an example, using 8 Atlas 800T A3 servers to deploy the model. Assume the IP of the servers starts from 192.0.0.1 and ends by 192.0.0.8. Use the first 4 servers as prefiller 节点类型s and the last 4 servers as decoder 节点类型s. And the prefiller 节点类型s are deployed as master 节点类型s independently, while the decoder 节点类型s use the 192.0.0.5 节点类型 as the master 节点类型.

验证多节点通信环境

物理层要求

• 物理机必须位于同一局域网内，并具备网络连通性。
• 所有 NPU 必须互联。对于 Atlas A2 系列，节点内通过 HCCS 连接，节点间通过 RDMA 连接。对于 Atlas A3 系列，节点内和节点间均通过 HCCS 连接。

验证流程

A3A2

1. 单节点验证：

   依次在每个节点上执行以下命令。结果必须全部为 success 且状态为 UP：

   # Check the remote switch ports
   for i in {0..15}; do hccn_tool -i $i -lldp -g | grep Ifname; done
   # Get the link status of the Ethernet ports (UP or DOWN)
   for i in {0..15}; do hccn_tool -i $i -link -g ; done
   # Check the network health status
   for i in {0..15}; do hccn_tool -i $i -net_health -g ; done
   # View the network detected IP configuration
   for i in {0..15}; do hccn_tool -i $i -netdetect -g ; done
   # View gateway configuration
   for i in {0..15}; do hccn_tool -i $i -gateway -g ; done
   # View NPU network configuration
   cat /etc/hccn.conf
   

2. Get NPU IP Addresses

   for i in {0..15}; do hccn_tool -i $i -vnic -g;done
   

3. Get superpodid and SDID

   for i in {0..15}; do npu-smi info -t spod-info -i $i -c 0;npu-smi info -t spod-info -i $i -c 1;done
   

4. Cross-Node PING Test

   # Execute on the target node (replace 'x.x.x.x' with actual NPU IP address)
   for i in {0..15}; do hccn_tool -i $i -hccs_ping -g address x.x.x.x;done
   

1. Single Node Verification:

   Execute the following commands on each node in sequence. The results must all be success and the status must be UP:

   # Check the remote switch ports
   for i in {0..7}; do hccn_tool -i $i -lldp -g | grep Ifname; done
   # Get the link status of the Ethernet ports (UP or DOWN)
   for i in {0..7}; do hccn_tool -i $i -link -g ; done
   # Check the network health status
   for i in {0..7}; do hccn_tool -i $i -net_health -g ; done
   # View the network detected IP configuration
   for i in {0..7}; do hccn_tool -i $i -netdetect -g ; done
   # View gateway configuration
   for i in {0..7}; do hccn_tool -i $i -gateway -g ; done
   # View NPU network configuration
   cat /etc/hccn.conf
   

2. Get NPU IP Addresses

   for i in {0..7}; do hccn_tool -i $i -ip -g;done
   

3. Cross-Node PING Test

   # Execute on the target node (replace 'x.x.x.x' with actual NPU IP address)
   for i in {0..7}; do hccn_tool -i $i -ping -g address x.x.x.x;done
   

Large-Scale EP model deployment

Generate script with configurations

In the PD separation scenario, we provide an optimized configuration. You can use the following shell script for configuring the prefiller and decoder nodes respectively.

Prefiller nodeDecoder node

# run_dp_template.sh
#!/bin/sh

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip
nic_name="xxxx"
local_ip="xxxx"

# basic configuration for HCCL and connection
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export HCCL_BUFFSIZE=256

# obtain parameters from distributed DP server
export VLLM_DP_SIZE=$1
export VLLM_DP_MASTER_IP=$2
export VLLM_DP_MASTER_PORT=$3
export VLLM_DP_RANK_LOCAL=$4
export VLLM_DP_RANK=$5
export VLLM_DP_SIZE_LOCAL=$7

#pytorch_npu settings and vllm settings
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1
export VLLM_USE_MODELSCOPE="True"

# enable the distributed DP server
export VLLM_WORKER_MULTIPROC_METHOD="fork"
export VLLM_ASCEND_EXTERNAL_DP_LB_ENABLED=1

# The w8a8 weight can be obtained from https://www.modelscope.cn/models/vllm-ascend/DeepSeek-R1-W8A8
# "--additional-config" is used to enable characteristics from vllm-ascend
vllm serve vllm-ascend/DeepSeek-R1-W8A8 \
    --host 0.0.0.0 \
    --port $6 \
    --tensor-parallel-size 8 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name deepseek_r1 \
    --max-model-len 17000 \
    --max-num-batched-tokens 16384 \
    --trust-remote-code \
    --max-num-seqs 4 \
    --gpu-memory-utilization 0.9 \
    --quantization ascend \
    --speculative-config '{"num_speculative_tokens": 1, "method":"deepseek_mtp"}' \
    --enforce-eager \
    --kv-transfer-config \
    '{"kv_connector": "MooncakeConnectorV1",
      "kv_buffer_device": "npu",
      "kv_role": "kv_producer",
      "kv_parallel_size": "1",
      "kv_port": "20001",
    }' \
    --additional-config '{"enable_weight_nz_layout":true,"enable_prefill_optimizations":true}'

# run_dp_template.sh
#!/bin/sh

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip
nic_name="xxxx"
local_ip="xxxx"

# basic configuration for HCCL and connection
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export HCCL_BUFFSIZE=1024

# obtain parameters from distributed DP server
export VLLM_DP_SIZE=$1
export VLLM_DP_MASTER_IP=$2
export VLLM_DP_MASTER_PORT=$3
export VLLM_DP_RANK_LOCAL=$4
export VLLM_DP_RANK=$5
export VLLM_DP_SIZE_LOCAL=$7

#pytorch_npu settings and vllm settings
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1
export VLLM_USE_MODELSCOPE="True"

# enable the distributed DP server
export VLLM_WORKER_MULTIPROC_METHOD="fork"
export VLLM_ASCEND_EXTERNAL_DP_LB_ENABLED=1

# The w8a8 weight can be obtained from https://www.modelscope.cn/models/vllm-ascend/DeepSeek-R1-W8A8
# "--additional-config" is used to enable characteristics from vllm-ascend
vllm serve vllm-ascend/DeepSeek-R1-W8A8 \
    --host 0.0.0.0 \
    --port $6 \
    --tensor-parallel-size 1 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name deepseek_r1 \
    --max-model-len 17000 \
    --max-num-batched-tokens 256 \
    --trust-remote-code \
    --max-num-seqs 28 \
    --gpu-memory-utilization 0.9 \
    --quantization ascend \
    --speculative-config '{"num_speculative_tokens": 1, "method":"deepseek_mtp"}' \
    --kv-transfer-config \
        '{"kv_connector": "MooncakeConnectorV1",
        "kv_buffer_device": "npu",
        "kv_role": "kv_consumer",
        "kv_parallel_size": "1",
        "kv_port": "20001",
        }' \
    --additional-config '{"enable_weight_nz_layout":true}'

Start Distributed DP Server for prefill-decode disaggregation

Execute the following Python file on all nodes to use the distributed DP server. (We recommend using this feature on the v0.9.1 official release)

Prefiller nodeDecoder node

import multiprocessing
import os
import sys
dp_size = 2 # total number of DP engines for decode/prefill
dp_size_local = 2 # number of DP engines on the current node
dp_rank_start = 0 # starting DP rank for the current node
# dp_ip is different on prefiller nodes in this example
dp_ip = "192.0.0.1" # master node IP for DP communication
dp_port = 13395 # port used for DP communication
engine_port = 9000 # starting port for all DP groups on the current node
template_path = "./run_dp_template.sh"
if not os.path.exists(template_path):
  print(f"Template file {template_path} does not exist.")
  sys.exit(1)
def run_command(dp_rank_local, dp_rank, engine_port_):
  command = f"bash ./run_dp_template.sh {dp_size} {dp_ip} {dp_port} {dp_rank_local} {dp_rank} {engine_port_} {dp_size_local}"
  os.system(command)
processes = []
for i in range(dp_size_local):
  dp_rank = dp_rank_start + i
  dp_rank_local = i
  engine_port_ = engine_port + i
  process = multiprocessing.Process(target=run_command, args=(dp_rank_local, dp_rank, engine_port_))
  processes.append(process)
  process.start()
for process in processes:
  process.join()

import multiprocessing
import os
import sys
dp_size = 64 # total number of DP engines for decode/prefill
dp_size_local = 16 # number of DP engines on the current node
dp_rank_start = 0 # starting DP rank for the current node. e.g. 0/16/32/48
# dp_ip is the same on decoder nodes in this example
dp_ip = "192.0.0.5" # master node IP for DP communication.
dp_port = 13395 # port used for DP communication
engine_port = 9000 # starting port for all DP groups on the current node
template_path = "./run_dp_template.sh"
if not os.path.exists(template_path):
  print(f"Template file {template_path} does not exist.")
  sys.exit(1)
def run_command(dp_rank_local, dp_rank, engine_port_):
  command = f"bash ./run_dp_template.sh {dp_size} {dp_ip} {dp_port} {dp_rank_local} {dp_rank} {engine_port_} {dp_size_local}"
  os.system(command)
processes = []
for i in range(dp_size_local):
  dp_rank = dp_rank_start + i
  dp_rank_local = i
  engine_port_ = engine_port + i
  process = multiprocessing.Process(target=run_command, args=(dp_rank_local, dp_rank, engine_port_))
  processes.append(process)
  process.start()
for process in processes:
  process.join()

请注意，预填充节点和解码节点可能具有不同的配置。在此示例中，每个预填充节点独立部署为主节点，而解码节点则使用 192.0.0.5 节点作为主节点。这导致了 'dp_size_local' 和 'dp_rank_start' 的差异。

分布式 DP 服务端代理示例

在 PD 分离场景中，我们需要一个代理来分发请求。执行以下命令以启用示例代理：

python load_balance_proxy_server_example.py \
  --port 8000 \
  --host 0.0.0.0 \
  --prefiller-hosts \
    192.0.0.1 \
    192.0.0.2 \
    192.0.0.3 \
    192.0.0.4 \
  --prefiller-hosts-num \
    2 2 2 2 \
  --prefiller-ports \
    9000 9000 9000 9000 \
  --prefiller-ports-inc \
    2 2 2 2\
  --decoder-hosts \
    192.0.0.5 \
    192.0.0.6 \
    192.0.0.7 \
    192.0.0.8 \
  --decoder-hosts-num \
    16 16 16 16 \
  --decoder-ports  \
    9000 9000 9000 9000 \
  --decoder-ports-inc \
    16 16 16 16 \

参数	含义
--port	代理服务端口
--host	代理服务主机 IP
--prefiller-hosts	预填充节点主机列表
--prefiller-hosts-num	预填充节点主机的重复次数
--prefiller-ports	预填充节点端口列表
--prefiller-ports-inc	预填充节点端口的递增次数
--decoder-hosts	解码节点主机列表
--decoder-hosts-num	解码节点主机的重复次数
--decoder-ports	解码节点端口列表
--decoder-ports-inc	解码节点端口的递增次数

您可以在仓库的 examples 目录中找到代理程序： load_balance_proxy_server_example.py

基准测试

我们建议使用 aisbench 工具来评估性能。aisbench。执行以下命令安装 aisbench

git clone https://github.com/AISBench/benchmark.git
cd benchmark/
pip3 install -e ./

在评估性能之前，您需要取消 HTTP 代理，如下所示：

# unset proxy
unset http_proxy
unset https_proxy

• 您可以将数据集放置在目录：benchmark/ais_bench/datasets 中
• 您可以在目录 benchmark/ais_bench/benchmark/configs/models/vllm_api 中修改配置。以 vllm_api_stream_chat.py 为例：

models = [
    dict(
        attr="service",
        type=VLLMCustomAPIChatStream,
        abbr='vllm-api-stream-chat',
        path="vllm-ascend/DeepSeek-R1-W8A8",
        model="dsr1",
        request_rate = 28,
        retry = 2,
        host_ip = "192.0.0.1", # Proxy service host IP
        host_port = 8000,  # Proxy service Port
        max_out_len = 10,
        batch_size=1536,
        trust_remote_code=True,
        generation_kwargs = dict(
            temperature = 0,
            seed = 1024,
            ignore_eos=False,
        )
    )
]

• 以 gsm8k 数据集为例，执行以下命令评估性能。

ais_bench --models vllm_api_stream_chat --datasets gsm8k_gen_0_shot_cot_str_perf  --debug  --mode perf

• 有关 aisbench 命令和参数的更多详细信息，请参阅 aisbench

预填充与解码配置详情

在 PD 分离场景中，我们提供了一套优化配置。

• 预填充节点

• 设置 HCCL_BUFFSIZE=256

• 在 'vllm serve' 中添加 '--enforce-eager' 命令

• '--kv-transfer-config' 参数示例如下：

  --kv-transfer-config \
      '{"kv_connector": "MooncakeConnectorV1",
        "kv_buffer_device": "npu",
        "kv_role": "kv_producer",
        "kv_parallel_size": "1",
        "kv_port": "20001",
      }'
  

• Take '--additional-config' as follows:

  --additional-config '{"enable_weight_nz_layout":true,"enable_prefill_optimizations":true}'
  

• decoder node

• set HCCL_BUFFSIZE=1024

• Take '--kv-transfer-config' as follows:

  --kv-transfer-config
      '{"kv_connector": "MooncakeConnectorV1",
        "kv_buffer_device": "npu",
        "kv_role": "kv_consumer",
        "kv_parallel_size": "1",
        "kv_port": "20001",
      }'
  

• Take '--additional-config' as follows:

  --additional-config '{"enable_weight_nz_layout":true}'
  

Parameters Description

1. '--additional-config' Parameter Introduction:

2. "enable_weight_nz_layout": Whether to convert quantized weights to NZ format to accelerate matrix multiplication.

3. "enable_prefill_optimizations": Whether to enable DeepSeek models' prefill optimizations.
   

4. Enable MTP Add the following command to your configurations.

   --speculative-config '{"num_speculative_tokens": 1, "method":"deepseek_mtp"}'
   

推荐配置示例

例如，如果平均输入长度为 3.5k，输出长度为 1.1k，上下文长度为 16k，输入数据集的最大长度为 7k。在此场景下，我们为支持高 EP 的分布式 DP 服务器提供一套推荐配置。这里我们使用 4 个节点进行 Prefill，4 个节点进行 Decode。

节点类型	DP	TP	EP	max-model-len	max-num-batched-tokens	max-num-seqs	gpu-memory-utilization
prefill	2	8	16	17000	16384	4	0.9
decode	64	1	64	17000	256	28	0.9

Note

请注意，这些配置与优化逻辑无关。您需要根据实际场景调整这些参数。

常见问题解答 (FAQ)

1.Prefiller 节点需要预热

由于某些 NPU 算子的计算需要多轮预热才能达到最佳性能，我们建议在进行性能测试前先用一些请求预热服务，以获得最佳的端到端吞吐量。