基于 Mooncake 多实例的 PD 共置部署

快速开始

vLLM-Ascend now supports PD-colocated deployment with Mooncake features. This guide provides step-by-step instructions to test these features with constrained resources.

Using the Qwen2.5-72B-Instruct model as an example, this guide demonstrates how to use vllm-ascend v0.22.1rc1 (with vLLM v0.22.1) on two Atlas 800T A2 nodes to deploy two vLLM instances. Each instance occupies 4 NPU cards and uses PD-colocated deployment.

验证多节点通信环境

物理层要求

• The two Atlas 800T A2 nodes must be physically interconnected via a RoCE network. Without RoCE interconnection, cross-node KV Cache access performance will be significantly degraded.
• All NPU cards must communicate properly. Intra-node communication uses HCCS, while inter-node communication uses the RoCE network.

验证流程

The following process serves as a reference example. Please modify parameters such as IP addresses according to your actual environment.

1. 单节点验证：

Execute the following commands sequentially. The results must all be success and the status must be UP:

# Check the remote switch ports
for i in {0..7}; do hccn_tool -i $i -lldp -g | grep Ifname; done
# Get the link status of the Ethernet ports (UP or DOWN)
for i in {0..7}; do hccn_tool -i $i -link -g ; done
# Check the network health status
for i in {0..7}; do hccn_tool -i $i -net_health -g ; done
# View the network detected IP configuration
for i in {0..7}; do hccn_tool -i $i -netdetect -g ; done
# View gateway configuration
for i in {0..7}; do hccn_tool -i $i -gateway -g ; done

1. Check NPU HCCN Configuration:

Ensure that the hccn.conf file exists in the environment. If using Docker, mount it into the container.

cat /etc/hccn.conf

1. Get NPU IP Addresses:

for i in {0..7}; do hccn_tool -i $i -ip -g; done

1. Cross-Node PING Test:

# Execute the following command on each node, replacing x.x.x.x
# with the target node's NPU card address.
for i in {0..7}; do hccn_tool -i $i -ping -g address x.x.x.x; done

1. Check NPU TLS Configuration

# The tls settings should be consistent across all nodes.
for i in {0..7}; do hccn_tool -i $i -tls -g ; done | grep switch

使用 Docker 运行

在每个节点上启动 Docker 容器。

# Update the vllm-ascend image
export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1
export NAME=vllm-ascend

# Run the container using the defined variables
# This test uses four NPU cards to create the container.
# Mount the hccn.conf file from the host node into the container.
docker run --rm \
--name $NAME \
--net=host \
--shm-size=1g \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
--device /dev/davinci3 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/Ascend/driver/tools/hccn_tool:\
/usr/local/Ascend/driver/tools/hccn_tool \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /etc/hccn.conf:/etc/hccn.conf \
-v /root/.cache:/root/.cache \
-it $IMAGE bash

（可选）安装 Mooncake

Mooncake is pre-installed and functional in the v0.22.1rc1 image. The following installation steps are optional.

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI. Installation and compilation guide: https://github.com/kvcache-ai/Mooncake?tab=readme-ov-file#build-and-use-binaries.

首先，使用以下命令获取 Mooncake 项目：

git clone -b v0.3.9 --depth 1 https://github.com/kvcache-ai/Mooncake.git
cd Mooncake
git submodule update --init --recursive

安装 MPI：

apt-get install mpich libmpich-dev -y

安装相关依赖（无需安装 Go）：

bash dependencies.sh -y

编译并安装：

mkdir build
cd build
cmake .. -DUSE_ASCEND_DIRECT=ON
make -j
make install

安装完成后，验证 Mooncake 是否正确安装：

python -c "import mooncake; print(mooncake.__file__)"
# Expected output path:
# /usr/local/Ascend/ascend-toolkit/latest/python/
# site-packages/mooncake/__init__.py

启动 Mooncake 主服务

To start the Mooncake master service in one of the node containers, use the following command:

docker exec -it vllm-ascend bash
cd /vllm-workspace/Mooncake
mooncake_master --port 50088 \
  --eviction_high_watermark_ratio 0.95 \
  --eviction_ratio 0.05

参数	值	说明
port	50088	主服务的端口
eviction_high_watermark_ratio	0.95	高水位线比例（95% 阈值）
eviction_ratio	0.05	存满时的淘汰比例（5%）

创建名为 mooncake.json 的 Mooncake 配置文件

mooncake.json 文件的模板如下：

{
    "metadata_server": "P2PHANDSHAKE",
    "protocol": "ascend",
    "device_name": "",
    "master_server_address": "<your_server_ip>:50088",
    "global_segment_size": 107374182400
}

参数	值	说明
metadata_server	P2PHANDSHAKE	点对点握手模式
protocol	ascend	Ascend 专有协议
master_server_address	90.90.100.188:50088（示例）	主服务器地址
global_segment_size	107374182400	每段大小（100 GB）

vLLM 实例部署

Create containers on both Node 1 and Node 2, and launch the Qwen2.5-72B-Instruct model service in each to test the reusability and performance of cross-node, cross-instance KV Cache. Instance 1 utilizes NPU cards [0-3] on the first Atlas 800T A2 server, while Instance 2 utilizes cards [0-3] on the second server.

部署实例 1

Replace file paths, host, and port parameters based on your actual environment configuration.

export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/\
latest/python/site-packages:$LD_LIBRARY_PATH
export MOONCAKE_CONFIG_PATH="/vllm-workspace/mooncake.json"
# NPU buffer pool: quantity:size(MB)
# Allocates 4 buffers of 8MB each for KV transfer
export ASCEND_BUFFER_POOL=4:8

vllm serve <path_to_your_model>/Qwen2.5-72B-Instruct/ \
--served-model-name qwen \
--dtype bfloat16 \
--max-model-len 25600 \
--tensor-parallel-size 4 \
--host <your_server_ip> \
--port 8002 \
--max-num-batched-tokens 4096 \
--gpu-memory-utilization 0.9 \
--kv-transfer-config '{
      "kv_connector": "MooncakeConnectorStoreV1",
      "kv_role": "kv_both",
      "kv_connector_extra_config": {
          "use_layerwise": false,
          "mooncake_rpc_port": "0",
          "load_async": true,
          "register_buffer": true
      }
  }'

部署实例 2

The deployment method for Instance 2 is identical to Instance 1. Simply modify the --host and --port parameters according to your Instance 2 configuration.

配置参数

参数	值	说明
kv_connector	MooncakeConnectorStoreV1	使用 StoreV1 版本
kv_role	kv_both	同时启用生产和消费
use_layerwise	false	传输整个缓存（参见说明）
mooncake_rpc_port	0	自动分配端口
load_async	true	启用异步加载
register_buffer	true	PD 共置模式下必需

关于 use_layerwise 的说明：

• false: Transfer entire KV Cache (suitable for cross-node with sufficient bandwidth)
• true: Layer-by-layer transfer (suitable for single-node memory constraints)

基准测试

We recommend using the AISBench tool to assess performance. The test uses Dataset A, consisting of fully random data, with the following configuration:

• 输入/输出 token：1024/10
• 总请求数：100
• 并发数：25

测试流程包含三个步骤：

步骤 1：基线（无缓存）

Send Dataset A to Instance 1 on Node 1 and record the Time to First Token (TTFT) as TTFT1.

步骤 2 的准备

Before Step 2, send a fully random Dataset B to Instance 1. Due to the unified on-chip memory/DRAM KV Cache with LRU (Least Recently Used) eviction policy, Dataset B's cache evicts Dataset A's cache from on-chip memory, leaving Dataset A's cache only in Node 1's DRAM.

步骤 2：本地 DRAM 命中

Send Dataset A to Instance 1 again to measure the performance when hitting the KV Cache in local DRAM. Record the TTFT as TTFT2.

步骤 3：跨节点 DRAM 命中

Send Dataset A to Instance 2. With the Mooncake KV Cache pool, this results in a cross-node KV Cache hit from Node 1's DRAM. Record the TTFT as TTFT3.

模型配置：

from ais_bench.benchmark.models import VLLMCustomAPIChatStream
from ais_bench.benchmark.utils.model_postprocessors import extract_non_reasoning_content

models = [
    dict(
        attr="service",
        type=VLLMCustomAPIChatStream,
        abbr='vllm-api-stream-chat',
        path="<path_to_your_model>/Qwen2.5-72B-Instruct",
        model="qwen",
        request_rate = 0,
        retry = 2,
        host_ip = "<your_server_ip>",
        host_port = 8002,
        max_out_len = 10,
        batch_size= 25,
        trust_remote_code=False,
        generation_kwargs = dict(
            temperature = 0,
            ignore_eos = True,
        ),
    )
]

性能基准测试命令：

ais_bench --models vllm_api_stream_chat \
  --datasets gsm8k_gen_0_shot_cot_str_perf \
  --debug --summarizer default_perf --mode perf

测试结果

请求数	并发数	TTFT1（毫秒）	TTFT2（毫秒）	TTFT3（毫秒）
100	25	2322	739	948