GLM-5.3¶

1 简介¶

GLM-5.3 使用与 GLM-5.2 相同的基础模型——所有提升均来自后训练。与 GLM-5.2 相比，它在复杂编码和长周期任务上表现更佳。

本文档将展示该模型的主要验证步骤，包括支持的特性、特性配置、环境准备、多节点部署、精度和性能评估。

2 支持的特性¶

请参阅支持的特性列表以获取该模型的支持特性矩阵。

请参阅特性指南以获取特性的配置。

3 前提条件¶

3.1 模型权重¶

权重版本	硬件要求	下载链接
`GLM-5.3-w8a8c8`	2 Atlas 800I A3 (128GB × 8) node or 4 Atlas 800I A2 (64GB × 8)	ModelScope

你可以使用 msmodelslim 直接对模型进行量化。

建议将模型权重下载到多节点的共享目录中，例如 /root/.cache/。

路径说明：将模型权重下载到你选择的目录中并记录该路径。确保后续部署命令中的模型路径与该目录一致。

3.2 验证多节点通信（可选）¶

如果您想部署多节点环境，需要按照验证多节点通信环境中的说明验证多节点通信。

4 安装¶

4.1 Docker 镜像安装¶

您可以直接使用我们的官方 Docker 镜像来运行 GLM-5.3。

A3 系列A2 系列

在每个节点上启动 Docker 镜像。

export IMAGE=quay.io/ascend/vllm-ascend:v0.30.0rc1-a3
export NAME=vllm-ascend

# Run the container using the defined variables
# Note: If you are running bridge network with docker, please expose available ports for multiple nodes communication in advance
docker run --rm \
--name $NAME \
--net=host \
--shm-size=1g \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
--device /dev/davinci3 \
--device /dev/davinci4 \
--device /dev/davinci5 \
--device /dev/davinci6 \
--device /dev/davinci7 \
--device /dev/davinci8 \
--device /dev/davinci9 \
--device /dev/davinci10 \
--device /dev/davinci11 \
--device /dev/davinci12 \
--device /dev/davinci13 \
--device /dev/davinci14 \
--device /dev/davinci15 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-it $IMAGE bash

在您的每个节点上启动 Docker 镜像。

export IMAGE=quay.io/ascend/vllm-ascend:v0.30.0rc1
docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --net=host \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /root/.cache:/root/.cache \
    -it $IMAGE bash

如果您想部署多节点环境，需要在每个节点上配置环境。

4.2 源码安装¶

如果您不想使用上述 Docker 镜像，也可以从源码构建所有内容：

从源码安装 vllm-ascend，请参阅安装。

5 在线服务部署¶

5.1 上下文低于 1M¶

5.1.1 多节点部署¶

如果您想部署多节点环境，需要按照验证多节点通信环境中的说明验证多节点通信。 The scripts in Section 5.1.1 were tested on v0.23.0.

常见问题提示：如果遇到问题，请参阅公共 FAQ。

A3 系列A2 系列

GLM-5.3-w8a8c8：可部署在 2 个 Atlas 800I A3（64GB × 16）节点上。

分别在两个节点上运行以下脚本。

节点 0

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="xxx"
local_ip="xxx"

# The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)
node0_ip="xxxx"

export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export HCCL_TRANSFER_TIMEOUT=600
export HCCL_EXEC_TIMEOUT=3600
export HCCL_CONNECT_TIMEOUT=3600
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export HCCL_BUFFSIZE=400
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export VLLM_ASCEND_ENABLE_MLAPO=1

# Ensure the model path matches the directory recorded during download
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/GLM-5.3-w8a8c8 \
    --host 0.0.0.0 \
    --port 8000 \
    --safetensors-load-strategy prefetch \
    --api-server-count 1 \
    --data-parallel-size 8 \
    --data-parallel-start-rank 0 \
    --data-parallel-size-local 4 \
    --data-parallel-address $node0_ip \
    --data-parallel-rpc-port 12980 \
    --tensor-parallel-size 4 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name glm-5 \
    --tool-call-parser glm47 \
    --reasoning-parser glm47 \
    --enable-auto-tool-choice \
    --max-num-seqs 6 \
    --max-model-len 202752 \
    --max-num-batched-tokens 4096 \
    --trust-remote-code \
    --gpu-memory-utilization 0.90 \
    --quantization ascend \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --kv-cache-dtype int8 \
    --attention_config.indexer_kv_dtype int8 \
    --additional-config '{"enable_dsa_cp": true, "enable_balance_scheduling": true, "enable_fused_mc2": 1, "enable_flashcomm1": true}'  \
    --speculative-config '{"num_speculative_tokens": 3, "method": "deepseek_mtp", "enforce_eager": true}'

节点 1

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="xxx"
local_ip="xxx"

# The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)
node0_ip="xxxx"

export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export HCCL_TRANSFER_TIMEOUT=600
export HCCL_EXEC_TIMEOUT=3600
export HCCL_CONNECT_TIMEOUT=3600
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export HCCL_BUFFSIZE=400
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export VLLM_ASCEND_ENABLE_MLAPO=1

# Ensure the model path matches the directory recorded during download
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/GLM-5.3-w8a8c8 \
    --host 0.0.0.0 \
    --port 8000 \
    --headless \
    --data-parallel-size 8 \
    --data-parallel-start-rank 4 \
    --data-parallel-size-local 4 \
    --data-parallel-address $node0_ip \
    --data-parallel-rpc-port 12980 \
    --tensor-parallel-size 4 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name glm-5 \
    --tool-call-parser glm47 \
    --reasoning-parser glm47 \
    --enable-auto-tool-choice \
    --max-num-seqs 6 \
    --max-model-len 202752 \
    --max-num-batched-tokens 4096 \
    --trust-remote-code \
    --gpu-memory-utilization 0.92 \
    --quantization ascend \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --async-scheduling \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --kv-cache-dtype int8 \
    --attention_config.indexer_kv_dtype int8 \
    --additional-config '{"enable_dsa_cp": true, "enable_balance_scheduling": true, "enable_fused_mc2": 1, "enable_flashcomm1": true}' \
    --speculative-config '{"num_speculative_tokens": 3, "method": "deepseek_mtp", "enforce_eager": true}'

GLM-5.3-w8a8c8: can be deployed on 4 Atlas 800I A2 (64GB × 8).

分别在四个节点上运行以下脚本。

节点 0

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="xxx"
local_ip="xxx"

# The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)
node0_ip="xxx"

export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export VLLM_RPC_TIMEOUT=360000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=200
export HCCL_CONNECT_TIMEOUT=120
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export ACL_OP_INIT_MODE=1
export CPU_AFFINITY_CONF=1
export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_ENGINE_READY_TIMEOUT_S=1200

# Ensure the model path matches the directory recorded during download
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/GLM-5.3-w8a8c8 \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 135000 \
    --data-parallel-size 4 \
    --data-parallel-size-local 1 \
    --data-parallel-start-rank 0 \
    --data-parallel-address "${node0_ip}" \
    --data-parallel-rpc-port 12980 \
    --tensor-parallel-size 8 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name glm-5 \
    --reasoning-parser glm47 \
    --safetensors-load-strategy prefetch \
    --max-num-seqs 128 \
    --max-num-batched-tokens 8192 \
    --trust-remote-code \
    --quantization ascend \
    --gpu-memory-utilization 0.92 \
    --speculative-config '{"num_speculative_tokens": 3, "method": "deepseek_mtp", "enforce_eager": true}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --kv-cache-dtype int8 \
    --attention_config.indexer_kv_dtype int8 \
    --additional-config '{"enable_dsa_cp": true, "enable_balance_scheduling": true, "fuse_muls_add": true, "multistream_overlap_shared_expert": true, "enable_flashcomm1": true}' \
    --enable-prefix-caching \
    --async-scheduling \
    --api-server-count 1

节点 1-3

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="xxx"
local_ip="xxx"

# The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)
node0_ip="xxx"

# node1: dp_start_rank=1, node2: dp_start_rank=2, node3: dp_start_rank=3
dp_start_rank=1

export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export VLLM_RPC_TIMEOUT=360000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=200
export HCCL_CONNECT_TIMEOUT=120
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export ACL_OP_INIT_MODE=1
export CPU_AFFINITY_CONF=1
export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_ENGINE_READY_TIMEOUT_S=1200

# Ensure the model path matches the directory recorded during download
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/GLM-5.3-w8a8c8 \
    --host 0.0.0.0 \
    --port 8000 \
    --headless \
    --max-model-len 135000 \
    --data-parallel-size 4 \
    --data-parallel-size-local 1 \
    --data-parallel-start-rank ${dp_start_rank} \
    --data-parallel-address "${node0_ip}" \
    --data-parallel-rpc-port 12980 \
    --tensor-parallel-size 8 \
    --enable-expert-parallel \
    --seed 1024 \
    --served-model-name glm-5 \
    --reasoning-parser glm47 \
    --safetensors-load-strategy prefetch \
    --max-num-seqs 128 \
    --max-num-batched-tokens 8192 \
    --trust-remote-code \
    --quantization ascend \
    --gpu-memory-utilization 0.92 \
    --speculative-config '{"num_speculative_tokens": 3, "method": "deepseek_mtp", "enforce_eager": true}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --kv-cache-dtype int8 \
    --attention_config.indexer_kv_dtype int8 \
    --additional-config '{"enable_dsa_cp": true, "enable_balance_scheduling": true,"fuse_muls_add": true, "multistream_overlap_shared_expert": true, "enable_flashcomm1": true}' \
    --enable-prefix-caching \
    --async-scheduling

关键参数说明：

下面仅描述该模型/场景特有的关键参数。max-model-len 和 max-num-seqs 需要根据实际使用场景进行设置。

多节点网络和数据并行配置：

HCCL_IF_IP、GLOO_SOCKET_IFNAME、TP_SOCKET_IFNAME、HCCL_SOCKET_IFNAME：用于多节点通信的网络接口配置。将 nic_name 设置为网络接口名称（通过 ifconfig 获取），将 local_ip 设置为当前节点的 IP 地址。必须在每个节点上正确配置这些参数，才能实现成功的多节点通信。
--data-parallel-size 8：所有节点上的数据并行 rank 总数（此场景中每个节点 4 个 rank）。
--data-parallel-size-local 4：当前节点上的数据并行 rank 数量。
--data-parallel-start-rank：当前节点上数据并行 rank 的起始偏移量。节点 0 使用 0，节点 1 使用 4。
--data-parallel-address：数据并行主节点（节点 0）的 IP 地址。必须与主节点的 local_ip 一致。
--data-parallel-rpc-port 12980：数据并行主节点通信的 RPC 端口。所有节点必须保持一致。
--headless：表示非主节点（在节点 1-3 上使用）。不要在节点 0 上使用。

A2 特有的环境变量：

CPU_AFFINITY_CONF=1：为工作进程启用CPU核心亲和性绑定。
ACL_OP_INIT_MODE=1：ACL算子初始化模式，用于加速算子编译。
VLLM_RPC_TIMEOUT=360000 / VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3000 / HCCL_EXEC_TIMEOUT=200 / HCCL_CONNECT_TIMEOUT=120 / VLLM_ENGINE_READY_TIMEOUT_S=1200：在较慢的A2平台上进行多节点启动和模型执行的超时设置。如果引擎未能及时就绪，请增大这些值。

注意： 此场景启用了additional_config.enable_fused_mc2=1（融合的dispatch_ffn_combine/mega_moe算子）。融合MC2与multistream_overlap_shared_expert存在冲突——这两种优化不能同时启用（当融合MC2开启时，运行时将强制禁用multistream_overlap_shared_expert）。

5.1.2 Prefill-Decode Disaggregation¶

This A3 reference serves GLM-5.3 W8A8C8 with a 200,000-token maximum sequence length for the 198K workload. It uses one Prefill engine spanning P0/P1 with TP16 and PP2, and sixteen TP2 Decode engines on D0/D1. Section 5.1.3 adds MemCache KV pooling to the same deployment.

5.1.2.1 Topology and Prerequisites¶

Role	Nodes	Global DP	TP	PP	Placement	Devices per role
Prefill	P0, P1	1	16	2	One PP stage per node	32
Decode	D0, D1	16	2	1	Eight DP ranks per node	32

Each container exposes devices 0-15. P0 exposes the only Prefill API; P1 runs the second PP stage without an API server. Decode ranks 0-7 run on D0 and ranks 8-15 on D1, with API ports 8000-8007 on each node.

Use the A3 v0.30.0rc1-a3 image from Section 4.1 on all four nodes.

Prepare the same weights and software on all four nodes. Set MODEL_PATH and PYTHON_LIB_DIR in each role script to the installed paths. Install fastokens on Prefill nodes for VLLM_USE_FASTOKENS=1.
Verify multi-node communication. Set LOCAL_IP="<PREFILL_NODE_IP>" on each Prefill node and LOCAL_IP="<DECODE_NODE_IP>" on each Decode node to that node's IP. Set NIC_NAME="<NETWORK_INTERFACE>" to its matching interface and MODEL_PATH="<YOUR_MODEL_PATH>" to its model weight directory.
Reserve the PP master port 7060, Decode DP RPC port 16600, API ports, proxy port 8000, and the topology-derived Mooncake transfer/handshake ports. Prefill and Decode run on separate node groups.
Keep the 78-layer PP partition 42,36 consistent in the Prefill environment and both roles' connector topology. Both roles explicitly use int8 KV cache and int8 indexer KV cache for these W8A8C8 weights.

5.1.2.2 Prepare the Scripts¶

Before you start, please

prepare the script launch_online_dp.py on each Decode node:

import argparse
import multiprocessing
import subprocess
import sys


def run_command(args, local_rank):
    dp_rank = args.dp_rank_start + local_rank
    devices = ",".join(str(i) for i in range(
        local_rank * args.tp_size, (local_rank + 1) * args.tp_size))
    subprocess.run([
        "bash", args.script, devices, str(args.vllm_start_port + local_rank),
        str(args.dp_size), str(dp_rank), args.dp_address,
        str(args.dp_rpc_port), str(args.tp_size),
    ], check=True)


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--dp-size", type=int, required=True)
    parser.add_argument("--tp-size", type=int, required=True)
    parser.add_argument("--dp-size-local", type=int, required=True)
    parser.add_argument("--dp-rank-start", type=int, required=True)
    parser.add_argument("--dp-address", required=True)
    parser.add_argument("--dp-rpc-port", type=int, default=16600)
    parser.add_argument("--vllm-start-port", type=int, default=8000)
    parser.add_argument("--script", default="./run_dp_template.sh")
    args = parser.parse_args()
    if min(args.dp_size, args.tp_size, args.dp_size_local) <= 0:
        parser.error("Parallel sizes must be positive")
    if args.dp_rank_start < 0 or args.dp_rank_start + args.dp_size_local > args.dp_size:
        parser.error("Local ranks must be within the global DP group")
    if args.dp_size_local * args.tp_size > 16:
        parser.error("This A3 configuration exposes 16 devices per node")
    processes = []
    for local_rank in range(args.dp_size_local):
        process = multiprocessing.Process(target=run_command, args=(args, local_rank))
        processes.append(process)
        process.start()
    for process in processes:
        process.join()
    return 1 if any(process.exitcode != 0 for process in processes) else 0


if __name__ == "__main__":
    sys.exit(main())

prepare the script run_dp_template.sh on each node.

Prefill node 0

#!/usr/bin/env bash

LOCAL_IP="<PREFILL_NODE_IP>"
NIC_NAME="<NETWORK_INTERFACE>"
MODEL_PATH="<YOUR_MODEL_PATH>"
PYTHON_LIB_DIR=/path/to/python/lib

# Set the current node IP and shared P0 master IP before starting.

export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=1800
export HCCL_CONNECT_TIMEOUT=1800
export ASCEND_TRANSFER_TIMEOUT=10000

NODE_P0_IP="$LOCAL_IP"

export VLLM_HOST_IP="$LOCAL_IP"
export HCCL_IF_IP="$LOCAL_IP"
export GLOO_SOCKET_IFNAME="$NIC_NAME"
export TP_SOCKET_IFNAME="$NIC_NAME"
export HCCL_SOCKET_IFNAME="$NIC_NAME"

export LD_LIBRARY_PATH="${PYTHON_LIB_DIR}:/usr/local/lib:${LD_LIBRARY_PATH}"

export HCCL_BUFFSIZE=1024
export HCCL_OP_EXPANSION_MODE="AIV"

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1

export VLLM_USE_V2_MODEL_RUNNER=0

export VLLM_USE_FASTOKENS=1
export VLLM_PP_LAYER_PARTITION="42,36"

export ASCEND_LOCAL_COMM_RES='{"version":"1.2"}'

exec vllm serve "${MODEL_PATH}" \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 16 \
    --enable-expert-parallel \
    --pipeline-parallel-size 2 \
    --distributed-executor-backend mp \
    --master-addr "$NODE_P0_IP" \
    --master-port 7060 \
    --nnodes 2 \
    --node-rank 0 \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --seed 1024 \
    --served-model-name glm5 \
    --max-model-len 200000 \
    --max-num-batched-tokens 16384 \
    --trust-remote-code \
    --max-num-seqs 64 \
    --gpu-memory-utilization 0.92 \
    --async-scheduling \
    --quantization ascend \
    --safetensors-load-strategy 'prefetch' \
    --enable-auto-tool-choice \
    --tool-call-parser glm47 \
    --reasoning-parser glm47 \
    --enforce-eager \
    --kv-cache-dtype int8 \
    --attention_config.indexer_kv_dtype int8 \
    --additional-config '{
        "enable_dsa_cp":true,
        "enable_fused_mc2": 1,
        "enable_flashcomm1": true
    }' \
    --speculative-config '{"num_speculative_tokens": 1,  "method":"deepseek_mtp","enforce_eager":true}' \
    --kv-transfer-config '{
        "kv_connector": "MooncakeConnectorV2",
        "kv_role": "kv_producer",
        "kv_port": "30000",
        "engine_id": "glm53-prefill",
        "kv_connector_extra_config": {
            "use_ascend_direct": true,
            "prefill": {"dp_size": 1, "pp_size": 2, "tp_size": 16, "pp_layer_partition": "42,36"},
            "decode": {"dp_size": 16, "tp_size": 2}
        }
    }'

Prefill node 1

#!/usr/bin/env bash

LOCAL_IP="<PREFILL_NODE_IP>"
NIC_NAME="<NETWORK_INTERFACE>"
MODEL_PATH="<YOUR_MODEL_PATH>"
PYTHON_LIB_DIR=/path/to/python/lib

# Set the current node IP and shared P0 master IP before starting.

export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=1800
export HCCL_CONNECT_TIMEOUT=1800
export ASCEND_TRANSFER_TIMEOUT=10000

NODE_P0_IP="<PREFILL_NODE0_IP>"

export VLLM_HOST_IP="$LOCAL_IP"
export HCCL_IF_IP="$LOCAL_IP"
export GLOO_SOCKET_IFNAME="$NIC_NAME"
export TP_SOCKET_IFNAME="$NIC_NAME"
export HCCL_SOCKET_IFNAME="$NIC_NAME"

export LD_LIBRARY_PATH="${PYTHON_LIB_DIR}:/usr/local/lib:${LD_LIBRARY_PATH}"

export HCCL_BUFFSIZE=1024
export HCCL_OP_EXPANSION_MODE="AIV"

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1

export VLLM_USE_V2_MODEL_RUNNER=0

export VLLM_USE_FASTOKENS=1
export VLLM_PP_LAYER_PARTITION="42,36"

export ASCEND_LOCAL_COMM_RES='{"version":"1.2"}'

exec vllm serve "${MODEL_PATH}" \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 16 \
    --enable-expert-parallel \
    --pipeline-parallel-size 2 \
    --distributed-executor-backend mp \
    --master-addr "$NODE_P0_IP" \
    --master-port 7060 \
    --nnodes 2 \
    --node-rank 1 \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --seed 1024 \
    --served-model-name glm5 \
    --max-model-len 200000 \
    --max-num-batched-tokens 16384 \
    --trust-remote-code \
    --max-num-seqs 64 \
    --gpu-memory-utilization 0.92 \
    --async-scheduling \
    --quantization ascend \
    --safetensors-load-strategy 'prefetch' \
    --enable-auto-tool-choice \
    --tool-call-parser glm47 \
    --reasoning-parser glm47 \
    --enforce-eager \
    --kv-cache-dtype int8 \
    --attention_config.indexer_kv_dtype int8 \
    --additional-config '{
        "enable_dsa_cp":true,
        "enable_fused_mc2": 1,
        "enable_flashcomm1": true
    }' \
    --speculative-config '{"num_speculative_tokens": 1,  "method":"deepseek_mtp","enforce_eager":true}' \
    --kv-transfer-config '{
        "kv_connector": "MooncakeConnectorV2",
        "kv_role": "kv_producer",
        "kv_port": "30000",
        "engine_id": "glm53-prefill",
        "kv_connector_extra_config": {
            "use_ascend_direct": true,
            "prefill": {"dp_size": 1, "pp_size": 2, "tp_size": 16, "pp_layer_partition": "42,36"},
            "decode": {"dp_size": 16, "tp_size": 2}
        }
    }' \
    --headless

Decode node 0

#!/usr/bin/env bash

LOCAL_IP="<DECODE_NODE_IP>"
NIC_NAME="<NETWORK_INTERFACE>"
MODEL_PATH="<YOUR_MODEL_PATH>"
PYTHON_LIB_DIR=/path/to/python/lib

# Arguments $1-$7 are supplied by launch_online_dp.py.

export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=1800
export HCCL_CONNECT_TIMEOUT=1800
export ASCEND_TRANSFER_TIMEOUT=10000


export VLLM_HOST_IP="$LOCAL_IP"
export HCCL_IF_IP="$LOCAL_IP"
export GLOO_SOCKET_IFNAME="$NIC_NAME"
export TP_SOCKET_IFNAME="$NIC_NAME"
export HCCL_SOCKET_IFNAME="$NIC_NAME"

export LD_LIBRARY_PATH="${PYTHON_LIB_DIR}:/usr/local/lib:${LD_LIBRARY_PATH}"

export HCCL_BUFFSIZE=1024
export HCCL_OP_EXPANSION_MODE="AIV"

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1

export VLLM_USE_V2_MODEL_RUNNER=0
export ASCEND_RT_VISIBLE_DEVICES=$1
export ASCEND_LOCAL_COMM_RES='{"version":"1.2"}'

exec vllm serve "${MODEL_PATH}" \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --seed 1024 \
    --served-model-name glm5 \
    --max-model-len 200000 \
    --max-num-batched-tokens 256 \
    --trust-remote-code \
    --max-num-seqs 64 \
    --gpu-memory-utilization 0.92 \
    --async-scheduling \
    --quantization ascend \
    --safetensors-load-strategy 'prefetch' \
    --enable-auto-tool-choice \
    --tool-call-parser glm47 \
    --reasoning-parser glm47 \
    --kv-cache-dtype int8 \
    --attention_config.indexer_kv_dtype int8 \
    --additional-config '{
        "recompute_scheduler_enable": true,
        "enable_fused_mc2": 1
    }' \
    --speculative-config '{"num_speculative_tokens": 5,  "method":"deepseek_mtp","enforce_eager":true}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --kv-transfer-config '{
        "kv_connector": "MooncakeConnectorV2",
        "kv_role": "kv_consumer",
        "kv_port": "30100",
        "engine_id": "glm53-decode-dp'"$4"'",
        "kv_connector_extra_config": {
            "use_ascend_direct": true,
            "prefill": {"dp_size": 1, "pp_size": 2, "tp_size": 16, "pp_layer_partition": "42,36"},
            "decode": {"dp_size": 16, "tp_size": 2}
        }
    }'

Decode node 1

#!/usr/bin/env bash

LOCAL_IP="<DECODE_NODE_IP>"
NIC_NAME="<NETWORK_INTERFACE>"
MODEL_PATH="<YOUR_MODEL_PATH>"
PYTHON_LIB_DIR=/path/to/python/lib

# Arguments $1-$7 are supplied by launch_online_dp.py.

export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=1800
export HCCL_CONNECT_TIMEOUT=1800
export ASCEND_TRANSFER_TIMEOUT=10000


export VLLM_HOST_IP="$LOCAL_IP"
export HCCL_IF_IP="$LOCAL_IP"
export GLOO_SOCKET_IFNAME="$NIC_NAME"
export TP_SOCKET_IFNAME="$NIC_NAME"
export HCCL_SOCKET_IFNAME="$NIC_NAME"

export LD_LIBRARY_PATH="${PYTHON_LIB_DIR}:/usr/local/lib:${LD_LIBRARY_PATH}"

export HCCL_BUFFSIZE=1024
export HCCL_OP_EXPANSION_MODE="AIV"

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1

export VLLM_USE_V2_MODEL_RUNNER=0
export ASCEND_RT_VISIBLE_DEVICES=$1
export ASCEND_LOCAL_COMM_RES='{"version":"1.2"}'

exec vllm serve "${MODEL_PATH}" \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --seed 1024 \
    --served-model-name glm5 \
    --max-model-len 200000 \
    --max-num-batched-tokens 256 \
    --trust-remote-code \
    --max-num-seqs 64 \
    --gpu-memory-utilization 0.92 \
    --async-scheduling \
    --quantization ascend \
    --safetensors-load-strategy 'prefetch' \
    --enable-auto-tool-choice \
    --tool-call-parser glm47 \
    --reasoning-parser glm47 \
    --kv-cache-dtype int8 \
    --attention_config.indexer_kv_dtype int8 \
    --additional-config '{
        "recompute_scheduler_enable": true,
        "enable_fused_mc2": 1
    }' \
    --speculative-config '{"num_speculative_tokens": 5,  "method":"deepseek_mtp","enforce_eager":true}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --kv-transfer-config '{
        "kv_connector": "MooncakeConnectorV2",
        "kv_role": "kv_consumer",
        "kv_port": "30100",
        "engine_id": "glm53-decode-dp'"$4"'",
        "kv_connector_extra_config": {
            "use_ascend_direct": true,
            "prefill": {"dp_size": 1, "pp_size": 2, "tp_size": 16, "pp_layer_partition": "42,36"},
            "decode": {"dp_size": 16, "tp_size": 2}
        }
    }'

5.1.2.3 Start the Engines¶

Once the preparation is done, you can start the server with the following command on each node:

Prefill node 0
```
bash run_dp_template.sh
```
Prefill node 1
```
bash run_dp_template.sh
```

Decode node 0

D0_IP="<DECODE_NODE0_IP>"
python launch_online_dp.py --dp-size 16 --tp-size 2 --dp-size-local 8 --dp-rank-start 0 --dp-address "$D0_IP" --dp-rpc-port 16600 --vllm-start-port 8000

Decode node 1

D0_IP="<DECODE_NODE0_IP>"
python launch_online_dp.py --dp-size 16 --tp-size 2 --dp-size-local 8 --dp-rank-start 8 --dp-address "$D0_IP" --dp-rpc-port 16600 --vllm-start-port 8000

5.1.2.4 Start the Proxy¶

Replace each node's IP and installation-path placeholders before starting. Run the four commands in Section 5.1.2.3 in separate terminals. Keep the master processes running while the other nodes join; start both nodes of each group before waiting for the group to become healthy.

After the Prefill API on P0:8000 and all sixteen Decode APIs respond to GET /v1/models, start the PD load-balancing proxy from the matching checkout on a separate host that can reach all P/D API endpoints. The proxy listens on port 8000; using a separate host avoids a port conflict with P0:8000 and the Decode APIs on 8000-8007. Register P0 once and all sixteen Decode endpoints. P1 is not a separate Prefill API endpoint.

P0_IP="<P0_IP>"
D0_IP="<D0_IP>"
D1_IP="<D1_IP>"

python examples/disaggregated_prefill_v1/load_balance_proxy_server_example.py \
    --host 0.0.0.0 --port 8000 \
    --prefiller-hosts "$P0_IP" --prefiller-ports 8000 \
    --decoder-hosts "$D0_IP" "$D0_IP" "$D0_IP" "$D0_IP" \
        "$D0_IP" "$D0_IP" "$D0_IP" "$D0_IP" \
        "$D1_IP" "$D1_IP" "$D1_IP" "$D1_IP" \
        "$D1_IP" "$D1_IP" "$D1_IP" "$D1_IP" \
    --decoder-ports 8000 8001 8002 8003 8004 8005 8006 8007 \
        8000 8001 8002 8003 8004 8005 8006 8007

Key parameters:

Prefill uses TP16/PP2 over two nodes with the mp executor and VLLM_PP_LAYER_PARTITION="42,36". Its batch budget is 16,384 tokens, maximum sequence count is 64, and it runs in eager mode with one MTP token.
Decode uses DP16/TP2, a 256-token batch budget, 64 sequences, five MTP tokens with an eager draft model, and FULL_DECODE_ONLY graph mode. recompute_scheduler_enable=true allows recomputation scheduling.
Both roles retain chunked prefill, prefix caching, async scheduling, seed 1024, memory utilization 0.92, and fused MC2. Prefill also enables DSA CP and FlashComm1. Both use glm47 tool and reasoning parsers and the served model name glm5.
MooncakeConnectorV2 declares the same Prefill DP1/PP2/TP16 and Decode DP16/TP2 topology on both sides. The non-pooled base KV ports are 30000 on Prefill and 30100 on Decode; reserve derived worker ports.
The examples give Prefill and each Decode engine distinct engine_id values so P-to-D peer registration cannot conflict.

5.1.3 MemCache KV Cache Pool Deployment¶

This extends the same four-node PP/PD topology with MultiConnector: MooncakeConnectorV2 transfers KV cache from Prefill to Decode, and AscendStoreConnector provides non-layerwise MemCache pooling. Retain the launcher, PP placement, and proxy routing from Section 5.1.2.

5.1.3.1 Prepare MemCache¶

Complete MemCache backend setup, including the hardware/CANN prerequisites, memfabric-hybrid and memcache-hybrid installation, configuration files, and metadata service. All P/D instances must access the same pool; connector arguments do not start the MemCache service. Set MEMCACHE_ROOT to the installed memcache_hybrid directory and PYTHON_LIB_DIR to the library directory of the Python installation actually used by vLLM on each node.

5.1.3.2 Prepare the Scripts¶

Before you start, please

prepare the script launch_online_dp.py on each Decode node, using the launcher in Section 5.1.2.2.

prepare the pooled script run_dp_template.sh on each node.

Prefill node 0

#!/usr/bin/env bash

LOCAL_IP="<PREFILL_NODE_IP>"
NIC_NAME="<NETWORK_INTERFACE>"
MODEL_PATH="<YOUR_MODEL_PATH>"
PYTHON_LIB_DIR=/path/to/python/lib
MEMCACHE_ROOT=/path/to/site-packages/memcache_hybrid

# Set the current node IP and shared P0 master IP before starting.

export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=1800
export HCCL_CONNECT_TIMEOUT=1800
export ASCEND_TRANSFER_TIMEOUT=10000

NODE_P0_IP="$LOCAL_IP"

export VLLM_HOST_IP="$LOCAL_IP"
export HCCL_IF_IP="$LOCAL_IP"
export GLOO_SOCKET_IFNAME="$NIC_NAME"
export TP_SOCKET_IFNAME="$NIC_NAME"
export HCCL_SOCKET_IFNAME="$NIC_NAME"

export LD_LIBRARY_PATH="${PYTHON_LIB_DIR}:/usr/local/lib:${LD_LIBRARY_PATH}"

export HCCL_BUFFSIZE=1024
export HCCL_OP_EXPANSION_MODE="AIV"

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1

export VLLM_USE_V2_MODEL_RUNNER=0

export VLLM_USE_FASTOKENS=1
export VLLM_PP_LAYER_PARTITION="42,36"

export PYTHONHASHSEED=0
export ACL_OP_INIT_MODE=1
export MMC_LOCAL_CONFIG_PATH="${MEMCACHE_ROOT}/config/mmc-local.conf"
export LD_LIBRARY_PATH="${MEMCACHE_ROOT}/lib:${PYTHON_LIB_DIR}:${LD_LIBRARY_PATH}"

export ASCEND_LOCAL_COMM_RES='{"version":"1.2"}'

exec vllm serve "${MODEL_PATH}" \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 16 \
    --enable-expert-parallel \
    --pipeline-parallel-size 2 \
    --distributed-executor-backend mp \
    --master-addr "$NODE_P0_IP" \
    --master-port 7060 \
    --nnodes 2 \
    --node-rank 0 \
    --enable-chunked-prefill \
    --async-scheduling \
    --enable-prefix-caching \
    --seed 1024 \
    --served-model-name glm5 \
    --max-model-len 200000 \
    --max-num-batched-tokens 16384 \
    --trust-remote-code \
    --max-num-seqs 64 \
    --gpu-memory-utilization 0.92 \
    --quantization ascend \
    --safetensors-load-strategy 'prefetch' \
    --enable-auto-tool-choice \
    --tool-call-parser glm47 \
    --reasoning-parser glm47 \
    --enforce-eager \
    --kv-cache-dtype int8 \
    --attention_config.indexer_kv_dtype int8 \
    --additional-config '{
        "enable_dsa_cp":true,
        "enable_fused_mc2": 1,
        "enable_flashcomm1": true
    }' \
    --speculative-config '{"num_speculative_tokens": 1,  "method":"deepseek_mtp","enforce_eager":true}' \
    --kv-transfer-config '{
        "kv_connector": "MultiConnector",
        "kv_role": "kv_producer",
        "kv_port": "30000",
        "engine_id": "glm53-prefill",
        "kv_connector_extra_config": {
            "connectors":[
            {
                "kv_connector": "MooncakeConnectorV2",
                "kv_role": "kv_producer",
                "kv_port": "30000",
                "kv_connector_extra_config": {
                    "use_ascend_direct": true,
                    "prefill": {"dp_size": 1, "pp_size": 2, "tp_size": 16, "pp_layer_partition": "42,36"},
                    "decode": { "dp_size": 16, "tp_size": 2}
                }
            },
            {
                "kv_connector": "AscendStoreConnector",
                "kv_role": "kv_producer",
                "kv_connector_extra_config": {
                    "lookup_rpc_port":"0",
                    "backend": "memcache",
                    "use_layerwise": false
                }
            }
            ]
        }
    }'

Prefill node 1

#!/usr/bin/env bash

LOCAL_IP="<PREFILL_NODE_IP>"
NIC_NAME="<NETWORK_INTERFACE>"
MODEL_PATH="<YOUR_MODEL_PATH>"
PYTHON_LIB_DIR=/path/to/python/lib
MEMCACHE_ROOT=/path/to/site-packages/memcache_hybrid

# Set the current node IP and shared P0 master IP before starting.

export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=1800
export HCCL_CONNECT_TIMEOUT=1800
export ASCEND_TRANSFER_TIMEOUT=10000

NODE_P0_IP="<PREFILL_NODE0_IP>"

export VLLM_HOST_IP="$LOCAL_IP"
export HCCL_IF_IP="$LOCAL_IP"
export GLOO_SOCKET_IFNAME="$NIC_NAME"
export TP_SOCKET_IFNAME="$NIC_NAME"
export HCCL_SOCKET_IFNAME="$NIC_NAME"

export LD_LIBRARY_PATH="${PYTHON_LIB_DIR}:/usr/local/lib:${LD_LIBRARY_PATH}"

export HCCL_BUFFSIZE=1024
export HCCL_OP_EXPANSION_MODE="AIV"

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1

export VLLM_USE_V2_MODEL_RUNNER=0

export VLLM_USE_FASTOKENS=1
export VLLM_PP_LAYER_PARTITION="42,36"

export PYTHONHASHSEED=0
export ACL_OP_INIT_MODE=1
export MMC_LOCAL_CONFIG_PATH="${MEMCACHE_ROOT}/config/mmc-local.conf"
export LD_LIBRARY_PATH="${MEMCACHE_ROOT}/lib:${PYTHON_LIB_DIR}:${LD_LIBRARY_PATH}"

export ASCEND_LOCAL_COMM_RES='{"version":"1.2"}'

exec vllm serve "${MODEL_PATH}" \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 16 \
    --enable-expert-parallel \
    --pipeline-parallel-size 2 \
    --distributed-executor-backend mp \
    --master-addr "$NODE_P0_IP" \
    --master-port 7060 \
    --nnodes 2 \
    --node-rank 1 \
    --enable-chunked-prefill \
    --async-scheduling \
    --enable-prefix-caching \
    --seed 1024 \
    --served-model-name glm5 \
    --max-model-len 200000 \
    --max-num-batched-tokens 16384 \
    --trust-remote-code \
    --max-num-seqs 64 \
    --gpu-memory-utilization 0.92 \
    --quantization ascend \
    --safetensors-load-strategy 'prefetch' \
    --enable-auto-tool-choice \
    --tool-call-parser glm47 \
    --reasoning-parser glm47 \
    --enforce-eager \
    --kv-cache-dtype int8 \
    --attention_config.indexer_kv_dtype int8 \
    --additional-config '{
        "enable_dsa_cp":true,
        "enable_fused_mc2": 1,
        "enable_flashcomm1": true
    }' \
    --speculative-config '{"num_speculative_tokens": 1,  "method":"deepseek_mtp","enforce_eager":true}' \
    --kv-transfer-config '{
        "kv_connector": "MultiConnector",
        "kv_role": "kv_producer",
        "kv_port": "30000",
        "engine_id": "glm53-prefill",
        "kv_connector_extra_config": {
            "connectors":[
            {
                "kv_connector": "MooncakeConnectorV2",
                "kv_role": "kv_producer",
                "kv_port": "30000",
                "kv_connector_extra_config": {
                    "use_ascend_direct": true,
                    "prefill": {"dp_size": 1, "pp_size": 2, "tp_size": 16, "pp_layer_partition": "42,36"},
                    "decode": { "dp_size": 16, "tp_size": 2}
                }
            },
            {
                "kv_connector": "AscendStoreConnector",
                "kv_role": "kv_producer",
                "kv_connector_extra_config": {
                    "lookup_rpc_port":"0",
                    "backend": "memcache",
                    "use_layerwise": false
                }
            }
            ]
        }
    }' \
    --headless

Decode node 0

#!/usr/bin/env bash

LOCAL_IP="<DECODE_NODE_IP>"
NIC_NAME="<NETWORK_INTERFACE>"
MODEL_PATH="<YOUR_MODEL_PATH>"
PYTHON_LIB_DIR=/path/to/python/lib
MEMCACHE_ROOT=/path/to/site-packages/memcache_hybrid

# Arguments $1-$7 are supplied by launch_online_dp.py.

export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=1800
export HCCL_CONNECT_TIMEOUT=1800
export ASCEND_TRANSFER_TIMEOUT=10000


export VLLM_HOST_IP="$LOCAL_IP"
export HCCL_IF_IP="$LOCAL_IP"
export GLOO_SOCKET_IFNAME="$NIC_NAME"
export TP_SOCKET_IFNAME="$NIC_NAME"
export HCCL_SOCKET_IFNAME="$NIC_NAME"

export LD_LIBRARY_PATH="${PYTHON_LIB_DIR}:/usr/local/lib:${LD_LIBRARY_PATH}"

export HCCL_BUFFSIZE=1024
export HCCL_OP_EXPANSION_MODE="AIV"

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1

export VLLM_USE_V2_MODEL_RUNNER=0

export ASCEND_RT_VISIBLE_DEVICES=$1

export PYTHONHASHSEED=0
export ACL_OP_INIT_MODE=1
export MMC_LOCAL_CONFIG_PATH="${MEMCACHE_ROOT}/config/mmc-local.conf"
export LD_LIBRARY_PATH="${MEMCACHE_ROOT}/lib:${PYTHON_LIB_DIR}:${LD_LIBRARY_PATH}"

export ASCEND_LOCAL_COMM_RES='{"version":"1.2"}'

exec vllm serve "${MODEL_PATH}" \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --seed 1024 \
    --served-model-name glm5 \
    --max-model-len 200000 \
    --max-num-batched-tokens 256 \
    --trust-remote-code \
    --max-num-seqs 64 \
    --gpu-memory-utilization 0.92 \
    --async-scheduling \
    --quantization ascend \
    --safetensors-load-strategy 'prefetch' \
    --enable-auto-tool-choice \
    --tool-call-parser glm47 \
    --reasoning-parser glm47 \
    --kv-cache-dtype int8 \
    --attention_config.indexer_kv_dtype int8 \
    --additional-config '{
        "recompute_scheduler_enable": true,
        "enable_fused_mc2": 1
    }' \
    --speculative-config '{"num_speculative_tokens": 5,  "method":"deepseek_mtp","enforce_eager":true}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --kv-transfer-config '{
        "kv_connector": "MultiConnector",
        "kv_role": "kv_consumer",
        "kv_port": "30200",
        "engine_id": "glm53-decode-dp'"$4"'",
        "kv_connector_extra_config": {
            "connectors":[
            {
                "kv_connector": "MooncakeConnectorV2",
                "kv_role": "kv_consumer",
                "kv_port": "30200",
                "kv_connector_extra_config": {
                    "use_ascend_direct": true,
                    "prefill": {"dp_size": 1, "pp_size": 2, "tp_size": 16, "pp_layer_partition": "42,36"},
                    "decode": { "dp_size": 16, "tp_size": 2}
                }
            },
            {
                "kv_connector": "AscendStoreConnector",
                "kv_role": "kv_consumer",
                "kv_connector_extra_config": {
                    "lookup_rpc_port":"0",
                    "backend": "memcache",
                    "use_layerwise": false
                }
            }
            ]
        }
    }'

Decode node 1

#!/usr/bin/env bash

LOCAL_IP="<DECODE_NODE_IP>"
NIC_NAME="<NETWORK_INTERFACE>"
MODEL_PATH="<YOUR_MODEL_PATH>"
PYTHON_LIB_DIR=/path/to/python/lib
MEMCACHE_ROOT=/path/to/site-packages/memcache_hybrid

# Arguments $1-$7 are supplied by launch_online_dp.py.

export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=1800
export HCCL_CONNECT_TIMEOUT=1800
export ASCEND_TRANSFER_TIMEOUT=10000


export VLLM_HOST_IP="$LOCAL_IP"
export HCCL_IF_IP="$LOCAL_IP"
export GLOO_SOCKET_IFNAME="$NIC_NAME"
export TP_SOCKET_IFNAME="$NIC_NAME"
export HCCL_SOCKET_IFNAME="$NIC_NAME"

export LD_LIBRARY_PATH="${PYTHON_LIB_DIR}:/usr/local/lib:${LD_LIBRARY_PATH}"

export HCCL_BUFFSIZE=1024
export HCCL_OP_EXPANSION_MODE="AIV"

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1

export VLLM_USE_V2_MODEL_RUNNER=0

export ASCEND_RT_VISIBLE_DEVICES=$1

export PYTHONHASHSEED=0
export ACL_OP_INIT_MODE=1
export MMC_LOCAL_CONFIG_PATH="${MEMCACHE_ROOT}/config/mmc-local.conf"
export LD_LIBRARY_PATH="${MEMCACHE_ROOT}/lib:${PYTHON_LIB_DIR}:${LD_LIBRARY_PATH}"

export ASCEND_LOCAL_COMM_RES='{"version":"1.2"}'

exec vllm serve "${MODEL_PATH}" \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --seed 1024 \
    --served-model-name glm5 \
    --max-model-len 200000 \
    --max-num-batched-tokens 256 \
    --trust-remote-code \
    --max-num-seqs 64 \
    --gpu-memory-utilization 0.92 \
    --async-scheduling \
    --quantization ascend \
    --safetensors-load-strategy 'prefetch' \
    --enable-auto-tool-choice \
    --tool-call-parser glm47 \
    --reasoning-parser glm47 \
    --kv-cache-dtype int8 \
    --attention_config.indexer_kv_dtype int8 \
    --additional-config '{
        "recompute_scheduler_enable": true,
        "enable_fused_mc2": 1
    }' \
    --speculative-config '{"num_speculative_tokens": 5,  "method":"deepseek_mtp","enforce_eager":true}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --kv-transfer-config '{
        "kv_connector": "MultiConnector",
        "kv_role": "kv_consumer",
        "kv_port": "30200",
        "engine_id": "glm53-decode-dp'"$4"'",
        "kv_connector_extra_config": {
            "connectors":[
            {
                "kv_connector": "MooncakeConnectorV2",
                "kv_role": "kv_consumer",
                "kv_port": "30200",
                "kv_connector_extra_config": {
                    "use_ascend_direct": true,
                    "prefill": {"dp_size": 1, "pp_size": 2, "tp_size": 16, "pp_layer_partition": "42,36"},
                    "decode": { "dp_size": 16, "tp_size": 2}
                }
            },
            {
                "kv_connector": "AscendStoreConnector",
                "kv_role": "kv_consumer",
                "kv_connector_extra_config": {
                    "lookup_rpc_port":"0",
                    "backend": "memcache",
                    "use_layerwise": false
                }
            }
            ]
        }
    }'

The pooled configuration uses base KV ports 30000 on Prefill and 30200 on Decode. The outer engine_id is unique per engine and is inherited by the child connectors. This differs from the supplied pooled reference's repeated "0" IDs to avoid Mooncake peer identity conflicts. lookup_rpc_port="0" is retained: Ascend Store uses an IPC endpoint that also includes the DP rank, and P/D run on separate hosts. It is not a TCP listener on port zero. backend="memcache" and use_layerwise=false are set on both sides.

5.1.3.3 Start the Engines¶

Start the MemCache metadata service and complete its configuration on all four nodes before starting vLLM.

Once the preparation is done, you can start the server with the following command on each node:

Prefill node 0
```
bash run_dp_template.sh
```
Prefill node 1
```
bash run_dp_template.sh
```

Decode node 0

D0_IP="<DECODE_NODE0_IP>"
python launch_online_dp.py --dp-size 16 --tp-size 2 --dp-size-local 8 --dp-rank-start 0 --dp-address "$D0_IP" --dp-rpc-port 16600 --vllm-start-port 8000

Decode node 1

D0_IP="<DECODE_NODE0_IP>"
python launch_online_dp.py --dp-size 16 --tp-size 2 --dp-size-local 8 --dp-rank-start 8 --dp-address "$D0_IP" --dp-rpc-port 16600 --vllm-start-port 8000

5.1.3.4 Start the Proxy and Verify Pooling¶

Start the MemCache metadata service and complete its configuration on all four nodes before starting vLLM.
Start P0, P1, D0, and D1 with the commands in Section 5.1.3.3. Both Prefill scripts already specify port 8000.
Wait for P0:8000 and all sixteen Decode APIs to become healthy. Start the proxy from Section 5.1.2.4 on a separate host, using the same Prefill endpoint and Decode ports 8000-8007. Its public port is 8000.
Send requests to the proxy using the served model name glm5:

curl "http://<proxy_ip>:8000/v1/chat/completions" \
    -H "Content-Type: application/json" \
    -d '{
        "model": "glm5",
        "messages": [{"role": "user", "content": "Explain prefill and decode in an LLM."}],
        "max_tokens": 128,
        "temperature": 0
    }'

For either mode, check P/D logs for successful KV transfer. For pooling, also check MemCache initialization, then warm the pool with a prompt longer than one cache block and repeat requests with the same prefix. Inspect Ascend Store/MemCache saves, lookups, loads, and hit metrics. A successful response or a local prefix-cache hit alone does not demonstrate shared pool reuse; verify a pool load when the prefix is absent from the engine's local cache. These references do not establish A2, 1M-context, or other parallel-layout support, and do not add accuracy or performance claims for the 198K workload.

5.2 1M 上下文配置¶

5.2.1 1M Prefill-Decode Disaggregation (DCP)¶

This configuration uses GLM-5.3 W8A8C8 weights on four Atlas 800I A3 nodes, with 16 logical NPU devices per node. Use the A3 image from Docker Image Installation on all nodes. The maximum context length is 1,048,576 tokens, including input and output.

5.2.1.1 Topology and Prerequisites¶

Role	Nodes	Global DP	TP	PP	DCP	Local DP ranks	Devices per role
Prefill	P0, P1	2	16	1	16	One per node	32
Decode	D0, D1	4	8	1	8	Two per node	32

P0 and P1 run DP ranks 0 and 1. D0 runs ranks 0-1 and D1 runs ranks 2-3. Prefill context parallel size is 1 on both roles. This DCP layout is separate from the PP2 layout in Sections 5.1.2 and 5.1.3.

Prepare identical weights on all four nodes and verify multi-node communication.
Set LOCAL_IP, NIC_NAME, and MODEL_PATH separately on each node. Replace the master address placeholders in the startup commands with P0's IP for Prefill and D0's IP for Decode.
Reserve DP RPC ports 16591 and 16600, API port 8000 on each Prefill node, API ports 8000-8001 on each Decode node, and Mooncake transfer ports derived from the configured base ports 30000 and 30200. Run the public proxy on a separate host at port 8000.

5.2.1.2 Prepare the Scripts¶

Before you start, please

prepare the script launch_online_dp.py on each node:

import argparse
import multiprocessing
import subprocess
import sys


def run_command(args, local_rank):
    dp_rank = args.dp_rank_start + local_rank
    devices = ",".join(str(i) for i in range(
        local_rank * args.tp_size, (local_rank + 1) * args.tp_size))
    subprocess.run([
        "bash", args.script, devices, str(args.vllm_start_port + local_rank),
        str(args.dp_size), str(dp_rank), args.dp_address,
        str(args.dp_rpc_port), str(args.tp_size),
    ], check=True)


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--dp-size", type=int, required=True)
    parser.add_argument("--tp-size", type=int, required=True)
    parser.add_argument("--dp-size-local", type=int, required=True)
    parser.add_argument("--dp-rank-start", type=int, required=True)
    parser.add_argument("--dp-address", required=True)
    parser.add_argument("--dp-rpc-port", type=int, default=16600)
    parser.add_argument("--vllm-start-port", type=int, default=8000)
    parser.add_argument("--script", default="./run_dp_template.sh")
    args = parser.parse_args()
    if min(args.dp_size, args.tp_size, args.dp_size_local) <= 0:
        parser.error("Parallel sizes must be positive")
    if args.dp_rank_start < 0 or args.dp_rank_start + args.dp_size_local > args.dp_size:
        parser.error("Local ranks must be within the global DP group")
    if args.dp_size_local * args.tp_size > 16:
        parser.error("This A3 configuration exposes 16 devices per node")
    processes = []
    for local_rank in range(args.dp_size_local):
        process = multiprocessing.Process(target=run_command, args=(args, local_rank))
        processes.append(process)
        process.start()
    for process in processes:
        process.join()
    return 1 if any(process.exitcode != 0 for process in processes) else 0


if __name__ == "__main__":
    sys.exit(main())

prepare the script run_dp_template.sh on each node.

Prefill node 0

#!/usr/bin/env bash

LOCAL_IP="<PREFILL_NODE_IP>"
NIC_NAME="<NETWORK_INTERFACE>"
MODEL_PATH="<YOUR_MODEL_PATH>"

export HCCL_OP_EXPANSION_MODE="AIV"
export VLLM_ENGINE_READY_TIMEOUT_S=100000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=204
export HCCL_CONNECT_TIMEOUT=7200
export ASCEND_CONNECT_TIMEOUT=10000
export ASCEND_TRANSFER_TIMEOUT=10000
export MF_GROUP_JOIN_MAX_TIMEOUT=1200

export VLLM_HOST_IP="$LOCAL_IP"
export HCCL_IF_IP="$LOCAL_IP"
export GLOO_SOCKET_IFNAME="$NIC_NAME"
export TP_SOCKET_IFNAME="$NIC_NAME"
export HCCL_SOCKET_IFNAME="$NIC_NAME"

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=400

export ACL_OP_INIT_MODE=1
export ASCEND_A3_ENABLE=1
export ASCEND_RT_VISIBLE_DEVICES="$1"

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

export VLLM_USE_V2_MODEL_RUNNER=0

exec vllm serve "$MODEL_PATH" \
    --host 0.0.0.0 \
    --port "$2" \
    --data-parallel-size "$3" \
    --data-parallel-rank "$4" \
    --data-parallel-address "$5" \
    --data-parallel-rpc-port "$6" \
    --tensor-parallel-size "$7" \
    --enable-expert-parallel \
    --prefill-context-parallel-size 1 \
    --decode-context-parallel-size 16 \
    --cp-kv-cache-interleave-size 128 \
    --speculative-config '{"num_speculative_tokens": 1, "method":"deepseek_mtp","enforce_eager":true}' \
    --seed 1024 \
    --served-model-name glm-5 \
    --max-model-len 1048576 \
    --safetensors-load-strategy 'prefetch' \
    --additional-config '{
        "enable_dsa_cp": true,
        "enable_fused_mc2": 1,
        "enable_flashcomm1": true
    }' \
    --max-num-batched-tokens 8192 \
    --trust-remote-code \
    --enable-prefix-caching \
    --max-num-seqs 64 \
    --quantization ascend \
    --gpu-memory-utilization 0.85 \
    --enforce-eager \
    --enable-auto-tool-choice \
    --tool-call-parser glm47 \
    --reasoning-parser glm47 \
    --kv-cache-dtype int8 \
    --attention_config.indexer_kv_dtype int8 \
    --kv-transfer-config '{
        "kv_connector": "MooncakeConnectorV1",
        "kv_role": "kv_producer",
        "kv_port": "30000",
        "engine_id": "0",
        "kv_connector_extra_config": {
            "use_ascend_direct": true,
            "prefill": {
                "dp_size": 2,
                "tp_size": 16
            },
            "decode": {
                "dp_size": 4,
                "tp_size": 8
            }
        }
    }'

Prefill node 1

#!/usr/bin/env bash

LOCAL_IP="<PREFILL_NODE_IP>"
NIC_NAME="<NETWORK_INTERFACE>"
MODEL_PATH="<YOUR_MODEL_PATH>"

export HCCL_OP_EXPANSION_MODE="AIV"
export VLLM_ENGINE_READY_TIMEOUT_S=100000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=204
export HCCL_CONNECT_TIMEOUT=7200
export ASCEND_CONNECT_TIMEOUT=10000
export ASCEND_TRANSFER_TIMEOUT=10000
export MF_GROUP_JOIN_MAX_TIMEOUT=1200

export VLLM_HOST_IP="$LOCAL_IP"
export HCCL_IF_IP="$LOCAL_IP"
export GLOO_SOCKET_IFNAME="$NIC_NAME"
export TP_SOCKET_IFNAME="$NIC_NAME"
export HCCL_SOCKET_IFNAME="$NIC_NAME"

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=400

export ACL_OP_INIT_MODE=1
export ASCEND_A3_ENABLE=1
export ASCEND_RT_VISIBLE_DEVICES="$1"

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

export VLLM_USE_V2_MODEL_RUNNER=0

exec vllm serve "$MODEL_PATH" \
    --host 0.0.0.0 \
    --port "$2" \
    --data-parallel-size "$3" \
    --data-parallel-rank "$4" \
    --data-parallel-address "$5" \
    --data-parallel-rpc-port "$6" \
    --tensor-parallel-size "$7" \
    --enable-expert-parallel \
    --prefill-context-parallel-size 1 \
    --decode-context-parallel-size 16 \
    --cp-kv-cache-interleave-size 128 \
    --speculative-config '{"num_speculative_tokens": 1, "method":"deepseek_mtp","enforce_eager":true}' \
    --seed 1024 \
    --served-model-name glm-5 \
    --max-model-len 1048576 \
    --safetensors-load-strategy 'prefetch' \
    --additional-config '{
        "enable_dsa_cp": true,
        "enable_fused_mc2": 1,
        "enable_flashcomm1": true
    }' \
    --max-num-batched-tokens 8192 \
    --trust-remote-code \
    --enable-prefix-caching \
    --max-num-seqs 64 \
    --quantization ascend \
    --gpu-memory-utilization 0.85 \
    --enforce-eager \
    --enable-auto-tool-choice \
    --tool-call-parser glm47 \
    --reasoning-parser glm47 \
    --kv-cache-dtype int8 \
    --attention_config.indexer_kv_dtype int8 \
    --kv-transfer-config '{
        "kv_connector": "MooncakeConnectorV1",
        "kv_role": "kv_producer",
        "kv_port": "30000",
        "engine_id": "0",
        "kv_connector_extra_config": {
            "use_ascend_direct": true,
            "prefill": {
                "dp_size": 2,
                "tp_size": 16
            },
            "decode": {
                "dp_size": 4,
                "tp_size": 8
            }
        }
    }'

Decode node 0

#!/usr/bin/env bash

LOCAL_IP="<DECODE_NODE_IP>"
NIC_NAME="<NETWORK_INTERFACE>"
MODEL_PATH="<YOUR_MODEL_PATH>"

export HCCL_OP_EXPANSION_MODE="AIV"
export VLLM_ENGINE_READY_TIMEOUT_S=100000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=204
export HCCL_CONNECT_TIMEOUT=7200
export ASCEND_CONNECT_TIMEOUT=10000
export ASCEND_TRANSFER_TIMEOUT=10000
export MF_GROUP_JOIN_MAX_TIMEOUT=1200

export VLLM_HOST_IP="$LOCAL_IP"
export HCCL_IF_IP="$LOCAL_IP"
export GLOO_SOCKET_IFNAME="$NIC_NAME"
export TP_SOCKET_IFNAME="$NIC_NAME"
export HCCL_SOCKET_IFNAME="$NIC_NAME"

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=400

export ACL_OP_INIT_MODE=1
export ASCEND_A3_ENABLE=1
export TASK_QUEUE_ENABLE=1
export ASCEND_RT_VISIBLE_DEVICES="$1"

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export VLLM_USE_V2_MODEL_RUNNER=0

exec vllm serve "$MODEL_PATH" \
    --host 0.0.0.0 \
    --port "$2" \
    --data-parallel-size "$3" \
    --data-parallel-rank "$4" \
    --data-parallel-address "$5" \
    --data-parallel-rpc-port "$6" \
    --tensor-parallel-size "$7" \
    --enable-expert-parallel \
    --prefill-context-parallel-size 1 \
    --decode-context-parallel-size 8 \
    --cp-kv-cache-interleave-size 128 \
    --seed 1024 \
    --served-model-name glm-5 \
    --max-model-len 1048576 \
    --safetensors-load-strategy 'prefetch' \
    --max-num-batched-tokens 192 \
    --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
    --speculative-config '{"num_speculative_tokens": 5,  "method":"deepseek_mtp","enforce_eager":true}' \
    --additional-config '{
        "recompute_scheduler_enable": true,
        "enable_mlapo": true,
        "enable_fused_mc2": 1
    }' \
    --trust-remote-code \
    --max-num-seqs 32 \
    --gpu-memory-utilization 0.90 \
    --async-scheduling \
    --quantization ascend \
    --enable-auto-tool-choice \
    --tool-call-parser glm47 \
    --reasoning-parser glm47 \
    --kv-cache-dtype int8 \
    --attention_config.indexer_kv_dtype int8 \
    --kv-transfer-config '{
        "kv_connector": "MooncakeConnectorV1",
        "kv_role": "kv_consumer",
        "kv_port": "30200",
        "engine_id": "2",
        "kv_connector_extra_config": {
            "use_ascend_direct": true,
            "prefill": {
                "dp_size": 2,
                "tp_size": 16
            },
            "decode": {
                "dp_size": 4,
                "tp_size": 8
            }
        }
    }'

Decode node 1

#!/usr/bin/env bash

LOCAL_IP="<DECODE_NODE_IP>"
NIC_NAME="<NETWORK_INTERFACE>"
MODEL_PATH="<YOUR_MODEL_PATH>"

export HCCL_OP_EXPANSION_MODE="AIV"
export VLLM_ENGINE_READY_TIMEOUT_S=100000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
export HCCL_EXEC_TIMEOUT=204
export HCCL_CONNECT_TIMEOUT=7200
export ASCEND_CONNECT_TIMEOUT=10000
export ASCEND_TRANSFER_TIMEOUT=10000
export MF_GROUP_JOIN_MAX_TIMEOUT=1200

export VLLM_HOST_IP="$LOCAL_IP"
export HCCL_IF_IP="$LOCAL_IP"
export GLOO_SOCKET_IFNAME="$NIC_NAME"
export TP_SOCKET_IFNAME="$NIC_NAME"
export HCCL_SOCKET_IFNAME="$NIC_NAME"

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=400

export ACL_OP_INIT_MODE=1
export ASCEND_A3_ENABLE=1
export TASK_QUEUE_ENABLE=1
export ASCEND_RT_VISIBLE_DEVICES="$1"

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export VLLM_USE_V2_MODEL_RUNNER=0

exec vllm serve "$MODEL_PATH" \
    --host 0.0.0.0 \
    --port "$2" \
    --data-parallel-size "$3" \
    --data-parallel-rank "$4" \
    --data-parallel-address "$5" \
    --data-parallel-rpc-port "$6" \
    --tensor-parallel-size "$7" \
    --enable-expert-parallel \
    --prefill-context-parallel-size 1 \
    --decode-context-parallel-size 8 \
    --cp-kv-cache-interleave-size 128 \
    --seed 1024 \
    --served-model-name glm-5 \
    --max-model-len 1048576 \
    --safetensors-load-strategy 'prefetch' \
    --max-num-batched-tokens 192 \
    --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
    --speculative-config '{"num_speculative_tokens": 5,  "method":"deepseek_mtp","enforce_eager":true}' \
    --additional-config '{
        "recompute_scheduler_enable": true,
        "enable_mlapo": true,
        "enable_fused_mc2": 1
    }' \
    --trust-remote-code \
    --max-num-seqs 32 \
    --gpu-memory-utilization 0.90 \
    --async-scheduling \
    --quantization ascend \
    --enable-auto-tool-choice \
    --tool-call-parser glm47 \
    --reasoning-parser glm47 \
    --kv-cache-dtype int8 \
    --attention_config.indexer_kv_dtype int8 \
    --kv-transfer-config '{
        "kv_connector": "MooncakeConnectorV1",
        "kv_role": "kv_consumer",
        "kv_port": "30200",
        "engine_id": "2",
        "kv_connector_extra_config": {
            "use_ascend_direct": true,
            "prefill": {
                "dp_size": 2,
                "tp_size": 16
            },
            "decode": {
                "dp_size": 4,
                "tp_size": 8
            }
        }
    }'

5.2.1.3 Start the Engines¶

Run these commands from the directory containing both scripts on each node. The two Prefill commands use the same P0 master address, and the two Decode commands use the same D0 master address.

Prefill node 0

P0_IP="<PREFILL_NODE0_IP>"
python launch_online_dp.py --dp-size 2 --tp-size 16 --dp-size-local 1 --dp-rank-start 0 --dp-address "$P0_IP" --dp-rpc-port 16591 --vllm-start-port 8000

Prefill node 1

P0_IP="<PREFILL_NODE0_IP>"
python launch_online_dp.py --dp-size 2 --tp-size 16 --dp-size-local 1 --dp-rank-start 1 --dp-address "$P0_IP" --dp-rpc-port 16591 --vllm-start-port 8000

Decode node 0

D0_IP="<DECODE_NODE0_IP>"
python launch_online_dp.py --dp-size 4 --tp-size 8 --dp-size-local 2 --dp-rank-start 0 --dp-address "$D0_IP" --dp-rpc-port 16600 --vllm-start-port 8000

Decode node 1

D0_IP="<DECODE_NODE0_IP>"
python launch_online_dp.py --dp-size 4 --tp-size 8 --dp-size-local 2 --dp-rank-start 2 --dp-address "$D0_IP" --dp-rpc-port 16600 --vllm-start-port 8000

5.2.1.4 Start the Proxy¶

After all six APIs become healthy, start the proxy on its separate host:

P0_IP="<P0_IP>"
P1_IP="<P1_IP>"
D0_IP="<D0_IP>"
D1_IP="<D1_IP>"

python examples/disaggregated_prefill_v1/load_balance_proxy_server_example.py \
    --host 0.0.0.0 --port 8000 \
    --prefiller-hosts "$P0_IP" "$P1_IP" --prefiller-ports 8000 8000 \
    --decoder-hosts "$D0_IP" "$D0_IP" "$D1_IP" "$D1_IP" \
    --decoder-ports 8000 8001 8000 8001

Send requests to http://<PROXY_IP>:8000 using the served model name glm-5.

关键参数说明：

The launcher passes $1 = visible devices, $2 = API port, $3 = global DP size, $4 = DP rank, $5 = DP master IP, $6 = DP RPC port, and $7 = TP size to run_dp_template.sh.
--decode-context-parallel-size: 16 on Prefill and 8 on Decode; --prefill-context-parallel-size remains 1 and --cp-kv-cache-interleave-size is 128 on both roles.
--max-num-batched-tokens: 8192 on Prefill and 192 on Decode. --max-num-seqs: 64 on Prefill and 32 on Decode. These are scheduler limits; start the 1M workload with one concurrent request and increase concurrency according to available KV cache capacity.
MTP uses deepseek_mtp with an eager draft: 1 speculative token on Prefill and 5 on Decode. Decode uses FULL_DECODE_ONLY graph mode.
MooncakeConnectorV1 uses kv_producer on Prefill and kv_consumer on Decode, with the same prefill: DP2/TP16 and decode: DP4/TP8 topology on both roles. This example does not enable MemCache pooling.
Both roles use int8 KV and indexer KV caches. Prefill enables DSA CP, fused MC2 and FlashComm1; Decode enables fused MC2, MLAPO and the recomputation scheduler. See Performance Tuning.

6 功能验证¶

服务器启动后，您可以使用输入提示词查询模型：

curl http://<node0_ip>:<port>/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "glm-5",
        "messages":[
            {
                "role": "user",
                "content": "Who are you?"
            }
        ],
        "temperature": 0
    }'

预期结果应如下所示：

"message":{"role":"assistant","content":"I'm GLM, a large language model developed by Z.ai. I'm designed to understand and generate human-like text based on the conversations we have together. My role involves processing diverse text data to help answer questions and provide assistance across many topics.\n\nI don't store your personal data, and I'm continually learning to improve my capabilities. Is there something specific I can help you with today?","refusal":null,"annotations":null,"audio":null,"function_call":oning":"Let me analyze this question about my identity. First, I should acknowledge that this is a fundamental question about who and what I am. The key aspects are my identity as GLM, a large language model by Z.ai, and my core capabilities. I should explain my primary function of text processing and be transparent about my nature as an AI system. It's also important to clarify my role in helping users and my ability to engage with various topics. Mention my text processing abilities and learning from diverse datasets, but avoid making claims about consciousness or emotions. The response should be structured logically, starting with my basic identity and moving on to my capabilities and purpose. I'll organize this information in a clear, straightforward manner that addresses the user's query directly."}

7 精度评估¶

以下是两种精度评估方法。

7.1 使用AISBench¶

有关详细信息，请参阅使用AISBench。
The following table shows the results of GLM-5.3-w8a8c8 with Prefill-Decode disaggregation.

dataset	model	hardware	deployment	metric	mode	vllm-api-general-chat
GPQA Diamond	GLM-5.3-w8a8c8	A3	PD disaggregation	accuracy	gen	91.92
GPQA Diamond	GLM-5.3-w8a8c8	A2	PD disaggregation	accuracy	gen	89.39

7.2 使用Language Model Evaluation Harness¶

尚未测试。

8 性能评估¶

8.1 使用AISBench¶

有关详细信息，请参阅使用AISBench进行性能评估。

8.2 使用vLLM Benchmark¶

有关更多详细信息，请参阅vllm benchmark。

9 Performance Tuning¶

The following A3 W8A8C8 settings summarize the deployment scripts in Sections 5.1.2, 5.1.3 and 5.2.1. Global DP includes all nodes of one role; local DP is the number of ranks on one node. Each role uses 32 logical NPU devices.

9.1 Detailed Node Configuration¶

Scenario	Role	TP	Global DP	Local DP	PP	DCP	Max Num Seqs	Max Num Batched Tokens	Max Model Len	MTP Spec Num
198K PD, with or without pooling	Prefill	16	1	1	2	1	64	16384	200000	1
198K PD, with or without pooling	Decode	2	16	8	1	1	64	256	200000	5
1M PD (A3)	Prefill	16	2	1	1	16	64	8192	1048576	1
1M PD (A3)	Decode	8	4	2	1	8	32	192	1048576	5

9.2 Performance Workloads¶

The 0.30 RC performance reference covers the following workloads. Lengths use input/output notation. The 1M A3 case uses one concurrent request; reserve output tokens within the 1,048,576-token context limit.

Scenario	Input / Output	Prefix cache hit rate	Deployment reference
Prefill-only	16K / 1, 128K / 1	0%	Section 5.1.2
Decode-only	16K / 1K, 128K / 1K	100%	Section 5.1.2
Complete PD	16K / 1K	0%	Section 5.1.2
Complete PD, 1M context (A3)	Approximately 1M / 1K	0%	Section 5.2.1
Pooling	128K / 1K	90%, 32 distinct prefixes	Section 5.1.3
Pooling	40K-80K / 2.5K	Repeated prefixes	Section 5.1.3

9.3 Optimizations Requiring Explicit Enablement¶

Combine enabled options in one --additional-config JSON object, as in the deployment scripts.

Optimization	Scenario	Enablement	Purpose and constraints
FlashComm1	A3 Prefill	`"enable_flashcomm1": true`	Reduces TP communication overhead; the Decode reference keeps it disabled.
Fused MC2	A3 PD	`"enable_fused_mc2": 1`	Fuses MoE dispatch, computation and combine. Do not combine with shared-expert multistream overlap.
DSA CP	A3 Prefill	`"enable_dsa_cp": true`	Distributes long-context attention work; use the layout specified in each scenario.
DCP	1M PD (A3)	`--decode-context-parallel-size 16` on Prefill, `8` on Decode	Shards KV cache across the TP group. Keep PCP at 1 and KV interleave at 128 on both roles.
Int8 attention KV cache	A3 W8A8C8	`--kv-cache-dtype int8`	Reduces attention KV cache memory.
Int8 indexer KV cache	A3 W8A8C8	`--attention_config.indexer_kv_dtype int8`	Quantizes the indexer cache separately from the attention cache.
MLAPO	A3 1M Decode	`"enable_mlapo": true`	Fuses MLA preprocessing; account for its memory requirements.
Recompute scheduler	A3 Decode	`"recompute_scheduler_enable": true`	Enables Prefill-side recomputation when Decode KV cache capacity is insufficient.
Decode graph	A3 Decode	`--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'`	Captures Decode execution; the reference Prefill uses eager mode.
KV pooling	Repeated-prefix PD	Ascend Store/MemCache configuration in Section 5.1.3	Reuses KV across requests and engines; confirm pool hits in backend metrics.

10 常见问题¶

问：如何为GLM-5.3启用函数调用？

答：请在vLLM启动命令中添加以下配置

--tool-call-parser glm47 \
--reasoning-parser glm47 \
--enable-auto-tool-choice \

问：GLM-5.3 是否支持 enable_thinking: false？

答：不支持，GLM-5.3 不支持 enable_thinking。