DeepSeek-V3/3.1¶

1 简介¶

DeepSeek-V3.1 是一个混合模型，同时支持思考模式和非思考模式。与之前的版本相比，此次升级在多个方面带来了改进：

混合思考模式：通过更改对话模板，单个模型同时支持思考模式和非思考模式。
更智能的工具调用：通过后训练优化，模型在工具使用和智能体任务中的性能显著提升。
更高的思考效率：DeepSeek-V3.1-Think 在实现与 DeepSeek-R1-0528 相当的答案质量的同时，响应速度更快。

本文档将展示模型的主要验证步骤，包括支持的特性、特性配置、环境准备、单节点和多节点部署、精度评估和性能评估。

This document is validated and written based on vLLM-Ascend v0.9.1rc3. The current 模式l (DeepSeek-V3.1) is first supported in this 版本.

2 支持的特性¶

请参阅支持的特性矩阵获取模型支持的特性列表。

请参阅特性指南获取特性的配置方法。

3 Prerequisites¶

3.1 模型权重¶

DeepSeek-V3.1（BF16 版本）：下载模型权重。
DeepSeek-V3.1-w8a8-mtp-QuaRot（包含混合 MTP 的量化版本）：下载模型权重。
DeepSeek-V3.1-Terminus-w4a8-mtp-QuaRot（包含混合 MTP 的量化版本）：下载模型权重。
Quantization method：ms模式lslim。您可以使用此方法对模型进行量化。

建议将模型权重下载到多节点共享目录，例如 /root/.cache/。

3.2 Verify Multi-node Communication (Optional)¶

如果要部署多节点环境，需要按照验证多节点通信环境验证多节点通信。

4 安装¶

4.1 Docker Image 安装¶

您可以使用官方 Docker 镜像直接运行 DeepSeek-V3.1。

根据您的机器类型选择镜像，并在节点上启动 Docker 镜像，请参考使用 Docker 安装。

A3 seriesA2 series

Start the docker image on your each node.

export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1-a3
docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --net=host \
    --privileged=true \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci8 \
    --device /dev/davinci9 \
    --device /dev/davinci10 \
    --device /dev/davinci11 \
    --device /dev/davinci12 \
    --device /dev/davinci13 \
    --device /dev/davinci14 \
    --device /dev/davinci15 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /root/.cache:/root/.cache \
    -it $IMAGE bash

Start the docker image on your each node.

export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1
docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --net=host \
    --privileged=true \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /root/.cache:/root/.cache \
    -it $IMAGE bash

After a successful docker run, you can verify the running container service by executing the docker ps command.

4.2 Source Code Installation¶

If you don't want to use the docker image as above, you can also build all from source:

Install vllm-ascend from source, refer to installation.

If you want to deploy multi-node environment, you need to set up environment on each node.

5 Online Service Deployment¶

5.1 Single-Node Online Deployment¶

Single-node deployment completes both Prefill and Decode within the same node. The quantized model DeepSeek-V3.1-w8a8-mtp-QuaRot can be deployed on 1 台 Atlas 800 A3（64G × 16）.

Startup Command:

#!/bin/sh
# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="xxxx"
local_ip="xxxx"

# [Optional] jemalloc
# jemalloc is for better performance, if `libjemalloc.so` is installed on your machine, you can turn it on.
# export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD

# AIV
export HCCL_OP_EXPANSION_MODE="AIV"

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export VLLM_ASCEND_BALANCE_SCHEDULING=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True

vllm serve /weights/DeepSeek-V3.1-w8a8-mtp-QuaRot \
--host 0.0.0.0 \
--port 8015 \
--data-parallel-size 4 \
--tensor-parallel-size 4 \
--quantization ascend \
--seed 1024 \
--served-model-name deepseek_v3 \
--enable-expert-parallel \
--max-num-seqs 16 \
--max-model-len 16384 \
--max-num-batched-tokens 4096 \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.92 \
--speculative-config '{"num_speculative_tokens": 3, "method": "mtp"}' \
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}'

Key Parameter Descriptions:

设置环境变量 VLLM_ASCEND_BALANCE_SCHEDULING=1 可启用均衡调度。这有助于提高 v1 调度器的输出吞吐量并降低 TPOT。但在某些场景下 TTFT 可能会下降。此外，不建议在 PD 分离场景下启用此特性。
对于单节点部署，建议使用 dp4tp4 而不是 dp2tp8。
--max-model-len 指定最大上下文长度，即单个请求的输入 token 和输出 token 之和。对于输入长度 3.5K、输出长度 1.5K 的性能测试，16384 的值已经足够；但对于精度测试，请至少设置为 35000。
--no-enable-prefix-caching 表示禁用前缀缓存。要启用前缀缓存，请移除该选项。
如果使用 w4a8 权重，更多内存将分配给 kvcache，您可以尝试增加系统吞吐量以获得更高的吞吐性能。

Common Issues Tip: If you encounter issues, please refer to the Public FAQ for troubleshooting.

Service Verification:

curl http://<node_ip>:8015/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "deepseek_v3",
        "messages": [{
            "role": "user",
            "content": [
            {
                "type": "text",
                "text": "The future of AI is"
            }]
        }],
        "max_tokens": 1024,
        "temperature": 1.0,
        "top_p": 0.95
    }'

Expected Result:

The service returns HTTP 200 OK with a JSON response containing the choices field. Example output:

{
    "id": "chatcmpl-xxxxxxxxxxxxx",
    "object": "chat.completion",
    "model": "deepseek_v3",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "Of course. The future of AI is not a single..."
            },
            "finish_reason": "length"
        }
    ],
    "usage": {
        "prompt_tokens": 9,
        "total_tokens": 1033,
        "completion_tokens": 1024
    }
}

5.2 Multi-Node Data Parallel 部署¶

DeepSeek-V3.1-w8a8-mtp-QuaRot：至少需要 2 台 Atlas 800 A2（64G × 8）。

分别在两个节点上运行以下脚本。

Node 0Node 1

Startup Command:

#!/bin/sh

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="xxxx"
local_ip="xxxx"

# The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)
node0_ip="xxxx"

# [Optional] jemalloc
# jemalloc is for better performance, if `libjemalloc.so` is installed on your machine, you can turn it on.
# export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export HCCL_BUFFSIZE=200
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export VLLM_ASCEND_BALANCE_SCHEDULING=1
export HCCL_INTRA_PCIE_ENABLE=1
export HCCL_INTRA_ROCE_ENABLE=0

vllm serve /weights/DeepSeek-V3.1-w8a8-mtp-QuaRot \
--host 0.0.0.0 \
--port 8004 \
--data-parallel-size 4 \
--data-parallel-size-local 2 \
--data-parallel-address $node0_ip \
--data-parallel-rpc-port 13389 \
--tensor-parallel-size 4 \
--quantization ascend \
--seed 1024 \
--served-model-name deepseek_v3 \
--enable-expert-parallel \
--max-num-seqs 16 \
--max-model-len 16384 \
--max-num-batched-tokens 4096 \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.92 \
--speculative-config '{"num_speculative_tokens": 3, "method": "mtp"}' \
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}'

Startup Command:

#!/bin/sh

# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="xxx"
local_ip="xxx"

# The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)
node0_ip="xxxx"

# [Optional] jemalloc
# jemalloc is for better performance, if `libjemalloc.so` is installed on your machine, you can turn it on.
# export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export HCCL_BUFFSIZE=200
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export VLLM_ASCEND_BALANCE_SCHEDULING=1
export HCCL_INTRA_PCIE_ENABLE=1
export HCCL_INTRA_ROCE_ENABLE=0

vllm serve /weights/DeepSeek-V3.1-w8a8-mtp-QuaRot \
--host 0.0.0.0 \
--port 8004 \
--headless \
--data-parallel-size 4 \
--data-parallel-size-local 2 \
--data-parallel-start-rank 2 \
--data-parallel-address $node0_ip \
--data-parallel-rpc-port 13389 \
--tensor-parallel-size 4 \
--quantization ascend \
--seed 1024 \
--served-model-name deepseek_v3 \
--enable-expert-parallel \
--max-num-seqs 16 \
--max-model-len 16384 \
--max-num-batched-tokens 4096 \
--trust-remote-code \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.92 \
--speculative-config '{"num_speculative_tokens": 3, "method": "mtp"}' \
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}'

Key Parameter Descriptions:

--data-parallel-size: total number of data parallel ranks across all nodes. In this example, 4 means the model is split across 4 DP ranks total (2 per node).
--data-parallel-size-local: number of data parallel ranks running on the current node. In this example, each node runs 2 DP ranks.
--data-parallel-start-rank: starting rank offset for data parallel ranks on this node. Node 0 starts at rank 0 (default), Node 1 starts at rank 2. This ensures each node's DP ranks occupy distinct positions in the overall rank space.
--data-parallel-address: IP address of the data parallel master node (Node 0). This value must be consistent with local_ip set on Node 0.
--data-parallel-rpc-port: RPC port for data parallel master communication. Must be the same across all nodes.
--headless: indicates that this vLLM instance is not the master service node. Only set on non-master nodes (Node 1). The master node (Node 0) should NOT set this flag.
For single-node deployment, we recommend using dp4 tp4 instead of dp2 tp8.

Common Issues Tip: If you encounter issues, please refer to the Public FAQ for troubleshooting.

Service Verification:

curl http://<node_ip>:8015/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "deepseek_v3",
        "messages": [{
            "role": "user",
            "content": [
            {
                "type": "text",
                "text": "The future of AI is"
            }]
        }],
        "max_tokens": 1024,
        "temperature": 1.0,
        "top_p": 0.95
    }'

Expected Result:

The service returns HTTP 200 OK. The JSON response contains the choices field with the 生成erated text.

5.3 Multi-Node PD Separation 部署¶

我们推荐使用 Mooncake 进行部署：Mooncake。

In the standard single-node deployment 模式, Prefill (prompt processing) and Decode (token 生成eration) tasks run on the same set of NPUs. PD (Prefill-Decode) separation addresses this by running Prefill and Decode on dedicated node groups, each configured independently:

Prefill nodes focus on high-throughput prompt processing, optimized for compute and communication.
Decode nodes focus on low-latency token 生成eration, optimized for memory bandwidth.

This architecture is recommended for production deployments with concurrent multi-user workloads, where stable latency and high throughput are both required.

以 Atlas 800 A3（64G × 16）为例，我们建议部署 2P1D（4 节点）而不是 1P1D（2 节点），因为在 1P1D 情况下没有足够的 NPU 内存来支持高并发。

DeepSeek-V3.1-w8a8-mtp-QuaRot 2P1D Layerwise需要 4 台 Atlas 800 A3（64G × 16）。

要运行 vllm-ascend 的预填充-解码分离服务，您需要在每个节点上部署 Prefill-Decode Disaggregation 脚本和 launch_online_dp.py 脚本，并在预填充主节点上部署 run_dp_template.sh 脚本以转发请求。

[launch_online_dp.pyproxy.sh

Parameter descriptions:

Parameter	Type	Required	Default	Description
](https://github.com/vllm-project/vllm-ascend/blob/main/examples/external_online_dp/launch_online_dp.py)	int	Yes	-	Data parallel size (total number of DP ranks across all nodes).
`--dp-size`	int	No	1	Tensor parallel size within each DP rank.
`--tp-size`	int	No	(same as `--dp-size-local`)	Number of DP ranks on the current node. If not set, defaults to `--dp-size`.
`--dp-size`	int	No	0	Starting rank offset for data parallel ranks on this node.
`--dp-rank-start`	str	Yes	-	IP address of the data parallel master node (node 0).
`--dp-address`	str	No	12345	RPC port for data parallel master communication.
`--dp-rpc-port`	int	No	9000	Starting port for each vLLM engine instance on this node. Each DP rank's engine port = `--vllm-start-port` + local rank index.

vllm_start_port script

Node 0(Prefill)Node 1(Prefill)Node 0(Decode)Node 1(Decode)

``run_dp_template.shlibjemalloc.so`shell

this obtained through ifconfig¶

nic_name is the network interface name corresponding to local_ip of the current node¶

nic_name="xxx" local_ip="141.xx.xx.1"

The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)¶

node0_ip="xxxx"

[Optional] jemalloc¶

jemalloc is for better performance, if ``` is installed on your machine, you can turn it on.¶

export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD¶

export HCCL_IF_IP=$local_ip export GLOO_SOCKET_IFNAME=$nic_name export TP_SOCKET_IFNAME=$nic_name export HCCL_SOCKET_IFNAME=$nic_name

export VLLM_RPC_TIMEOUT=3600000 export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000 export HCCL_EXEC_TIMEOUT=204 export HCCL_CONNECT_TIMEOUT=120

export OMP_PROC_BIND=false export OMP_NUM_THREADS=10 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export HCCL_BUFFSIZE=256 export TASK_QUEUE_ENABLE=1 export HCCL_OP_EXPANSION_MODE="AIV" export VLLM_USE_V1=1 export ASCEND_RT_VISIBLE_DEVICES=$1 export ASCEND_BUFFER_POOL=4:8 export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/mooncake:$LD_LIBRARY_PATH

export VLLM_ASCEND_ENABLE_FLASHCOMM1=1

vllm serve /weights/DeepSeek-V3.1-w8a8-mtp-QuaRot \ --host 0.0.0.0 \ --port $2 \ --data-parallel-size $3 \ --data-parallel-rank $4 \ --data-parallel-address $5 \ --data-parallel-rpc-port $6 \ --tensor-parallel-size $7 \ --enable-expert-parallel \ --seed 1024 \ --served-model-name deepseek_v3 \ --max-model-len 65536 \ --max-num-batched-tokens 16384 \ --max-num-seqs 8 \ --enforce-eager \ --trust-remote-code \ --gpu-memory-utilization 0.9 \ --quantization ascend \ --no-enable-prefix-caching \ --speculative-config '{"num_speculative_tokens": 1, "method": "mtp"}' \ --kv-transfer-config \ '{"kv_connector": "MooncakeConnectorV1", "kv_role": "kv_producer", "kv_port": "30000", "kv_connector_extra_config": { "prefill": { "dp_size": 2, "tp_size": 8 }, "decode": { "dp_size": 32, "tp_size": 1 } } }' ```

libjemalloc.soshell

this obtained through ifconfig¶

nic_name is the network interface name corresponding to local_ip of the current node¶

nic_name="xxx" local_ip="141.xx.xx.2"

The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)¶

node0_ip="xxxx"

[Optional] jemalloc¶

jemalloc is for better performance, if ``` is installed on your machine, you can turn it on.¶

export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD¶

export HCCL_IF_IP=$local_ip export GLOO_SOCKET_IFNAME=$nic_name export TP_SOCKET_IFNAME=$nic_name export HCCL_SOCKET_IFNAME=$nic_name

export VLLM_RPC_TIMEOUT=3600000 export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000 export HCCL_EXEC_TIMEOUT=204 export HCCL_CONNECT_TIMEOUT=120

export OMP_PROC_BIND=false export OMP_NUM_THREADS=10 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export HCCL_BUFFSIZE=256 export TASK_QUEUE_ENABLE=1 export HCCL_OP_EXPANSION_MODE="AIV" export VLLM_USE_V1=1 export ASCEND_RT_VISIBLE_DEVICES=$1 export ASCEND_BUFFER_POOL=4:8 export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/mooncake:$LD_LIBRARY_PATH

export VLLM_ASCEND_ENABLE_FLASHCOMM1=1

vllm serve /weights/DeepSeek-V3.1-w8a8-mtp-QuaRot \ --host 0.0.0.0 \ --port $2 \ --data-parallel-size $3 \ --data-parallel-rank $4 \ --data-parallel-address $5 \ --data-parallel-rpc-port $6 \ --tensor-parallel-size $7 \ --enable-expert-parallel \ --seed 1024 \ --served-model-name deepseek_v3 \ --max-model-len 65536 \ --max-num-batched-tokens 16384 \ --max-num-seqs 8 \ --enforce-eager \ --trust-remote-code \ --gpu-memory-utilization 0.9 \ --quantization ascend \ --no-enable-prefix-caching \ --speculative-config '{"num_speculative_tokens": 1, "method": "mtp"}' \ --kv-transfer-config \ '{"kv_connector": "MooncakeConnectorV1", "kv_role": "kv_producer", "kv_port": "30100", "kv_connector_extra_config": { "prefill": { "dp_size": 2, "tp_size": 8 }, "decode": { "dp_size": 32, "tp_size": 1 } } }' ```

libjemalloc.soshell

this obtained through ifconfig¶

nic_name is the network interface name corresponding to local_ip of the current node¶

nic_name="xxx" local_ip="141.xx.xx.3"

The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)¶

node0_ip="xxxx"

[Optional] jemalloc¶

jemalloc is for better performance, if ``` is installed on your machine, you can turn it on.¶

export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD¶

export HCCL_IF_IP=$local_ip export GLOO_SOCKET_IFNAME=$nic_name export TP_SOCKET_IFNAME=$nic_name export HCCL_SOCKET_IFNAME=$nic_name

export VLLM_RPC_TIMEOUT=3600000 export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000 export HCCL_EXEC_TIMEOUT=204 export HCCL_CONNECT_TIMEOUT=120

export OMP_PROC_BIND=false export OMP_NUM_THREADS=10 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export HCCL_BUFFSIZE=1100 export TASK_QUEUE_ENABLE=1 export HCCL_OP_EXPANSION_MODE="AIV" export VLLM_USE_V1=1 export ASCEND_RT_VISIBLE_DEVICES=$1 export ASCEND_BUFFER_POOL=4:8 export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/mooncake:$LD_LIBRARY_PATH

vllm serve /weights/DeepSeek-V3.1-w8a8-mtp-QuaRot \ --host 0.0.0.0 \ --port $2 \ --data-parallel-size $3 \ --data-parallel-rank $4 \ --data-parallel-address $5 \ --data-parallel-rpc-port $6 \ --tensor-parallel-size $7 \ --enable-expert-parallel \ --seed 1024 \ --served-model-name deepseek_v3 \ --max-model-len 65536 \ --max-num-batched-tokens 256 \ --max-num-seqs 28 \ --trust-remote-code \ --gpu-memory-utilization 0.92 \ --quantization ascend \ --no-enable-prefix-caching \ --speculative-config '{"num_speculative_tokens": 1, "method": "mtp"}' \ --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \ --additional-config '{"recompute_scheduler_enable":true,"multistream_overlap_shared_expert": true,"finegrained_tp_config": {"lmhead_tensor_parallel_size":16}}' \ --kv-transfer-config \ '{"kv_connector": "MooncakeConnectorV1", "kv_role": "kv_consumer", "kv_port": "30200", "kv_connector_extra_config": { "prefill": { "dp_size": 2, "tp_size": 8 }, "decode": { "dp_size": 32, "tp_size": 1 } } }' ```

libjemalloc.soshell

this obtained through ifconfig¶

nic_name is the network interface name corresponding to local_ip of the current node¶

nic_name="xxx" local_ip="141.xx.xx.4"

The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)¶

node0_ip="xxxx"

[Optional] jemalloc¶

jemalloc is for better performance, if ``` is installed on your machine, you can turn it on.¶

export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD¶

export HCCL_IF_IP=$local_ip export GLOO_SOCKET_IFNAME=$nic_name export TP_SOCKET_IFNAME=$nic_name export HCCL_SOCKET_IFNAME=$nic_name

export VLLM_RPC_TIMEOUT=3600000 export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000 export HCCL_EXEC_TIMEOUT=204 export HCCL_CONNECT_TIMEOUT=120

export OMP_PROC_BIND=false export OMP_NUM_THREADS=10 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export HCCL_BUFFSIZE=1100 export TASK_QUEUE_ENABLE=1 export HCCL_OP_EXPANSION_MODE="AIV" export VLLM_USE_V1=1 export ASCEND_RT_VISIBLE_DEVICES=$1 export ASCEND_BUFFER_POOL=4:8 export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/mooncake:$LD_LIBRARY_PATH

vllm serve /weights/DeepSeek-V3.1-w8a8-mtp-QuaRot \ --host 0.0.0.0 \ --port $2 \ --data-parallel-size $3 \ --data-parallel-rank $4 \ --data-parallel-address $5 \ --data-parallel-rpc-port $6 \ --tensor-parallel-size $7 \ --enable-expert-parallel \ --seed 1024 \ --served-model-name deepseek_v3 \ --max-model-len 65536 \ --max-num-batched-tokens 256 \ --max-num-seqs 28 \ --trust-remote-code \ --gpu-memory-utilization 0.92 \ --quantization ascend \ --no-enable-prefix-caching \ --speculative-config '{"num_speculative_tokens": 1, "method": "mtp"}' \ --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \ --additional-config '{"recompute_scheduler_enable":true,"multistream_overlap_shared_expert": true,"finegrained_tp_config": {"lmhead_tensor_parallel_size":16}}' \ --kv-transfer-config \ '{"kv_connector": "MooncakeConnectorV1", "kv_role": "kv_consumer", "kv_port": "30200", "kv_connector_extra_config": { "prefill": { "dp_size": 2, "tp_size": 8 }, "decode": { "dp_size": 32, "tp_size": 1 } } }' VLLM_ASCEND_ENABLE_FLASHCOMM1=1

Key Parameter Descriptions:

VLLM_ASCEND_ENABLE_MLAPO=1: enables the communication optimization function on the prefill nodes.
recompute_scheduler_enable: true: enables the fusion operator, which can significantly improve performance but consumes more NPU memory. In the Prefill-Decode (PD) separation scenario, enable MLAPO only on decode nodes.
multistream_overlap_shared_expert: true: enables the recomputation scheduler. When the Key-Value Cache (KV Cache) of the decode node is insufficient, requests will be sent to the prefill node to recompute the KV Cache. In the PD separation scenario, enable this configuration only on decode nodes.
enable_shared_expert_dp: true: When the Tensor Parallelism (TP) size is 1 or lmhead_tensor_parallel_size: 16, an additional stream is enabled to overlap the computation process of shared experts for improved efficiency.
```: When the Tensor Parallelism (TP) size of the decode node is 1, this parameter allows the TP size of the LMHead embedding layer to be greater than 1, which is used to reduce the computational load of each card on the LMHead embedding layer.
run server for each node:

```shell

p0¶

python launch_online_dp.py --dp-size 2 --tp-size 8 --dp-size-local 2 --dp-rank-start 0 --dp-address 141.xx.xx.1 --dp-rpc-port 12321 --vllm-start-port 7100

p1¶

python launch_online_dp.py --dp-size 2 --tp-size 8 --dp-size-local 2 --dp-rank-start 0 --dp-address 141.xx.xx.2 --dp-rpc-port 12321 --vllm-start-port 7100

d0¶

python launch_online_dp.py --dp-size 32 --tp-size 1 --dp-size-local 16 --dp-rank-start 0 --dp-address 141.xx.xx.3 --dp-rpc-port 12321 --vllm-start-port 7100

d1¶

python launch_online_dp.py --dp-size 32 --tp-size 1 --dp-size-local 16 --dp-rank-start 16 --dp-address 141.xx.xx.3 --dp-rpc-port 12321 --vllm-start-port 7100 proxy.sh

Run the ``` script on the prefill master node

Run a proxy server on the same node with the prefiller service instance. You can get the proxy program in the repository's examples: load_balance_proxy_server_example.py

python load_balance_proxy_server_example.py \
  --port 1999 \
  --host 141.xx.xx.1 \
  --prefiller-hosts \
    141.xx.xx.1 \
    141.xx.xx.1 \
    141.xx.xx.2 \
    141.xx.xx.2 \
  --prefiller-ports \
    7100 7101 7100 7101 \
  --decoder-hosts \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.3 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
    141.xx.xx.4 \
  --decoder-ports \
    7100 7101 7102 7103 7104 7105 7106 7107 7108 7109 7110 7111 7112 7113 7114 7115 \
    7100 7101 7102 7103 7104 7105 7106 7107 7108 7109 7110 7111 7112 7113 7114 7115 \

```

部署 Verification:

After the PD separation service is fully started, send a request through the proxy port on the prefill master node to verify that Prefill and Decode nodes are working correctly together:

curl http://141.xx.xx.1:1999/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "deepseek_v3",
        "messages": [{
            "role": "user",
            "content": [
            {
                "type": "text",
                "text": "The future of AI is"
            }]
        }],
        "max_tokens": 1024,
        "temperature": 1.0,
        "top_p": 0.95
    }'

Expected Result:

The proxy returns HTTP 200 OK. The JSON response contains the choices field with the 生成erated text, confirming that Prefill nodes have successfully processed the prompt and Decode nodes have 生成erated the response:

{
    "id": "chatcmpl-xxxxxxxxxxxxx",
    "object": "chat.completion",
    "model": "deepseek_v3",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "The future of AI is not a destination we are passively approaching...",
                "finish_reason": "length"
            }
        }
    ],
    "usage": {
        "prompt_tokens": 13,
        "total_tokens": 1037,
        "completion_tokens": 1024
    }
}

Common Issues Tip: If you encounter issues with PD separation deployment, please refer to the Public FAQ for troubleshooting.

6 功能验证¶

服务器启动后，您可以使用输入提示词查询模型：

curl http://<node0_ip>:<port>/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "deepseek_v3",
        "prompt": "The future of AI is",
        "max_completion_tokens": 50,
        "temperature": 0
    }'

7 精度评估¶

Here is one 准确率 evaluation method.

使用 AISBench¶

详细信息请参考使用 AISBench 进行精度评估。
After execution, you can get the result, here is the result of DeepSeek-V3.1-w8a8-mtp-QuaRot in vllm-ascend:0.18.0 for reference only.

数据集	版本	指标	模式	vllm-api-生成eral-chat	备注
ceval	-	准确率	生成	90.94	1 台 Atlas 800 A3（64G × 16）
gsm8k	-	准确率	生成	96.28	1 台 Atlas 800 A3（64G × 16）

使用 Language Model Evaluation Harness¶

尚未测试。

8 性能 Evaluation¶

使用 AISBench¶

详细信息请参考使用 AISBench 进行性能评估。

性能结果如下：

硬件：A3-752T，4 节点

部署：2P1D，预填充节点：DP2+TP8，解码节点：DP32+TP1

输入/输出：3.5k/1.5k

性能：TTFT = 6.16 秒，TPOT = 48.82 毫秒，每卡平均性能为 478 TPS（每秒处理 Token 数）。

使用 vLLM 基准测试¶

以 DeepSeek-V3.1-w8a8-mtp-QuaRot 为例运行性能评估。

更多详细信息请参考 vLLM 基准测试。

有三个 vllm bench 子命令：

latency：基准测试单批次请求的延迟。
serve：基准测试在线服务的吞吐量。
throughput：基准测试离线推理的吞吐量。

以 serve 为例，运行如下代码。

vllm bench serve --model /weights/DeepSeek-V3.1-w8a8-mtp-QuaRot  --dataset-name random --random-input 1024 --num-prompts 200 --request-rate 1 --save-result --result-dir ./

大约几分钟后，即可获得性能评估结果。

9 性能 Tuning¶

9.1 Recommended Configurations¶

Note: The following configurations are validated in specific test environments and are for reference only. The optimal configuration depends on factors such as maximum input/output length, prefix cache hit rate, precision requirements, and deployment machine ratios. It is recommended to refer to Section 9.2 for tuning based on actual conditions.

Table 1: Scenario Overview¶

*Total NPUs indicates the total number of NPUs used across all nodes. 1 node = 1 Atlas 800 A3 server (64G × 16 NPUs).

Scenario	部署 Mode	*Total NPUs	Weight Version	Key Considerations
High Throughput (3.5K/16K input)	Single-Node Mixed	16 (A3)	DeepSeek-V3.1-w4a8-perchannle	Use dp4 tp4 to balance memory capacity and compute efficiency
Low Latency (3.5K/16K input)	Single-Node Mixed	16 (A3)	DeepSeek-V3.1-w4a8-perchannle	Use dp2 tp8 to balance memory capacity and compute efficiency
High Throughput / Low Latency (64K input)	Single-Node Mixed	16 (A3)	DeepSeek-V3.1-w4a8-perchannle	Use dp2 tp8 to balance memory capacity and compute efficiency
High Throughput / Low Latency (3.5K input)	2P1D deployment	64 (A3)	DeepSeek-V3.1-w4a8-perchannle	Use dp2 tp8 to balance memory capacity and compute efficiency
High Throughput / Low Latency (16K input)	2P1D deployment	64 (A3)	DeepSeek-V3.1-w4a8-perchannle	Use dp2 tp8 to balance memory capacity and compute efficiency
Long Context (64K input, no prefix cache)	2P1D deployment	64 (A3)	DeepSeek-V3.1-w4a8-perchannle	Use dp1 tp8 to balance memory capacity and compute efficiency

Table 2: Detailed Node Configuration¶

Scenario	Configuration	NPUs	TP	DP	Max Model Len	MTP Speculation Num
High Throughput (3.5K)	Server / Single Machine	16	4	4	39K	3
High Throughput (16K)	Server / Single Machine	16	4	4	36K	3
Low Latency (3.5K)	Server / Single Machine	16	8	2	36K	3
Low Latency (16K)	Server / Single Machine	16	8	2	36K	3
High Throughput / Low Latency (64K)	Server / Single Machine	16	8	2	132K	3

|High Throughput (16K)|Server-P Node|16|8|2|36K|1| |High Throughput (16K)|Server-D Node|16|4|8|36K|1| |Low Latency (16K)|Server-P Node|16|8|2|36K|3| |Low Latency (16K)|Server-D Node|16|4|8|36K|3| |Long Context (64K)|Server-P Node|16|8|1(PCP2DCP8)|36K|3| |Long Context (64K)|Server-D Node|16|4|8|36K|3|

For complete startup commands and parameter descriptions, please refer to the deployment examples in Chapter 5.

Notice: max-model-len and max-num-seqs need to be set according to the actual usage scenario. For other settings, please refer to the 部署 chapter.

9.2 Tuning Guidelines¶

Please refer to the Public 性能 Tuning Documentation for tuning methods.

Please refer to the Feature Guide for detailed feature descriptions.

10 FAQ¶

For common environment, installation, and 生成eral parameter issues, please refer to the Public FAQ.

DeepSeek-V3/3.1¶

1 简介¶

2 支持的特性¶

3 Prerequisites¶

3.1 模型权重¶

3.2 Verify Multi-node Communication (Optional)¶

4 安装¶

4.1 Docker Image 安装¶

4.2 Source Code Installation¶

5 Online Service Deployment¶

5.1 Single-Node Online Deployment¶

5.2 Multi-Node Data Parallel 部署¶

5.3 Multi-Node PD Separation 部署¶

this obtained through ifconfig¶

nic_name is the network interface name corresponding to local_ip of the current node¶

The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)¶

[Optional] jemalloc¶

jemalloc is for better performance, if ``` is installed on your machine, you can turn it on.¶

export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD¶

this obtained through ifconfig¶

nic_name is the network interface name corresponding to local_ip of the current node¶

The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)¶

[Optional] jemalloc¶

jemalloc is for better performance, if ``` is installed on your machine, you can turn it on.¶

export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD¶

this obtained through ifconfig¶

nic_name is the network interface name corresponding to local_ip of the current node¶

The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)¶

[Optional] jemalloc¶

jemalloc is for better performance, if ``` is installed on your machine, you can turn it on.¶

export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD¶

this obtained through ifconfig¶

nic_name is the network interface name corresponding to local_ip of the current node¶

The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)¶

[Optional] jemalloc¶

jemalloc is for better performance, if ``` is installed on your machine, you can turn it on.¶

export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD¶

p0¶

p1¶

d0¶

d1¶

6 功能验证¶

7 精度评估¶

使用 AISBench¶

使用 Language Model Evaluation Harness¶

8 性能 Evaluation¶

使用 AISBench¶

使用 vLLM 基准测试¶

9 性能 Tuning¶

9.1 Recommended Configurations¶

Table 1: Scenario Overview¶

Table 2: Detailed Node Configuration¶

9.2 Tuning Guidelines¶

10 FAQ¶