DeepSeek-V4-Flash

1.简介

DeepSeek-V4在DeepSeek-V3基础上引入了多项关键升级：

• 流形约束超连接(mHC)增强传统残差连接。
• 混合注意力架构，通过Compress-4-Attention和Compress-128-Attention大幅提升长上下文效率。对于混合专家(MoE)组件，仍采用DeepSeekMoE架构，仅做少量调整。

DeepSeek-V4-Flash是DeepSeek-V4系列的轻量级变体，适用于高吞吐量和低延迟服务场景。

本文档将展示模型的主要验证步骤，包括支持特性、特性配置、环境准备、单节点和多节点部署、精度及性能评估。

注意：请将上述版本占位符替换为您实际的验证版本。

2.支持特性

参考支持特性获取模型的支持特性矩阵。

参考特性指南获取特性的配置。

3.前提条件

3.1.模型权重

• DeepSeek-V4-Flash-w8a8-mtp(量化版本)：需要1个Atlas 800 A3 (128G × 8)节点或1个Atlas 800 A2 (64G × 8)节点。下载模型权重

建议将模型权重下载到多节点共享目录，例如/root/.cache/。

3.2.验证多节点通信(可选)

如需部署多节点环境，需按照验证多节点通信环境验证多节点通信。

4.安装

4.1.Docker镜像安装

根据机器类型选择镜像并在节点上启动docker镜像，参考使用docker。

A3系列A2 series

在每个节点上启动docker镜像。

export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1-a3
docker run --rm \
    --name vllm-ascend \
    --shm-size=512g \
    --net=host \
    --privileged=true \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci8 \
    --device /dev/davinci9 \
    --device /dev/davinci10 \
    --device /dev/davinci11 \
    --device /dev/davinci12 \
    --device /dev/davinci13 \
    --device /dev/davinci14 \
    --device /dev/davinci15 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /etc/hccn.conf:/etc/hccn.conf \
    -v /root/.cache:/root/.cache \
    -it $IMAGE bash

Start the docker image on each node.

export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1
docker run --rm \
    --name vllm-ascend \
    --shm-size=512g \
    --net=host \
    --privileged=true \
    --device /dev/davinci0 \
    --device /dev/davinci1 \
    --device /dev/davinci2 \
    --device /dev/davinci3 \
    --device /dev/davinci4 \
    --device /dev/davinci5 \
    --device /dev/davinci6 \
    --device /dev/davinci7 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /etc/hccn.conf:/etc/hccn.conf \
    -v /root/.cache:/root/.cache \
    -it $IMAGE bash

After a successful docker run, you can verify the running container service by executing the docker ps command.

4.2 Source Code Installation

If you don't want to use the docker image as above, you can also build all from source:

• Install vllm-ascend from source, refer to installation.

If you want to deploy a multi-node environment, you need to set up the environment on each node.

5 Online Service Deployment

Note

In this tutorial, we suppose you downloaded the model weight to /root/.cache/模式lscope/hub/模式ls/vllm-ascend/. Feel free to change it to your own path.

It is recommended that the following service code be encapsulated in a .sh script file and executed in Bash mode.

5.1 Single-Node Online Deployment

Single-node deployment completes both Prefill and Decode within the same node. The quantized model DeepSeek-V4-Flash-w8a8-mtp can be deployed on 1 Atlas 800 A3 (128G × 8) or 1 Atlas 800 A2 (64G × 8).

A2 seriesA3 series

Run the following script to execute online inference.

export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
export HCCL_BUFFSIZE=1024
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
export TASK_QUEUE_ENABLE=1
export HCCL_OP_EXPANSION_MODE="AIV"

vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Flash-w8a8-mtp \
    --max-model-len 133120 \
    --max-num-batched-tokens 8192 \
    --served-model-name dsv4 \
    --gpu-memory-utilization 0.9 \
    --max-num-seqs 32 \
    --data-parallel-size 1 \
    --tensor-parallel-size 8 \
    --enable-expert-parallel \
    --tokenizer-mode deepseek_v4 \
    --tool-call-parser deepseek_v4 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v4 \
    --safetensors-load-strategy 'prefetch' \
    --no-enable-prefix-caching \
    --model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 128}' \
    --quantization ascend \
    --port 8900 \
    --block-size 128 \
    --speculative-config '{"num_speculative_tokens": 1,"method": "mtp","enforce_eager": true}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --async-scheduling \
    --additional-config '
    {"ascend_compilation_config":{
        "enable_npugraph_ex":true,
        "enable_static_kernel":false
        },
    "enable_cpu_binding": true,
    "enable_dsa_cp": true,
    "multistream_overlap_shared_expert":true}'

Run the following script to execute online inference.

export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
export HCCL_BUFFSIZE=1024
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
export TASK_QUEUE_ENABLE=1
export HCCL_OP_EXPANSION_MODE="AIV"

vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Flash-w8a8-mtp \
    --max-model-len 1048576 \
    --max-num-batched-tokens 10240 \
    --served-model-name dsv4 \
    --gpu-memory-utilization 0.9 \
    --api-server-count 1 \
    --max-num-seqs 64 \
    --data-parallel-size 4 \
    --tensor-parallel-size 4 \
    --enable-expert-parallel \
    --tokenizer-mode deepseek_v4 \
    --tool-call-parser deepseek_v4 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v4 \
    --safetensors-load-strategy 'prefetch' \
    --model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 128}' \
    --quantization ascend \
    --port 8900 \
    --block-size 128 \
    --speculative-config '{"num_speculative_tokens": 1,"method": "mtp","enforce_eager": true}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
    --async-scheduling \
    --additional-config '
    {"ascend_compilation_config":{
        "enable_npugraph_ex":true,
        "enable_static_kernel":false
        },
    "enable_cpu_binding": true,
    "multistream_overlap_shared_expert":true}'

Key Parameter Descriptions:

• --max-模式l-len specifies the maximum context length - that is, the sum of input and output tokens for a single request. Adjust it according to your actual scenario.
• --no-enable-prefix-caching indicates that prefix caching is disabled. To enable it, remove this option.
• --speculative-config configures the MTP (Multi-Token Prediction) speculative decoding to accelerate inference.
• --compilation-config '{"cudagraph_模式":"FULL_DECODE_ONLY"}' enables full ACL graph execution in the decode phase to reduce scheduling latency.
• --async-scheduling enables asynchronous scheduling to overlap CPU scheduling with NPU computation.
• VLLM_ASCEND_ENABLE_FLASHCOMM1=1启用FlashComm通信优化。

Common Issues Tip: If you encounter issues, please refer to the Public FAQ for troubleshooting.

服务验证：

curl http://<node0_ip>:8900/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "dsv4",
        "messages": [
            {
                "role": "user",
                "content": "Who are you?"
            }
        ],
        "max_tokens": 256,
        "temperature": 0
    }'

预期结果：

服务返回HTTP 200 OK，JSON响应中包含choices字段。

5.2.多节点PD分离部署

建议使用Mooncake进行部署：Mooncake。

在标准单节点部署模式下，Prefill(提示处理)和Decode(token生成)任务在同一组NPU上运行。这可能导致两个问题：

1. Prefill抢占中断Decode：Prefill是计算密集型任务，一次性处理整个输入上下文，而Decode逐个生成token。当新用户请求到达时，其Prefill阶段可能抢占并中断正在进行的Decode任务，导致抖动和更高的每输出token时间(TPOT)延迟。
2. 资源分配不灵活：Prefill和Decode具有根本不同的计算特性——Prefill是计算受限且内存带宽密集型的，而Decode是内存带宽受限的。在同一硬件上运行它们迫使做出折衷，无法最优满足任何一方。

PD(Prefill-Decode)分离通过在专用节点组上运行Prefill和Decode来解决这些问题，每个节点组独立配置。此架构推荐用于具有并发多用户工作负载的生产部署，其中需要稳定的延迟和高吞吐量。

以下章节描述在Atlas 800 A3 (128G × 8)和Atlas 800 A2 (64G × 8)多节点环境上的PD分离部署。

5.2.1.A3系列PD分离部署

本节展示DeepSeek-V4-Flash在Atlas 800 A3 (128G × 8)多节点环境下的部署指南，采用1P1D配置以获得更优性能。

开始之前，请：

1. 在每个节点上准备脚本launch_online_dp.py。

   import argparse
   import multiprocessing
   import os
   import subprocess
   import sys
   
   def parse_args():
       parser = argparse.ArgumentParser()
       parser.add_argument(
           "--dp-size",
           type=int,
           required=True,
           help="Data parallel size."
       )
       parser.add_argument(
           "--tp-size",
           type=int,
           default=1,
           help="Tensor parallel size."
       )
       parser.add_argument(
           "--dp-size-local",
           type=int,
           default=-1,
           help="Local data parallel size."
       )
       parser.add_argument(
           "--dp-rank-start",
           type=int,
           default=0,
           help="Starting rank for data parallel."
       )
       parser.add_argument(
           "--dp-address",
           type=str,
           required=True,
           help="IP address for data parallel master node."
       )
       parser.add_argument(
           "--dp-rpc-port",
           type=str,
           default=12345,
           help="Port for data parallel master node."
       )
       parser.add_argument(
           "--vllm-start-port",
           type=int,
           default=9000,
           help="Starting port for the engine."
       )
       return parser.parse_args()
   
   args = parse_args()
   dp_size = args.dp_size
   tp_size = args.tp_size
   dp_size_local = args.dp_size_local
   if dp_size_local == -1:
       dp_size_local = dp_size
   dp_rank_start = args.dp_rank_start
   dp_address = args.dp_address
   dp_rpc_port = args.dp_rpc_port
   vllm_start_port = args.vllm_start_port
   
   def run_command(visible_devices, dp_rank, vllm_engine_port):
       command = [
           "bash",
           "./run_dp_template.sh",
           visible_devices,
           str(vllm_engine_port),
           str(dp_size),
           str(dp_rank),
           dp_address,
           dp_rpc_port,
           str(tp_size),
       ]
       subprocess.run(command, check=True)
   
   if __name__ == "__main__":
       template_path = "./run_dp_template.sh"
       if not os.path.exists(template_path):
           print(f"Template file {template_path} does not exist.")
           sys.exit(1)
   
       processes = []
       num_cards = dp_size_local * tp_size
       for i in range(dp_size_local):
           dp_rank = dp_rank_start + i
           vllm_engine_port = vllm_start_port + i
           visible_devices = ",".join(str(x) for x in range(i * tp_size, (i + 1) * tp_size))
           process = multiprocessing.Process(target=run_command,
                                           args=(visible_devices, dp_rank,
                                                   vllm_engine_port))
           processes.append(process)
           process.start()
   
       for process in processes:
           process.join()
   

   Parameter descriptions:

   Parameter	Type	Required	Default	Description
   --dp-size	int	Yes	-	Data parallel size (total number of DP ranks across all nodes).
   --tp-size	int	No	1	Tensor parallel size within each DP rank.
   --dp-size-local	int	No	(same as --dp-size)	Number of DP ranks on the current node. If not set, defaults to --dp-size.
   --dp-rank-start	int	No	0	Starting rank offset for data parallel ranks on this node.
   --dp-address	str	Yes	-	IP address of the data parallel master node.
   --dp-rpc-port	str	No	12345	RPC port for data parallel master communication.
   --vllm-start-port	int	No	9000	Starting port for each vLLM engine instance on this node.

2. Prepare the script run_dp_template.sh on each node.

   1. Prefill node

      nic_name="xxxx" # change to your own nic name
      local_ip=xx.xx.xx.1 # change to your own ip
      
      export HCCL_IF_IP=$local_ip
      export GLOO_SOCKET_IFNAME=$nic_name
      export TP_SOCKET_IFNAME=$nic_name
      export HCCL_SOCKET_IFNAME=$nic_name
      export VLLM_RPC_TIMEOUT=3600000
      export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
      export HCCL_EXEC_TIMEOUT=204
      export HCCL_CONNECT_TIMEOUT=120
      export OMP_PROC_BIND=false
      export OMP_NUM_THREADS=10
      export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
      export HCCL_BUFFSIZE=2560
      export TASK_QUEUE_ENABLE=1
      export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
      export HCCL_OP_EXPANSION_MODE="AIV"
      export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
      export ASCEND_RT_VISIBLE_DEVICES=$1
      
      vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Flash-w8a8-mtp \
          --host 0.0.0.0 \
          --port $2 \
          --data-parallel-size $3 \
          --data-parallel-rank $4 \
          --data-parallel-address $5 \
          --data-parallel-rpc-port $6 \
          --tensor-parallel-size $7 \
          --enable-expert-parallel \
          --seed 1024 \
          --served-model-name dsv4 \
          --max-model-len 1048576 \
          --max-num-batched-tokens 8192 \
          --max-num-seqs 16 \
          --no-disable-hybrid-kv-cache-manager \
          --model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 128}' \
          --no-enable-prefix-caching \
          --safetensors-load-strategy 'prefetch' \
          --speculative-config '{"num_speculative_tokens": 1,"method": "mtp","enforce_eager": true}' \
          --trust-remote-code \
          --block-size 128 \
          --tokenizer-mode deepseek_v4 \
          --tool-call-parser deepseek_v4 \
          --enable-auto-tool-choice \
          --reasoning-parser deepseek_v4 \
          --gpu-memory-utilization 0.9 \
          --quantization ascend \
          --enforce-eager \
          --additional-config '{"enable_cpu_binding": true, "enable_shared_expert_dp": true,  "enable_dsa_cp": true}' \
          --kv-transfer-config \
          '{"kv_connector": "MooncakeHybridConnector",
          "kv_role": "kv_producer",
          "kv_port": "30000",
          "engine_id": "0",
          "kv_connector_extra_config": {
                      "prefill": {
                              "dp_size": 4,
                              "tp_size": 4
                      },
                      "decode": {
                              "dp_size": 16,
                              "tp_size": 1
                      }
              }
          }'
      

   2. Decode node

      nic_name="xxxx" # change to your own nic name
      local_ip=xx.xx.xx.2 # change to your own ip
      
      export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
      export HCCL_OP_EXPANSION_MODE="AIV"
      export TASK_QUEUE_ENABLE=1
      export VLLM_RPC_TIMEOUT=3600000
      export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
      export HCCL_EXEC_TIMEOUT=204
      export HCCL_CONNECT_TIMEOUT=1200
      export HCCL_IF_IP=$local_ip
      export GLOO_SOCKET_IFNAME=$nic_name
      export TP_SOCKET_IFNAME=$nic_name
      export HCCL_SOCKET_IFNAME=$nic_name
      export OMP_PROC_BIND=false
      export OMP_NUM_THREADS=10
      export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
      export HCCL_BUFFSIZE=1024
      export ASCEND_RT_VISIBLE_DEVICES=$1
      
      vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Flash-w8a8-mtp \
          --host 0.0.0.0 \
          --port $2 \
          --data-parallel-size $3 \
          --data-parallel-rank $4 \
          --data-parallel-address $5 \
          --data-parallel-rpc-port $6 \
          --tensor-parallel-size $7 \
          --enable-expert-parallel \
          --seed 1024 \
          --served-model-name dsv4 \
          --max-model-len 1048576 \
          --max-num-batched-tokens 120 \
          --max-num-seqs 60 \
          --async-scheduling \
          --block-size 128 \
          --no-disable-hybrid-kv-cache-manager \
          --no-enable-prefix-caching \
          --safetensors-load-strategy 'prefetch' \
          --trust-remote-code \
          --tokenizer-mode deepseek_v4 \
          --model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 128}' \
          --tool-call-parser deepseek_v4 \
          --enable-auto-tool-choice \
          --reasoning-parser deepseek_v4 \
          --gpu-memory-utilization 0.9 \
          --quantization ascend \
          --speculative-config '{"num_speculative_tokens": 1,"method": "mtp","enforce_eager": true}' \
          --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
          --kv-transfer-config \
          '{"kv_connector": "MooncakeHybridConnector",
          "kv_role": "kv_consumer",
          "kv_port": "30100",
          "engine_id": "1",
          "kv_connector_extra_config": {
                      "prefill": {
                              "dp_size": 4,
                              "tp_size": 4
                      },
                      "decode": {
                              "dp_size": 16,
                              "tp_size": 1
                      }
              }
          }' \
          --additional-config '{
              "ascend_compilation_config":{
                  "enable_npugraph_ex":true,
                  "enable_static_kernel":false
              },
              "enable_cpu_binding":true,
              "multistream_overlap_shared_expert":true,
              "recompute_scheduler_enable":true
          }'
      

3. Start the server with the following command on each node.

   1. Prefill node

      # change ip to your own
      python launch_online_dp.py --dp-size 4 --tp-size 4 --dp-size-local 4 --dp-rank-start 0 --dp-address xx.xx.xx.1 --dp-rpc-port 12321 --vllm-start-port 7100
      

   2. Decode node

      # change ip to your own
      python launch_online_dp.py --dp-size 16 --tp-size 1 --dp-size-local 16 --dp-rank-start 0 --dp-address xx.xx.xx.2 --dp-rpc-port 12321 --vllm-start-port 7100
      

4. Deploy the P-D disaggregation proxy.

   Refer to Prefill-Decode Disaggregation (Deepseek) to deploy the P-D disaggregation proxy.

5.2.2 A2 Series PD Separation Deployment

This section shows the deployment guide of DeepSeek-V4-Flash on Atlas 800 A2 (64G × 8) multi-node environment with 4*1P 1*4D for better performance.

Before you start, please:

1. Prepare the script launch_online_dp.py on each node.

   import argparse
   import multiprocessing
   import os
   import subprocess
   import sys
   
   def parse_args():
       parser = argparse.ArgumentParser()
       parser.add_argument("--dp-size", type=int, required=True, help="Data parallel size.")
       parser.add_argument("--tp-size", type=int, default=1, help="Tensor parallel size.")
       parser.add_argument("--dp-size-local", type=int, default=-1, help="Local data parallel size.")
       parser.add_argument("--dp-rank-start", type=int, default=0, help="Starting rank for data parallel.")
       parser.add_argument("--dp-address", type=str, required=True, help="IP address for data parallel master node.")
       parser.add_argument("--dp-rpc-port", type=str, default=12345, help="Port for data parallel master node.")
       parser.add_argument("--vllm-start-port", type=int, default=9000, help="Starting port for the engine.")
       return parser.parse_args()
   
   args = parse_args()
   dp_size = args.dp_size
   tp_size = args.tp_size
   dp_size_local = args.dp_size_local
   if dp_size_local == -1:
       dp_size_local = dp_size
   dp_rank_start = args.dp_rank_start
   dp_address = args.dp_address
   dp_rpc_port = args.dp_rpc_port
   vllm_start_port = args.vllm_start_port
   
   def run_command(visible_devices, dp_rank, vllm_engine_port):
       command = [
           "bash",
           "./run_dp_template.sh",
           visible_devices,
           str(vllm_engine_port),
           str(dp_size),
           str(dp_rank),
           dp_address,
           dp_rpc_port,
           str(tp_size),
       ]
       subprocess.run(command, check=True)
   
   if __name__ == "__main__":
       template_path = "./run_dp_template.sh"
       if not os.path.exists(template_path):
           print(f"Template file {template_path} does not exist.")
           sys.exit(1)
   
       processes = []
       num_cards = dp_size_local * tp_size
       for i in range(dp_size_local):
           dp_rank = dp_rank_start + i
           vllm_engine_port = vllm_start_port + i
           visible_devices = ",".join(str(x) for x in range(i * tp_size, (i + 1) * tp_size))
           process = multiprocessing.Process(target=run_command, args=(visible_devices, dp_rank, vllm_engine_port))
           processes.append(process)
           process.start()
   
       for process in processes:
           process.join()
   

2. Prepare the script run_dp_template.sh on each node.

   1. Prefill node (4 P nodes share the same script)

      For each P instance, only these two configuration values need to be modified: kv_port and engine_id. The engine_id should start from 0 and increment sequentially, while the kv_port (e.g., 30100) must be unique for each P instance, such as 30000, 30100, etc.

      unset ftp_proxy
      unset https_proxy
      unset http_proxy
      rm -rf ~/ascend/log
      
      nic_name="xxxxxx" #eg."enp67s0f0np0"
      local_ip=`hostname -I|awk -F " " '{print$1}'`
      
      export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
      export HCCL_OP_EXPANSION_MODE="AIV"
      export TASK_QUEUE_ENABLE=1
      export VLLM_RPC_TIMEOUT=3600000
      export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
      export HCCL_EXEC_TIMEOUT=204
      export HCCL_CONNECT_TIMEOUT=1200
      
      export HCCL_IF_IP=$local_ip
      export GLOO_SOCKET_IFNAME=$nic_name
      export TP_SOCKET_IFNAME=$nic_name
      export HCCL_SOCKET_IFNAME=$nic_name
      export OMP_PROC_BIND=false
      export OMP_NUM_THREADS=10
      export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
      export HCCL_BUFFSIZE=1024
      
      export ASCEND_RT_VISIBLE_DEVICES=$1
      export TASK_QUEUE_ENABLE=1
      
      vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Flash-w8a8-mtp \
          --host 0.0.0.0 \
          --port $2 \
          --data-parallel-size $3 \
          --data-parallel-rank $4 \
          --data-parallel-address $5 \
          --data-parallel-rpc-port $6 \
          --tensor-parallel-size $7 \
          --enable-expert-parallel \
          --seed 1024 \
          --served-model-name dsv4 \
          --max-model-len 135000 \
          --max-num-batched-tokens 4096 \
          --max-num-seqs 16 \
          --block-size 128 \
          --enforce-eager \
          --async-scheduling \
          --no-disable-hybrid-kv-cache-manager \
          --enable-prefix-caching \
          --trust-remote-code \
          --gpu-memory-utilization 0.9 \
          --quantization ascend \
          --safetensors-load-strategy 'prefetch' \
          --model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 128}' \
          --tokenizer-mode deepseek_v4 \
          --tool-call-parser deepseek_v4 \
          --enable-auto-tool-choice \
          --reasoning-parser deepseek_v4 \
          --additional-config '{"enable_cpu_binding": true, "enable_shared_expert_dp": true}' \
          --speculative-config '{"num_speculative_tokens": 1, "method": "mtp","enforce_eager": true}' \
          --kv-transfer-config \
          '{"kv_connector": "MooncakeHybridConnector",
          "kv_role": "kv_producer",
          "kv_port": "30000",
          "engine_id": "0",
          "kv_connector_extra_config": {
                      "prefill": {
                          "dp_size": 8,
                          "tp_size": 1
                      },
                      "decode": {
                          "dp_size": 32,
                          "tp_size": 1
                      }
              }
          }'
      

   2. Decode node (4 D nodes share the same script)

      unset ftp_proxy
      unset https_proxy
      unset http_proxy
      rm -rf ~/ascend/log
      
      nic_name="xxxxxx" #eg."enp67s0f0np0"
      local_ip=`hostname -I|awk -F " " '{print$1}'`
      
      export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD
      export HCCL_OP_EXPANSION_MODE="AIV"
      export TASK_QUEUE_ENABLE=1
      export VLLM_RPC_TIMEOUT=3600000
      export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=30000
      export HCCL_EXEC_TIMEOUT=204
      export HCCL_CONNECT_TIMEOUT=1200
      
      export HCCL_IF_IP=$local_ip
      export GLOO_SOCKET_IFNAME=$nic_name
      export TP_SOCKET_IFNAME=$nic_name
      export HCCL_SOCKET_IFNAME=$nic_name
      export OMP_PROC_BIND=false
      export OMP_NUM_THREADS=10
      export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
      export HCCL_BUFFSIZE=1024
      
      export ASCEND_RT_VISIBLE_DEVICES=$1
      
      vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/DeepSeek-V4-Flash-w8a8-mtp \
          --host 0.0.0.0 \
          --port $2 \
          --data-parallel-size $3 \
          --data-parallel-rank $4 \
          --data-parallel-address $5 \
          --data-parallel-rpc-port $6 \
          --tensor-parallel-size $7 \
          --enable-expert-parallel \
          --seed 1024 \
          --served-model-name dsv4 \
          --max-model-len 135000 \
          --max-num-batched-tokens 60 \
          --max-num-seqs 30 \
          --async-scheduling \
          --block-size 128 \
          --no-disable-hybrid-kv-cache-manager \
          --no-enable-prefix-caching \
          --trust-remote-code \
          --gpu-memory-utilization 0.9 \
          --quantization ascend \
          --safetensors-load-strategy 'prefetch' \
          --model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 128}' \
          --tokenizer-mode deepseek_v4 \
          --tool-call-parser deepseek_v4 \
          --enable-auto-tool-choice \
          --reasoning-parser deepseek_v4 \
          --speculative-config '{"num_speculative_tokens": 1, "method": "mtp","enforce_eager": true}' \
          --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
          --kv-transfer-config \
          '{"kv_connector": "MooncakeHybridConnector",
          "kv_role": "kv_consumer",
          "kv_port": "30400",
          "engine_id": "4",
          "kv_connector_extra_config": {
                      "prefill": {
                          "dp_size": 8,
                          "tp_size": 1
                      },
                      "decode": {
                          "dp_size": 32,
                          "tp_size": 1
                      }
              }
          }' \
          --additional-config '{
              "ascend_compilation_config":{
                    "enable_npugraph_ex":true,
                    "enable_static_kernel":false
              },
             "enable_cpu_binding":true,
             "multistream_overlap_shared_expert":true,
             "recompute_scheduler_enable":true
          }'
      

3. Start the server with the following command on each node.

   1. Prefill node

      # change ip to your own
      python launch_online_dp.py --dp-size 8 --tp-size 1 --dp-size-local 8 --dp-rank-start 0 --dp-address x.x.x.x --dp-rpc-port 12321 --vllm-start-port 7100
      

      For each P instance, only the --dp-address parameter differs and must be configured as the IP address of the service within the same subnet as the other instances.

   2. Decode node

      # change ip to your own
      python launch_online_dp.py --dp-size 32 --tp-size 1 --dp-size-local 8 --dp-rank-start x --dp-address x.x.x.x --dp-rpc-port 12321 --vllm-start-port 7100
      

      For each D instance, only the --dp-rank-start parameter differs, which should be configured as 0, 8, 16, and 24 respectively. Each instance's --dp-address must be set to the IP address of the main D node, which is the IP of the Decode instance with --dp-rank-start set to 0.

4. Deploy the P-D disaggregation proxy.

   The proxy is also implemented by referring to Prefill-Decode Disaggregation (Deepseek).

Key Parameter Descriptions:

• VLLM_ASCEND_ENABLE_FLASHCOMM1=1: enables the communication optimization function on the prefill nodes.
• recompute_scheduler_enable: true: enables the recomputation scheduler. When the KV Cache of the decode node is insufficient, requests will be sent to the prefill node to recompute the KV Cache. In the PD separation scenario, enable this configuration only on decode nodes.
• MooncakeHybridConnector: the KV transfer connector used for PD separation, transferring KV Cache between prefill and decode nodes.
• enable_shared_expert_dp: true：启用共享专家的数据并行，适用于MoE模型。

部署验证：

PD分离服务完全启动后，通过prefill主节点上的代理端口发送请求，验证Prefill和Decode节点是否协同工作正常。代理验证方法请参考Prefill-Decode分离式部署(Deepseek)。

Common Issues Tip: If you encounter issues with PD separation deployment, please refer to the Public FAQ for troubleshooting.

5.2.3 超长序列部署

对于超长序列场景，可通过调整PD(Prefill/Decode)比例和模型并行策略实现支持。例如，在1M序列场景中，可使用1*4P-1*4D比例，模型并行设置为DP4TP8模式。

6 功能验证

服务器启动后，您可以使用输入提示查询模型：

在:中，使用主节点的IP地址和端口号。如果主备节点分离，请使用代理节点的IP地址和端口号。

curl http://<node0_ip>:<port>/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "dsv4",
        "messages": [
            {
                "role": "user",
                "content": "Who are you?"
            }
        ],
        "max_tokens": 256,
        "temperature": 0
    }'

预期结果：

服务返回HTTP 200 OK，JSON响应中包含choices字段。

7 精度评估

以下是两种精度评估方法。

使用AISBench

1. 详情请参考使用AISBench。

2. 执行后即可获取结果。

数据集	版本	指标	模式	vllm-api-生成eral-chat	备注
GPQA	-	准确率	生成	88.17	1台Atlas 800 A3 (128G × 8)
GSM8K	-	准确率	生成	96.30	1台Atlas 800 A3 (128G × 8)

8 性能评估

使用AISBench

详情请参考使用AISBench进行性能评估。

使用vLLM Benchmark

更多详情请参考vllm benchmark。

9 性能调优

9.1 推荐配置

注意：以下配置在特定测试环境中验证，仅供参考。最佳配置取决于最大输入/输出长度、前缀缓存命中率、精度要求和部署机器比例等因素。建议根据实际情况参考第9.2节进行调优。

表1：场景概览

*Total NPUs表示所有节点使用的NPU总数。

场景	部署模式	*Total NPUs	权重版本	关键考量
高吞吐	单节点混合	16 (A3)	DeepSeek-V4-Flash-w8a8-mtp	使用dp4 tp4平衡内存容量和计算效率
高吞吐	1P1D部署	32 (A3)	DeepSeek-V4-Flash-w8a8-mtp	P和D节点均使用dp16 tp1；平衡延迟和吞吐量
长上下文 (1M)	单节点 (A3)	8 (A3)	DeepSeek-V4-Flash-w8a8-mtp	使用dp4 tp4平衡内存容量和计算效率
长上下文 (1M)	1P1D部署	32 (A3)	DeepSeek-V4-Flash-w8a8-mtp	P和D节点均使用dp16 tp1；平衡延迟和吞吐量

表2：详细节点配置

场景	配置	NPUs	TP	DP	Max Num Seqs	Max Num Batched Tokens	Max Model Len	MTP Speculation Num
高吞吐 (A3)	服务器/单机	8	4	4	64	10240	1048576	1
长上下文 (1M, A3)	服务器/单机	8	4	4	64	10240	1048576	1
PD分离 (A3)	Server-P节点	8	4	4	16	8192	1048576	1
PD分离 (A3)	Server-D节点	8	1	16	60	120	1048576	1

完整的启动命令和参数说明，请参考第5章中的部署示例。

注意：

max-model-len和max-num-seqs需要根据实际使用场景设置。其他设置请参考部署章节。

目前，我们以实验性方式支持4K前缀缓存命中。您只需将服务中的--block-size值从128改为32即可。

9.2 调优指南

9.2.1 通用调优参考

调优方法请参考公共性能调优文档。

详细功能描述请参考功能指南。

10 常见问题

For common environment, installation, and 生成eral parameter issues, please refer to the Public FAQ; this chapter only covers 模式l-specific issues.