跳转至

KV 缓存池（Ascend Store）部署指南

目录

环境依赖

  • 软件要求：
    • CANN >= 8.5.0
    • vLLM：main 分支
    • vLLM-Ascend：main 分支
    • mooncake：>= 0.3.9

KV 池参数说明

kv_load_failure_policy：KV 加载失败处理策略

kv_load_failure_policy 是 kv-transfer-config 中的顶级字段。

  • recompute：当 KV 加载失败时，vLLM 将请求回滚到最后一个有效前缀，并重新调度以重新计算失败的 KV 块。混合注意力模型（例如 DeepSeekV4、Qwen 3.5）尚不支持。
  • fail：当 KV 加载失败时，受影响的请求将直接终止并返回错误。

vLLM 中的默认值为 fail。如果希望在 KV 加载失败后请求回退到重新计算，请将其设置为 recompute。

使用 MultiConnector 时，请在 kv_load_failure_policy 顶级 MultiConnector 上配置 kv-transfer-config，而不是在子连接器上配置。

kv_connector_extra_config：池化的额外可配置参数

参数 描述
lookup_rpc_port 池化调度器进程与工作进程间 RPC 通信的端口：每个实例需要配置唯一的端口。
load_async 是否启用异步加载。默认值为 false。
backend 设置 kvpool 的存储后端 (mooncake, memcache, yuanrong)，默认为 mooncake。
consumer_is_to_put 解码节点是否将 KV 缓存放入 KV 池。默认值为 false。
consumer_is_to_load 解码节点是否从 KV 池加载 KV 缓存。默认值为 false。
prefill_pp_size Prefill PP 大小，当 Prefill 节点启用 PP 时需要设置。
prefill_pp_layer_partition Prefill PP 层划分，当 Prefill 节点启用 PP 时需要设置。

环境变量配置

为保证生成的哈希值一致，在启用 KV 池时，必须在所有节点上同步 ```bash export PYTHONHASHSEED=0 

## 使用 Mooncake 作为 KV 池后端的示例

* 软件要求：
    * 检查 NPU HCCN 配置：

        确保环境中存在 `bash
        cat /etc/hccn.conf
        ` 文件。如果使用 Docker，请将其挂载到容器中。

        ```

    * Install Mooncake

        Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
        Installation and Compilation Guide: <https://github.com/kvcache-ai/Mooncake?tab=readme-ov-file#build-and-use-binaries>.
        First, we need to obtain the Mooncake project. Refer to the following command:

        ```shell
        git clone -b v0.3.9 --depth 1 https://github.com/kvcache-ai/Mooncake.git
        ```

        (Optional) Replace go install url if the network is poor

        ```shell
        cd Mooncake
        sed -i 's|https://go.dev/dl/|https://golang.google.cn/dl/|g' dependencies.sh
        ```

        Install mpi

        ```shell
        apt-get install mpich libmpich-dev -y
        ```

        Install the relevant dependencies. The installation of Go is not required.

        ```shell
        bash dependencies.sh -y
        ```

        Compile and install

        ```shell
        mkdir build
        cd build
        cmake .. -DUSE_ASCEND_DIRECT=ON
        make -j
        make install
        ```

        Set environment variables

        **Note:**

        * Adjust the Python path according to your specific Python installation
        * Ensure ``` and `/usr/local/lib` are in your `/usr/local/lib64`

        `LD_LIBRARY_PATH`shell
        export LD_LIBRARY_PATH=/usr/local/lib64/python3.12/site-packages/mooncake:$LD_LIBRARY_PATH
        ```

### Environment Variables Description

| Hardware | Dependencies | Export Command | Description |
| :--- | :--- | :--- | :--- |
| 800 I/T A3 series | HDK >= 26.0<br>or HDK >= 25.5 with mooncake >= v0.3.11<br>CANN >= 9.0.0<br>LingQu Computing Network >= 1.5 | ``` | **Recommended**. Enables unified memory address direct transmission scheme. With SSD offload, see [Fabric memory size alignment](#122-fabric-memory-size-alignment-a3--ascend_enable_use_fabric_mem1) — memory sizes must be aligned to 1GB. |
| 800 I/T A3 series | If any dependency above is not met | `export ASCEND_ENABLE_USE_FABRIC_MEM=1` | Configures the number and size of buffers on the NPU Device for aggregation and KV transfer (e.g., `export ASCEND_BUFFER_POOL=4:8` means 4 buffers of 8MB). |
| 800 I/T A2 series | HDK >= 25.5 is recommended | `4:8](https://github.com/kvcache-ai/Mooncake/blob/main/docs/source/deployment/mooncake-store-deployment-guide.md)export HCCL_INTRA_ROCE_ENABLE=1](https://github.com/kvcache-ai/Mooncake/blob/main/docs/source/deployment/ssd-offload.md)mooncake.json` 所在的完整路径。

```shell
{
    "metadata_server": "P2PHANDSHAKE",
    "protocol": "ascend",
    "device_name": "",
    "master_server_address": "xx.xx.xx.xx:50088",
    "global_segment_size": "1GB" (1024MB/1048576KB/1073741824B/1073741824),
    "preferred_segment": false,
    "prefer_alloc_in_same_node": true
}

metadata_server: Configured as P2PHANDSHAKE.
protocol: Must be set to 'Ascend' on the NPU. device_name: "" master_server_address: Configured with the IP and port of the master service. It can also be set via the MOONCAKE_MASTER environment variable, which takes precedence over this configuration item (useful for injecting the master address through Kubernetes).
global_segment_size: Registered memory size per card to the KV Pool. Needs to be aligned to 1GB. It can also be set via the MOONCAKE_GLOBAL_SEGMENT_SIZE environment variable, which takes precedence over this configuration item.
preferred_segment: Whether to prefer storing KV on the local segment when putting objects to the KV Pool. 默认值s to false.
prefer_alloc_in_same_node: Whether to prefer allocating KV on the same node. 默认值s to true.

2.启动 mooncake_master

在 mooncake 文件夹下：

mooncake_master --port 50088 --eviction_high_watermark_ratio 0.9 --eviction_ratio 0.1 --default_kv_lease_ttl 11000

eviction_high_watermark_ratio determines the watermark where Mooncake Store will perform eviction, and eviction_ratio determines the portion of stored objects that would be evicted. default_kv_lease_ttl controls the default lease TTL for KV objects (milliseconds); configure it via --default_kv_lease_ttl and keep it larger than ASCEND_CONNECT_TIMEOUT and ASCEND_TRANSFER_TIMEOUT.

PD 分离（Prefill-Decode Disaggregation）场景

1.运行 prefill 节点和 decode 节点

使用 MultiConnector 同时利用 MooncakeConnectorV1 和 AscendStoreConnector。MooncakeConnectorV1 负责 kv_transfer（KV 传输），而 AscendStoreConnector 作为前缀缓存（prefix-cache）节点。

prefill 节点：

bash multi_producer.sh

multi_producer.sh 脚本的内容：

export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:$LD_LIBRARY_PATH
export PYTHONHASHSEED=0
export PYTHONPATH=$PYTHONPATH:/xxxxx/vllm
export MOONCAKE_CONFIG_PATH="/xxxxxx/mooncake.json"
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3
export ACL_OP_INIT_MODE=1
#A3
export ASCEND_ENABLE_USE_FABRIC_MEM=1
#A2
#export HCCL_INTRA_ROCE_ENABLE=1

#Minimum retransmission timeout of the RDMA, equals 4.096 μs * 2 ^ timeout.
#Needs to satisfy the equation: ASCEND_TRANSFER_TIMEOUT > RDMA_TIMEOUT * 7, where 7 is the default number of retry for RDMA transfer.
#HCCL_RDMA_TIMEOUT also affects collective communication behavior and should be configured carefully.
export HCCL_RDMA_TIMEOUT=17

# Unit: ms. The timeout for one-sided communication connection establishment is set to 10 seconds by default (see PR: https://github.com/kvcache-ai/Mooncake/pull/1039). Users can adjust this value based on their specific setup.
# The recommended formula is: ASCEND_CONNECT_TIMEOUT = connection_time_per_card (typically within 500ms) × total_number_of_Decode_cards.
# This ensures that even in the worst-case scenario—where all Decode cards simultaneously attempt to connect to the same Prefill card the connection will not time out.
export ASCEND_CONNECT_TIMEOUT=10000

# Unit: ms. The timeout for one-sided communication transfer is set to 10 seconds by default (see PR: https://github.com/kvcache-ai/Mooncake/pull/1039).
export ASCEND_TRANSFER_TIMEOUT=10000

python3 -m vllm.entrypoints.openai.api_server \
    --model /xxxxx/Qwen2.5-7B-Instruct \
    --port 8100 \
    --trust-remote-code \
    --enforce-eager \
    --no-enable-prefix-caching \
    --tensor-parallel-size 1 \
    --data-parallel-size 1 \
    --max-model-len 32768 \
    --block-size 128 \
    --max-num-batched-tokens 16384 \
    --kv-transfer-config \
    '{
    "kv_connector": "MultiConnector",
    "kv_role": "kv_producer",
    "kv_load_failure_policy": "recompute",
    "kv_connector_extra_config": {
        "connectors": [
            {
                "kv_connector": "MooncakeConnectorV1",
                "kv_role": "kv_producer",
                "kv_port": "20001",
                "kv_connector_extra_config": {
                    "prefill": {
                        "dp_size": 1,
                        "tp_size": 1
                    },
                    "decode": {
                        "dp_size": 1,
                        "tp_size": 1
                    }
                }
            },
            {
                "kv_connector": "AscendStoreConnector",
                "kv_role": "kv_producer",
                "kv_connector_extra_config": {
                    "lookup_rpc_port":"0",
                    "backend": "mooncake"
                }
            }  
        ]
    }
    }'

decode 节点：

bash multi_consumer.sh

multi_consumer.sh 的内容：

export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:$LD_LIBRARY_PATH
export PYTHONPATH=$PYTHONPATH:/xxxxx/vllm
export PYTHONHASHSEED=0
export MOONCAKE_CONFIG_PATH="/xxxxx/mooncake.json"
export ASCEND_RT_VISIBLE_DEVICES=4,5,6,7
export ACL_OP_INIT_MODE=1
#A3
export ASCEND_ENABLE_USE_FABRIC_MEM=1
#A2
#export HCCL_INTRA_ROCE_ENABLE=1
export HCCL_RDMA_TIMEOUT=17
export ASCEND_CONNECT_TIMEOUT=10000
export ASCEND_TRANSFER_TIMEOUT=10000

python3 -m vllm.entrypoints.openai.api_server \
    --model /xxxxx/Qwen2.5-7B-Instruct \
    --port 8200 \
    --trust-remote-code \
    --enforce-eager \
    --no-enable-prefix-caching \
    --tensor-parallel-size 1 \
    --data-parallel-size 1 \
    --max-model-len 32768 \
    --block-size 128 \
    --max-num-batched-tokens 16384 \
    --kv-transfer-config \
    '{
    "kv_connector": "MultiConnector",
    "kv_role": "kv_consumer",
    "kv_load_failure_policy": "recompute",
    "kv_connector_extra_config": {
        "connectors": [
        {
                "kv_connector": "MooncakeConnectorV1",
                "kv_role": "kv_consumer",
                "kv_port": "20002",
                "kv_connector_extra_config": {
                    "prefill": {
                        "dp_size": 1,
                        "tp_size": 1
                    },
                    "decode": {
                        "dp_size": 1,
                        "tp_size": 1
                    }
                }
            },
            {
                "kv_connector": "AscendStoreConnector",
                "kv_role": "kv_consumer",
                "kv_connector_extra_config": {
                    "lookup_rpc_port":"0",
                    "backend": "mooncake"
                }
            }
        ]
    }
    }'

目前，PD 解耦场景中的键值池默认仅存储由 Prefill 节点生成的 KV 缓存。在使用 MLA 的模型中，现已支持 Decode 节点存储 KV 缓存供 Prefill 节点使用，可通过在 consumer_is_to_put: true 中添加 prefill_pp_size 来启用。如果 Prefill 节点启用了流水线并行（PP），则还需要设置 prefill_pp_layer_partition 或 ```python { "kv_connector": "AscendStoreConnector", "kv_role": "kv_consumer", "kv_load_failure_policy": "recompute", "kv_connector_extra_config": { "lookup_rpc_port": "0", "backend": "mooncake", "consumer_is_to_put": true, "prefill_pp_size": 2, "prefill_pp_layer_partition": "30,31" } } 

#### 2.启动代理服务器

```shell
python vllm-ascend/examples/disaggregated_prefill_v1/load_balance_proxy_server_example.py \
    --host localhost \
    --prefiller-hosts localhost \
    --prefiller-ports 8100 \
    --decoder-hosts localhost \
    --decoder-ports 8200 \

将 localhost 更改为您的实际 IP 地址。

3.运行推理

在命令中将 localhost、端口和模型权重路径配置为您自己的设置。

短问题：

curl -s http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{ "model": "/xxxxx/Qwen2.5-7B-Instruct", "prompt": "Hello. I have a question. The president of the United States is", "max_completion_tokens": 200, "temperature":0.0 }'

长问题：

curl -s http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{ "model": "/xxxxx/Qwen2.5-7B-Instruct", "prompt": "Given the accelerating impacts of climate change—including rising sea levels, increasing frequency of extreme weather events, loss of biodiversity, and adverse effects on agriculture and human health—there is an urgent need for a robust, globally coordinated response. However, international efforts are complicated by a range of factors: economic disparities between high-income and low-income countries, differing levels of industrialization, varying access to clean energy technologies, and divergent political systems that influence climate policy implementation. In this context, how can global agreements like the Paris Accord be redesigned or strengthened to not only encourage but effectively enforce emission reduction targets? Furthermore, what mechanisms can be introduced to promote fair and transparent technology transfer, provide adequate financial support for climate adaptation in vulnerable regions, and hold nations accountable without exacerbating existing geopolitical tensions or disproportionately burdening those with historically lower emissions?", "max_completion_tokens": 256, "temperature":0.0 }'

PD 混部推理

1.运行混部脚本

bash pd_mix.sh

pd_mix.sh 的内容：

export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:$LD_LIBRARY_PATH
export PYTHONPATH=$PYTHONPATH:/xxxxx/vllm
export MOONCAKE_CONFIG_PATH="/xxxxxx/mooncake.json"
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3
export PYTHONHASHSEED=0
export ACL_OP_INIT_MODE=1
#A3
export ASCEND_ENABLE_USE_FABRIC_MEM=1
#A2
#export HCCL_INTRA_ROCE_ENABLE=1
export HCCL_RDMA_TIMEOUT=17
export ASCEND_CONNECT_TIMEOUT=10000
export ASCEND_TRANSFER_TIMEOUT=10000

python3 -m vllm.entrypoints.openai.api_server \
    --model /xxxxx/Qwen2.5-7B-Instruct \
    --port 8100 \
    --trust-remote-code \
    --enforce-eager \
    --no-enable-prefix-caching \
    --tensor-parallel-size 1 \
    --data-parallel-size 1 \
    --max-model-len 32768 \
    --block-size 128 \
    --max-num-batched-tokens 16384 \
    --kv-transfer-config \
    '{
    "kv_connector": "AscendStoreConnector",
    "kv_role": "kv_both",
    "kv_load_failure_policy": "recompute",
    "kv_connector_extra_config": {
        "lookup_rpc_port":"1",
        "backend": "mooncake"
    }
}' > mix.log 2>&1

2.运行推理

在命令中将 localhost、端口和模型权重路径配置为您自己的设置。发送的请求将仅到达混部脚本所在的端口，无需启动单独的代理。

短问题：

curl -s http://localhost:8100/v1/completions -H "Content-Type: application/json" -d '{ "model": "/xxxxx/Qwen2.5-7B-Instruct", "prompt": "Hello. I have a question. The president of the United States is", "max_completion_tokens": 200, "temperature":0.0 }'

长问题：

curl -s http://localhost:8100/v1/completions -H "Content-Type: application/json" -d '{ "model": "/xxxxx/Qwen2.5-7B-Instruct", "prompt": "Given the accelerating impacts of climate change—including rising sea levels, increasing frequency of extreme weather events, loss of biodiversity, and adverse effects on agriculture and human health—there is an urgent need for a robust, globally coordinated response. However, international efforts are complicated by a range of factors: economic disparities between high-income and low-income countries, differing levels of industrialization, varying access to clean energy technologies, and divergent political systems that influence climate policy implementation. In this context, how can global agreements like the Paris Accord be redesigned or strengthened to not only encourage but effectively enforce emission reduction targets? Furthermore, what mechanisms can be introduced to promote fair and transparent technology transfer, provide adequate financial support for climate adaptation in vulnerable regions, and hold nations accountable without exacerbating existing geopolitical tensions or disproportionately burdening those with historically lower emissions?", "max_completion_tokens": 256, "temperature":0.0 }'

注意：对于启用了 ASCEND_BUFFER_POOL 的 MooncakeStore，建议在实际性能基准测试前进行预热阶段。

这是因为当涉及设备到设备通信时，HCCL 单边通信连接是在实例启动后延迟创建的。目前，需要在所有设备之间建立全连接。建立这些连接会引入一次性时间开销和持续的设备内存消耗（每个连接消耗 4 MB 设备内存）。

对于预热，建议发送输入序列长度为 8K、输出序列长度为 1 的请求，请求总数应为设备（卡/芯片）数量的 2-3 倍。

使用嵌入式真实客户端模式启用 MooncakeStore SSD 卸载

  • 此功能需要 mooncake >= v0.3.11。

启动 mooncake_master

按照运行 Mooncake Master 中的描述启动 Mooncake master。要启用 SSD 卸载，请在相同的 master 启动命令中添加 --enable_offload=true。例如：

mooncake_master --port 50088 --eviction_high_watermark_ratio 0.9 --eviction_ratio 0.1 --default_kv_lease_ttl 11000 --enable_offload=true --client_ttl=120
字段 描述
enable_offload 设置为 true 以在 Mooncake master 中启用 SSD 卸载。保持 master 端口与 master_server_address 中的 mooncake.json 一致。
client_ttl 客户端在最后一次 Ping 后保持存活的秒数。CLI 默认值为 10；请参见 SSD 卸载时的 SEGMENT_NOT_FOUND。

配置

从运行 Mooncake Mastermooncake.json 中配置的 开始，添加以下 SSD 卸载字段：

{
    "enable_ssd_offload": true,
    "ssd_offload_path": "/nvme/mooncake_offload"
}
字段 描述
enable_ssd_offload 设置为 true 以启用 SSD 卸载。不支持环境变量。
ssd_offload_path 当 enable_ssd_offload 为 true 时必填。 Mooncake 存储卸载的 KV 数据的本地目录的绝对路径（例如，/nvme/mooncake_offload）。该目录必须存在且可由 vLLM 进程写入；请在启动前创建它（mkdir -p <path>）。Mooncake 会拒绝相对路径、符号链接以及包含 .. 的路径。该值作为 SSD 存储根目录传递给 MooncakeDistributedStore.setup()（相当于独立客户端中的 MOONCAKE_OFFLOAD_FILE_STORAGE_PATH）。仅在 mooncake.json 中配置此字段；不支持环境变量。

运行嵌入式真实客户端

在模式 A（嵌入式真实客户端）下，Mooncake 嵌入在 vLLM 中。当 vLLM 服务启动时，AscendStoreConnector / MooncakeBackend 会使用 MooncakeDistributedStore.setup() 中的设置（包括启用 SSD 卸载时的 mooncake.json 和 enable_ssd_offload）自动调用 ssd_offload_path。无需单独的 mooncake_client 进程。

SSD 磁盘使用控制

以下环境变量控制 SSD 卸载（bucket 后端）的磁盘空间使用：

环境变量 默认值 描述
MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES 1342177280（1280 MB） 每个 rank 的 SSD 读/写缓冲区大小（字节）。不可**在 mooncake.json 中配置。如果遇到 BUFFER_OVERFLOW，请增大此值——请参见 调整 MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES 的大小。**在启用了 ASCEND_ENABLE_USE_FABRIC_MEM=1 的 A3 上，必须按 1GB 对齐，并计入每个 rank 的 fabric 内存配额（请参见 Fabric 内存大小对齐）。
MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE 0 以字节为单位的驱逐阈值。当设置为 0 时，后端使用**物理磁盘容量的 90%** 作为配额。设置显式值以精确控制磁盘使用。
MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY none 驱逐策略：none（满时写入失败）、fifo 或 lru。
MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES 2199023255552（2 TB） 每 rank 报告给 Mooncake master 的最大磁盘使用量。Master 跨客户端聚合此值（在 SSD Storage 总计中约为 2 TB × rank 数量）。**始终覆盖**以匹配实际磁盘容量——默认值通常超过可用空间。

**MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES 风险：**如果保持 2 TB 默认值，master 显示的总 SSD 配额远大于物理磁盘（例如，16 ranks → 在 1 TB NVMe 上显示约 32 TB）。磁盘填满时卸载仍会失败，而监控显示正常。在生产使用前，请将此值设置为实际的每 rank 预算。

由于每个 TP rank 在 rank_0/ 下使用独立的 SSD 子目录（rank_1/、ssd_offload_path、...），所有 rank 共享同一物理磁盘。为防止单个 rank 消耗过多空间，请设置显式的每 rank 配额。例如，对于 800 GB 磁盘和 8 个 TP rank：

# 800 GB total disk, 8 ranks, ~100 GB per rank
export MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES=$((100 * 1024 * 1024 * 1024))
export MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE=$((100 * 1024 * 1024 * 1024))
export MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY=lru
export MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES=1073741824   # 1 GB

使用 Memcache 作为 KV 池后端的示例

安装 Memcache

MemCache 依赖于 MemFabric。因此，必须先安装 MemFabric。在 MemFabric 安装完成后，再安装 Memcache。

pip install memfabric-hybrid
pip install memcache-hybrid

配置 Memcache 配置文件

mmc-meta.conf：

ock.mmc.meta_service_url = tcp://xx.xx.xx.xx:5000
ock.mmc.meta_service.config_store_url = tcp://xx.xx.xx.xx:6000
ock.mmc.log_level = error

mmc-local.conf：

ock.mmc.meta_service_url = tcp://xx.xx.xx.xx:5000
ock.mmc.local_service.config_store_url = tcp://xx.xx.xx.xx:6000
ock.mmc.log_level = error
ock.mmc.local_service.world_size = 256
ock.mmc.local_service.protocol = device_sdma
ock.mmc.local_service.dram.size = 1GB

关键要点：

参数 描述
ock.mmc.meta_service_url 配置主节点的 IP 地址和端口号。P 节点和 D 节点的 IP 地址和端口号可以相同。
ock.mmc.local_service.config_store_url 配置主节点的 IP 地址和端口号。P 节点和 D 节点的 IP 地址和端口号可以相同。
ock.mmc.local_service.world_size 本地服务的总数，包括未来将添加的服务。
ock.mmc.local_service.protocol device_rdma（当设备 ROCE 可用时，A2 和 A3 支持，推荐用于 A2），device_sdma（当 HCCS 可用时，A3 支持，推荐用于 A3）。目前不支持异构协议设置。
ock.mmc.local_service.dram.size 设置主节点占用的内存大小。配置的值是每张卡占用的内存大小。

运行 Memcache Master

启动 MetaService 服务。

export MMC_META_CONFIG_PATH=/usr/local/memcache_hybrid/latest/config/mmc-meta.conf

python -c "from memcache_hybrid import MetaService; MetaService.main()"

PD 分离（Prefill-Decode Disaggregation）场景

1.运行 prefill 节点和 decode 节点

使用 MultiConnector 同时利用 MooncakeConnectorV1 和 AscendStoreConnector。MooncakeConnectorV1 执行 kv_transfer，而 AscendStoreConnector 启用 KV 缓存池。

800I A2/800T A2/800I A3/800T A3 系列

run_prefill.sh/run_decode.sh:

#!/bin/bash

ROLE="prefill"              # prefill / decode
HARDWARE_SERIES="A2"        # A2 (800I/800T A2) or A3 (800I/800T A3)
LOCAL_IP="xx.xx.xx.xx"
NIC_NAME="xxxxxx"

MODEL_PATH="xxxxxxx/Qwen3-32B"
SERVED_MODEL_NAME="qwen3"
DATA_PARALLEL_SIZE=1
TENSOR_PARALLEL_SIZE=8
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

export MMC_LOCAL_CONFIG_PATH=/home/memcache/mmc-local.conf

if [ "$ROLE" == "prefill" ]; then
    KV_ROLE="kv_producer"
    KV_PORT="20001"
    LOOKUP_RPC_PORT="0"
else
    KV_ROLE="kv_consumer"
    KV_PORT="20002"
    LOOKUP_RPC_PORT="1"
fi

echo "Starting vLLM on Series: $HARDWARE_SERIES, Role: $ROLE"

rm -rf /root/ascend/log/*
rm -rf ./connector.log

if [ "$HARDWARE_SERIES" == "A2" ]; then
    echo 200000 > /proc/sys/vm/nr_hugepages
    export HCCL_IF_IP=$LOCAL_IP
    export GLOO_SOCKET_IFNAME=$NIC_NAME
    export TP_SOCKET_IFNAME=$NIC_NAME
    export HCCL_SOCKET_IFNAME=$NIC_NAME

elif [ "$HARDWARE_SERIES" == "A3" ]; then
    export ACL_OP_INIT_MODE=1
else
    echo "Error: Invalid HARDWARE_SERIES. Set to 'A2' or 'A3'."
    exit 1
fi

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTHONHASHSEED=0
export HCCL_BUFFSIZE=1024
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export VLLM_USE_V1=1

KV_CONFIG='{
  "kv_connector": "MultiConnector",
  "kv_role": "'$KV_ROLE'",
  "kv_connector_extra_config": {
    "connectors": [
      {
        "kv_connector": "MooncakeConnectorV1",
        "kv_role": "'$KV_ROLE'",
        "kv_port": "'$KV_PORT'",
        "kv_connector_extra_config": {
          "prefill": {
            "dp_size": '$DATA_PARALLEL_SIZE',
            "tp_size": '$TENSOR_PARALLEL_SIZE'
          },
          "decode": {
            "dp_size": '$DATA_PARALLEL_SIZE',
            "tp_size": '$TENSOR_PARALLEL_SIZE'
          }
        }
      },
      {
        "kv_connector": "AscendStoreConnector",
        "kv_role": "'$KV_ROLE'",
        "kv_connector_extra_config": {
          "backend": "memcache",
          "lookup_rpc_port": "'$LOOKUP_RPC_PORT'"
        }
      }
    ]
  }
}'

CMD_ARGS=(
  --model "$MODEL_PATH"
  --served-model-name "$SERVED_MODEL_NAME"
  --trust-remote-code
  --enforce-eager
  --data-parallel-size "$DATA_PARALLEL_SIZE"
  --tensor-parallel-size "$TENSOR_PARALLEL_SIZE"
  --port 30050
  --max-num_seqs 20
  --max-model-len 32768
  --max-num-batched-tokens 16384
  --gpu-memory-utilization 0.9
  --kv-transfer-config "$KV_CONFIG"
)

python -m vllm.entrypoints.openai.api_server "${CMD_ARGS[@]}" > log_${ROLE}.log 2>&1

echo "vLLM started. Log file: log_${ROLE}.log"

2.启动代理服务器

参考 MooncakeStore 部署章节中的启动代理服务器。

3.运行推理

参考 MooncakeStore 部署章节中的运行推理。

PD 混部场景

1.运行混部脚本

800I A2/800T A2/800I A3/800T A3 系列

Run_pd_mix.sh:

#!/bin/bash

HARDWARE_SERIES="A2"        # A2 (800I/800T A2) or A3 (800I/800T A3)
LOCAL_IP="xx.xx.xx.xx"
NIC_NAME="xxxxxx"

MODEL_PATH="xxxxxxx/Qwen3-32B"
SERVED_MODEL_NAME="qwen3"
DATA_PARALLEL_SIZE=1
TENSOR_PARALLEL_SIZE=8
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

export MMC_LOCAL_CONFIG_PATH=/home/memcache/mmc-local.conf

echo "Starting vLLM on Series: $HARDWARE_SERIES"

rm -rf /root/ascend/log/*
rm -rf ./connector.log

if [ "$HARDWARE_SERIES" == "A2" ]; then
    echo 200000 > /proc/sys/vm/nr_hugepages
    export HCCL_IF_IP=$LOCAL_IP
    export GLOO_SOCKET_IFNAME=$NIC_NAME
    export TP_SOCKET_IFNAME=$NIC_NAME
    export HCCL_SOCKET_IFNAME=$NIC_NAME

elif [ "$HARDWARE_SERIES" == "A3" ]; then
    export ACL_OP_INIT_MODE=1
else
    echo "Error: Invalid HARDWARE_SERIES. Set to 'A2' or 'A3'."
    exit 1
fi

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTHONHASHSEED=0
export HCCL_BUFFSIZE=1024
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export VLLM_USE_V1=1

KV_CONFIG='{
  "kv_connector": "AscendStoreConnector",
  "kv_role": "kv_both",
  "kv_connector_extra_config": {
     "backend": "memcache",
     "lookup_rpc_port": "0"
  }
}'

CMD_ARGS=(
  --model "$MODEL_PATH"
  --served-model-name "$SERVED_MODEL_NAME"
  --trust-remote-code
  --enforce-eager
  --data-parallel-size "$DATA_PARALLEL_SIZE"
  --tensor-parallel-size "$TENSOR_PARALLEL_SIZE"
  --port 30050
  --max-num_seqs 20
  --max-model-len 32768
  --max-num-batched-tokens 16384
  --gpu-memory-utilization 0.9
  --kv-transfer-config "$KV_CONFIG"
)

python -m vllm.entrypoints.openai.api_server "${CMD_ARGS[@]}" > log_mix.log 2>&1

echo "vLLM started. Log file: log_mix.log"

2、运行推理

使用元戎作为 KV 池后端的示例

  • 软件要求：
    • 在所有节点上安装 openyuanrong-datasystem（必须能够导入 yr.datasystem）。

安装元戎数据系统

pip install openyuanrong-datasystem

If the prebuilt package does not match the CANN or Ascend driver version in your environment, build Yuanrong Datasystem from source in the vLLM Ascend image. Follow the official Yuanrong Datasystem build instructions: https://atomgit.com/openeuler/yuanrong-datasystem

启动 etcd

Yuanrong Datasystem uses etcd for service discovery. The following example starts a single-node etcd cluster:

ETCD_VERSION="v3.5.12"
ETCD_IP="127.0.0.1"
if [ "$(uname -m)" = "aarch64" ]; then
  ETCD_ARCH="linux-arm64"
else
  ETCD_ARCH="linux-amd64"
fi
wget https://github.com/etcd-io/etcd/releases/download/${ETCD_VERSION}/etcd-${ETCD_VERSION}-${ETCD_ARCH}.tar.gz
tar -xvf etcd-${ETCD_VERSION}-${ETCD_ARCH}.tar.gz
cd etcd-${ETCD_VERSION}-${ETCD_ARCH}
sudo cp etcd etcdctl /usr/local/bin/

etcd \
  --name etcd-single \
  --data-dir /tmp/etcd-data \
  --listen-client-urls http://0.0.0.0:2379 \
  --advertise-client-urls http://${ETCD_IP}:2379 \
  --listen-peer-urls http://0.0.0.0:2380 \
  --initial-advertise-peer-urls http://${ETCD_IP}:2380 \
  --initial-cluster etcd-single=http://${ETCD_IP}:2380 &

etcdctl --endpoints "${ETCD_IP}:2379" put key "value"
etcdctl --endpoints "${ETCD_IP}:2379" get key

For production environments, refer to the official etcd clustering documentation: https://etcd.io/docs/v3.7/op-guide/clustering/

启动数据系统工作节点

Start a Datasystem worker on each node by using dscli. The following configuration is a recommended starting point for high-throughput KV Pool workloads:

WORKER_LOG_DIR="/var/log/yuanrong/worker"
sudo mkdir -p "${WORKER_LOG_DIR}"
sudo chown "$(id -u):$(id -g)" "${WORKER_LOG_DIR}"

dscli start -w \
  --worker_address "${WORKER_IP}:31501" \
  --etcd_address "${ETCD_IP}:2379" \
  --log_dir "${WORKER_LOG_DIR}" \
  --shared_memory_size_mb 40960 \
  --arena_per_tenant 1 \
  --enable_huge_tlb true \
  --enable_fallocate false \
  --rpc_thread_num 64 \
  --oc_thread_num 64 \
  --enable_worker_worker_batch_get true \
  --sc_regular_socket_num 0 \
  --sc_stream_socket_num 0

The --worker_address value is consumed later by DS_WORKER_ADDR, so keep the host and port identical on the same node.

The tuning parameters above have the following effects:

参数 描述
log_dir Sets the Datasystem worker log directory. Create the directory and grant the worker process write permission before startup.
arena_per_tenant=1 Uses one shared-memory arena per tenant as a conservative starting point for memory and file-descriptor usage.
enable_huge_tlb=true Backs worker shared memory with HugeTLB pages. Reserve enough 2 MiB huge pages before starting the worker.
enable_fallocate=false Disables fallocate for the shared-memory file; use this setting with the HugeTLB configuration above.
rpc_thread_num=64 Sets the RPC/ZMQ service concurrency.
oc_thread_num=64 Sets the Object Cache business-thread pool size.
enable_worker_worker_batch_get=true Enables batched Object Cache reads between Datasystem workers.
sc_regular_socket_num=0, sc_stream_socket_num=0 Disables the Stream Cache service. Both values must be greater than zero to enable it; keep them at zero when KV Pool does not use Stream Cache.

For shared_memory_size_mb=40960, reserve at least 20480 2 MiB huge pages and verify that they are available before starting the worker:

grep -E "HugePages_Total|HugePages_Free|Hugepagesize" /proc/meminfo

Worker logs, including files whose base name is normally datasystem_worker, are written under the --log_dir directory. Use an absolute path so the log location does not depend on the worker process's current directory.

These thread counts are tuning starting points rather than universal defaults. Adjust them according to the available CPU cores and measured request throughput. Because -w consumes the remaining command-line arguments, place any dscli start options such as --timeout before -w.

For more parameters, refer to the dscli usage documentation on the Yuanrong Datasystem official site: https://atomgit.com/openeuler/yuanrong-datasystem

要停止工作节点：

dscli stop --worker_address "${WORKER_IP}:31501"

环境变量配置

在启动 vLLM 之前，在每个节点上设置以下环境变量：

变量 必需 默认值 描述
PYTHONHASHSEED 是 0 必须在所有节点上保持一致，以确保哈希生成的统一性。
DS_WORKER_ADDR 是 不适用 数据系统工作节点地址，格式为 <host>:<port>。此值必须与本地 dscli start --worker_address 的值匹配。
DATASYSTEM_CLIENT_LOG_DIR 否 ~/.datasystem/logs Directory for Yuanrong client SDK logs created by the vLLM process. Use a directory separate from the worker logs.
DS_ENABLE_EXCLUSIVE_CONNECTION 否 0 传递给元戎 HeteroClient.enable_exclusive_connection。当您的部署需要时，使用 1 来启用独占连接模式。
DS_ENABLE_REMOTE_H2D 否 0 传递给元戎 HeteroClient.enable_remote_h2d。仅当满足以下远程 H2D 要求后才使用 1。 
export PYTHONHASHSEED=0
export DS_WORKER_ADDR="${WORKER_IP}:31501"
export DATASYSTEM_CLIENT_LOG_DIR="/var/log/yuanrong/client"
export DS_ENABLE_EXCLUSIVE_CONNECTION=0
export DS_ENABLE_REMOTE_H2D=0

mkdir -p "${DATASYSTEM_CLIENT_LOG_DIR}"

Set DATASYSTEM_CLIENT_LOG_DIR before starting vLLM because the Yuanrong client reads it during logging initialization. Client SDK logs, whose base name is normally ds_client, are written to this directory.

远程 H2D 要求

Set DS_ENABLE_REMOTE_H2D=1 only when Remote Host-to-Device transfer is enabled and verified in the Yuanrong Datasystem deployment:

  • Reserve enough 2 MiB HugeTLB pages before starting the worker. For 40 GiB shared memory, reserve at least 20480 2 MiB huge pages.
  • Start each Datasystem worker with Remote H2D enabled. The worker start command must include --remote_h2d_device_ids, --enable_huge_tlb true, --arena_per_tenant 1, and --enable_fallocate false. Using multiple available NPU device IDs is recommended, for example "0,1,2,3,4,5,6,7" on an 8-NPU node.
dscli start -w \
  --worker_address "${WORKER_IP}:31501" \
  --etcd_address "${ETCD_IP}:2379" \
  --log_dir "/var/log/yuanrong/worker" \
  --shared_memory_size_mb 40960 \
  --arena_per_tenant 1 \
  --enable_huge_tlb true \
  --enable_fallocate false \
  --rpc_thread_num 64 \
  --oc_thread_num 64 \
  --enable_worker_worker_batch_get true \
  --sc_regular_socket_num 0 \
  --sc_stream_socket_num 0 \
  --remote_h2d_device_ids "0,1,2,3,4,5,6,7"
  • Make sure the NPU driver, firmware, and CANN toolkit required by Yuanrong Remote H2D are installed and visible to the worker process. In containers, mount the Ascend driver path, npu-smi, hccn_tool, /etc/hccn.conf, /etc/ascend_install.info, and the required /dev/davinci* devices.
  • 在启用客户端标志前验证 NPU 和 RoCE 环境：
# Check the current 2 MiB HugeTLB page size, total count, and free count.
grep -E "HugePages_Total|HugePages_Free|Hugepagesize" /proc/meminfo

# Optional: check 2 MiB HugeTLB pages on each NUMA node.
for node in /sys/devices/system/node/node*/hugepages/hugepages-2048kB; do
  echo "$node total=$(cat "$node/nr_hugepages") free=$(cat "$node/free_hugepages")"
done

# Check that NPU devices and the driver are visible to the worker environment.
npu-smi info

# Check that the NPU topology is visible.
npu-smi info -t topo

# Check optical module detection on the selected local NPU.
hccn_tool -i <local_npu_id> -optical -g

# Check RoCE physical link status. The expected link status is UP.
for i in {0..7}; do hccn_tool -i $i -link -g; done

# Check the selected NPU IP address and reachability to the remote NPU.
hccn_tool -i <local_npu_id> -ip -g
hccn_tool -i <local_npu_id> -ping -g address <remote_npu_ip>

If these checks fail, keep DS_ENABLE_REMOTE_H2D=0 and use the default Datasystem transfer path.

使用元戎后端运行 AscendStoreConnector

使用 AscendStoreConnector 并设置 backend: "yuanrong"：

python3 -m vllm.entrypoints.openai.api_server \
    --model /xxxxx/Qwen2.5-7B-Instruct \
    --port 8100 \
    --trust-remote-code \
    --enforce-eager \
    --no-enable-prefix-caching \
    --tensor-parallel-size 1 \
    --data-parallel-size 1 \
    --max-model-len 10000 \
    --block-size 128 \
    --max-num-batched-tokens 4096 \
    --kv-transfer-config \
    '{
    "kv_connector": "AscendStoreConnector",
    "kv_role": "kv_both",
    "kv_load_failure_policy": "recompute",
    "kv_connector_extra_config": {
        "lookup_rpc_port": "1",
        "backend": "yuanrong"
    }
}'

lookup_rpc_port is the RPC port used between the pooling scheduler process and the worker process. Each instance must use a unique port value.

注意事项

  • The Yuanrong backend normalizes KV keys before calling Datasystem. Supported ASCII keys up to 1024 bytes are preserved. Longer keys or keys containing unsupported characters are rewritten to a maximum of 1024 characters with a hash suffix, so do not rely on the raw key string when debugging backend storage.
  • 否 extra buffer pre-registration step is required for Yuanrong. The backend uses device pointers directly when building blob lists.

2、运行推理

常见问题

1.Mooncake 常见问题解答

1.1 put/get 操作失败

当 vLLM 报告 put 或 get 操作失败时，首先检查错误是否由 Mooncake 自身报告。

  • 如果错误由 Mooncake 报告：
    • 对于 put 失败，检查 Mooncake 日志是否包含 NO_AVAILABLE_HANDLE 或 BatchPut failed ... due to insufficient space。这通常意味着逐出后的剩余空间不足以容纳一个 BatchPut 请求。确保逐出策略留下的空间（例如，1 - eviction_ratio 所隐含的容量）可以容纳一次批量 put，或者考虑增加可用容量、增加逐出余量或减小批量大小。
    • 对于 get 失败，检查 Mooncake 日志是否包含 lease_expired_before_data_transfer_completed key=... 或返回 LEASE_EXPIRED。这意味着 KV 对象租约在数据传输完成前已过期。根据需要增加 --default_kv_lease_ttl 的 mooncake_master，并使其大于 ASCEND_CONNECT_TIMEOUT 和 ASCEND_TRANSFER_TIMEOUT。
  • 如果错误不是由 Mooncake 报告的，则很可能是 HIXL (ascend_direct) 传输层问题。收集 /root/ascend/log/debug/plog 下的 plog 文件，并检查该问题是否与已知的 HIXL 问题匹配。

For common troubleshooting and issue localization guidance for HIXL (ascend_direct), see: https://gitcode.com/cann/hixl/wiki/HIXL%E5%B8%B8%E8%A7%81%E9%97%AE%E9%A2%98%E5%AE%9A%E4%BD%8D%E6%89%8B%E5%86%8C.md

1.2 SSD 常见问题

1.2.1 SSD 卸载时的 SEGMENT_NOT_FOUND 错误

如果客户端日志显示 OffloadObjectHeartbeat failed, error code is SEGMENT_NOT_FOUND，说明 Master 已卸载该 rank 的 LOCAL_DISK 段（通常在 Ping 停止刷新 TTL 导致 client_expired 后发生）。该 rank 上的 SSD 卸载将停止，直到段被重新注册。

典型触发条件（启用 enable_cpu_binding=true 时）： Mooncake 在初始化期间启动 Ping，然后 vLLM-Ascend 的 bind_cpus() 运行 migratepages/IRQ 绑定；Ping 线程未被固定，在默认 client_ttl=10 下可能错过心跳。

缓解措施 注意事项
临时方案： 提高 Master TTL 例如 mooncake_master ... --client_ttl=120。根据您的初始化/预热窗口调整（通常 60–120 足够）。不能修复根本原因。
恢复方案： 升级 Mooncake 版本 > v0.3.11（主分支）可以在 LOCAL_DISK 后重新挂载 SEGMENT_NOT_FOUND 并重新扫描元数据。这会在清理后**恢复**；但不能防止元数据丢失期间的过期或正在进行的请求失败。
根本修复： Mooncake Ping CPU 亲和性 将存储 Ping 线程固定到释放/隔离的 CPU（Mooncake 侧更改）。可选的 vLLM-Ascend 协作，以传递每个 rank 的释放 CPU。

同时重启 Master 和 vLLM，以避免调试重启时出现过时的 segment_already_exists 状态。

1.2.2 Fabric 内存大小对齐（A3 + ASCEND_ENABLE_USE_FABRIC_MEM=1）

在启用 fabric 内存的 A3 上，每个 fabric 内存分配必须是 1 GB（1073741824 字节）的整数倍。Mooncake 不会自动向上取整。

参数 配置来源 对齐要求
global_segment_size mooncake.json 或导出 MOONCAKE_GLOBAL_SEGMENT_SIZE 每个 rank 的段大小必须对齐到 1GB（例如 "1GB"、"20GB"）。
MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES 导出 MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES（仅在 enable_ssd_offload=true 时） 必须对齐到 1GB。默认值为 1280 MB（1.25 GB），这**未**对齐，且对于长上下文 SSD 负载来说太小——请按照 调整 MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES 大小 进行设置。

local_buffer_size 中的 mooncake.json 在 fabric 内存下**不**被使用（vLLM-Ascend 向 0 传递 setup()）。

未对齐的风险： adxl MallocMem / aclrtMapMem 失败并返回 Invalid_Argument。启用 SSD 卸载时，MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES 分配失败可能导致 FileStorage 初始化期间出现段错误并中止 vLLM 启动。避免使用 "1280MB"、"512MB" 或 "1.5GB" 等值。

Fabric 内存配额： global_segment_size 和 MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES 都是**每个 rank** 独立的 fabric 内存分配。它们的大小累加后受限于通过 ASCEND_GLOBAL_RESOURCE_CONFIG 配置的 HIXL fabric 内存限制（例如 "fabric_memory.max_capacity":32，单位 GB/进程——请参阅 HIXL 文档）。每个 rank 的大致预算：

fabric_memory.max_capacity  ≥  global_segment_size + MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES  (+ headroom)

配额过低的风险： 某些 rank 在 Memory_Allocation_Failure(EL0004) 成功但 global_segment_size 分配失败时，会出现 MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES 错误。增加 fabric_memory.max_capacity、减小 global_segment_size 或 MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES，或确保节点有足够的主机内存。

示例（启用 SSD 卸载时添加到您的 vLLM 启动脚本中）：

export ASCEND_ENABLE_USE_FABRIC_MEM=1
export MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES=1073741824   # 1 GB, fabric-mem aligned

仅在 fabric 内存过低时设置 ASCEND_GLOBAL_RESOURCE_CONFIG。

# Per-rank fabric mem budget: 20 GB segment + 1 GB MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES → set max_capacity ≥ 22 (GB)
export ASCEND_GLOBAL_RESOURCE_CONFIG='{"fabric_memory.max_capacity":32}'
1.2.3 调整 MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES 大小

当 enable_ssd_offload=true 时，Mooncake 会分配一个**独立的、每个 rank 的 SSD 读写缓冲区**，其大小由 MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES 决定。此缓冲区**独立于** global_segment_size 中的 mooncake.json——增加段大小并**不能**修复由 BUFFER_OVERFLOW 过小导致的 MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES。

如果缓冲区太小，SSD 读取会在 BUFFER_OVERFLOW 期间失败并返回 error_code=-10 （FileStorage::AllocateBatch），并且当 kv_load_failure_policy=fail 时，vLLM 可能会失败。

如果在使用过程中遇到 BUFFER_OVERFLOW，请尝试增加 MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES。不要将其设置得高于 vLLM worker 日志中显示的**可用 KV 缓存内存**值：

(Worker_TP0_EP0 pid=21240) INFO 06-23 17:41:09 [worker.py:552] Available KV cache memory: XX

示例：

export MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES=10737418240   # 10 GB

仅使用**字节字面量**（10737418240）。10G / 10GB 会被忽略并回退到 1280 MB 的默认值。

注意事项 * `--max-num-batched-tokens` 仅对预填充计算进行分块；它并**不能**减少 `MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES` 所需的内存。
主机内存预算（单节点）

MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES 是**每个 rank** 分配的，与 global_segment_size 分开：

host_memory_for_mooncake ≈ TP × (global_segment_size + MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES + local_buffer_size)

确保主机上 free -h 的**可用**内存超过此总和加上 vLLM 开销。MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES 并**不需要**适合 global_segment_size 内部。

调整后验证
  1. 启动时：每个 rank 会记录 AlignedClientBufferAllocator: allocated <N> bytes，其中包含您配置的大小。
  2. 负载下：无 BUFFER_OVERFLOW / Failed to get ... keys out of ... error_codes=[-10]。
  3. 如果使用大缓冲区时仍持续失败，请检查重叠加载 (load_async)。

2.Memcache 常见问题解答

For Memcache troubleshooting, see: https://gitcode.com/Ascend/memcache/wiki/FAQ.md

3.DSv4 已知问题（临时）

For the temporary DSv4 known issue, see: https://github.com/vllm-project/vllm-ascend/issues/9975