Skip to content

Mooncake Layerwise KV Cache Pooling

Feature Introduction

Mooncake layerwise KV cache pooling transfers KV caches between Ascend NPUs and a shared Mooncake Store one layer at a time. It uses AscendStoreConnector with backend="mooncake" and use_layerwise=true to reuse cached prefixes across requests and vLLM instances.

Layerwise transfer allows KV loading and saving to overlap attention computation. It also stores the layers of a cache block in one remote object, reducing metadata overhead compared with creating a separate object for every layer.

For Mooncake service deployment and general pool configuration, see KV Cache Pool (Ascend Store). For models with multiple cache groups, see Mooncake Layerwise Hybrid Attention.

Usage Scenarios

  • Repeated requests with long shared prefixes, such as system prompts or document contexts.
  • Multiple serving instances that share prefixes through the same Mooncake Store.
  • Prefill workloads where KV transfer can overlap per-layer attention computation.

Constraints and Limitations

Category Constraint
Software Requires a Mooncake NPU client with session and range transfer APIs; use Mooncake 0.3.14 or later. Required interfaces are checked at startup.
Deployment Use the corresponding producer/consumer role on each P/D node. Decode pool reads and writes are controlled separately by consumer_is_to_load and consumer_is_to_put.
Parallelism Instances sharing layerwise objects must use matching TP, PP, PCP, and DCP sizes and the same PP layer partition. PCP and DCP cannot be enabled together. TP mismatch is not supported.
PP Uneven layer partitions are supported, but each stage must own layers in every cache group. Empty stage-local cache groups are not supported.
Cache layout Readers and writers must agree on model weights, cache format, block sizes, and transfer mode. Use separate pool namespaces for incompatible configurations.
Buffer reuse Mooncake layerwise transfer does not enable cross-layer reuse of NPU cache buffers.
KVPP Experimental for non-hybrid MLA/SFA models with load_async=false; cannot be combined with Mooncake layerwise PCP. See KVPP for its support scope.

Hybrid models have additional cache-state and execution-mode requirements in the hybrid attention guide.

How It Works

Block Storage and Layerwise Transfer

For each cache block and storing rank, Mooncake keeps one object containing the cache entries of all layers owned by that rank and pipeline stage:

Remote object for one cache block
├── Layer 0: cache entries
├── Layer 1: cache entries
└── Layer 2: cache entries

Saving a layer writes only its byte ranges into this object. Loading a layer reads those ranges into the local NPU cache before the layer consumes them. Offsets follow the actual cache layout, including multiple entries in one physical layer and entries with different sizes.

For a single cache group and pipeline stage, B blocks, L layers, and R storing ranks require approximately B × R objects, compared with B × L × R objects in a per-layer object layout. This reduces metadata and lookup work; the total KV payload remains the same.

Computation and Transfer Overlap

The connector prefetches future layers while the current layer computes attention. layerwise_prefetch_layers controls how far ahead loads are submitted. The current layer waits for its own load to finish before using restored KV. Saves start after the corresponding cache updates are ready.

Transfers run asynchronously, with device events protecting cache access and queue backpressure limiting outstanding work. Transfer may continue into later computation or communication. The amount of overlap depends on attention time, cache size, and available bandwidth.

Complete Snapshots and Prefix Reuse

A remote object becomes reusable only after all of its layer ranges have been written and committed. A prefix hit requires readable objects from every required storing rank and pipeline stage. Partially written objects are not reported as reusable cache.

During chunked prefill, later chunks can restore the committed prefix layer by layer. Read sessions keep shared objects available while they are in use and are released when the reads finish. Failed writes are revoked so that later requests cannot consume an incomplete snapshot.

Usage

Prepare Mooncake

Install the Mooncake NPU client, start mooncake_master, and configure the Ascend network and mooncake.json following the Mooncake Backend section of KV Cache Pool (Ascend Store). Select a client version that meets the layerwise requirement above rather than the older non-layerwise installation example.

Set the configuration path and use the same hash seed on all sharing instances:

export MOONCAKE_CONFIG_PATH=/path/to/mooncake.json
export PYTHONHASHSEED=0

Sharing instances must connect to the same Store and tenant, with compatible model and cache configurations. For independently launched services on the same host, assign distinct lookup_rpc_port identifiers and HTTP ports.

Configure Prefill and Decode Nodes

Follow the PD Disaggregation Scenario in the Mooncake deployment guide for node startup, P/D transfer, and proxy configuration. Configure the AscendStoreConnector on each node as follows.

Prefill (P) node:

{
    "kv_connector": "AscendStoreConnector",
    "kv_role": "kv_producer",
    "kv_load_failure_policy": "recompute",
    "kv_connector_extra_config": {
        "lookup_rpc_port": "0",
        "backend": "mooncake",
        "use_layerwise": true,
        "layerwise_prefetch_layers": 2
    }
}

Decode (D) node:

{
    "kv_connector": "AscendStoreConnector",
    "kv_role": "kv_consumer",
    "kv_load_failure_policy": "recompute",
    "kv_connector_extra_config": {
        "lookup_rpc_port": "0",
        "backend": "mooncake",
        "use_layerwise": true,
        "layerwise_prefetch_layers": 2
    }
}

These examples show the Store connector configuration. In the deployment guide's MultiConnector, update the AscendStoreConnector entry and set kv_load_failure_policy on the top-level kv-transfer-config.

The Prefill node saves and reuses prefixes through the pool. Decode pool reads and writes are disabled by default; P/D KV transfer is handled by the other connector in MultiConnector. To enable Decode pool loading, add "consumer_is_to_load": true to its kv_connector_extra_config. Setting use_layerwise=true alone does not enable consumer pool access.

The recompute policy above applies to non-hybrid models. For hybrid models, use fail as shown in the hybrid attention guide. Select model, parallelism, token-budget, and execution-mode options according to the corresponding model deployment guide.

Decode Context Parallelism (DCP)

Mooncake block-key layerwise pooling supports standalone DCP (prefill_context_parallel_size=1, decode_context_parallel_size>1) with the same TP/PP/DCP topology and PP layer partition on both sides. See the Context Parallel Guide for the topology constraints. Keys append @dcp_size:N@dcp_rank:R; DCP=1 keys are unchanged. A block is a hit only after every PP stage and DCP rank committed it.

PCP and DCP cannot be combined. Every cache group must be DCP-sharded; sliding-window, recurrent/align-state, circular-buffer and other non-DCP-scaled groups must keep DCP disabled. The smallest cacheable block size must divide the others; --prefix-match-unit must be unset with DCP. GQA/MQA models do not support DCP.

In P/D disaggregation both sides must use the same DCP size; set prefill_dcp_size / decode_dcp_size in kv_connector_extra_config so the worker can verify it. See the hybrid guide.

Use Cached Prefixes

Send inference requests through the normal OpenAI-compatible API:

curl 'http://<proxy-host>:<proxy-port>/v1/completions' \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<served-model-name>",
    "prompt": "Explain how KV cache reuse reduces repeated prefill computation.",
    "max_tokens": 64,
    "temperature": 0
  }'

The response contains generated text in choices. To reuse the pool, submit requests with the same tokenized prefix long enough to cover complete cache blocks. The first request populates the Store; later requests can load committed blocks and compute the remaining tokens. A final partial block is recomputed.

To check cross-instance reuse, send a long shared prefix to one instance, wait for its request to finish, then send it to another instance connected to the same pool. Check external prefix-cache hits in the serving metrics or logs. Requests repeated on one instance may instead hit its local prefix cache.

Configuration Parameters

Set these options in kv_transfer_config.kv_connector_extra_config.

Parameter Type Default Description
backend string "mooncake" Selects the Mooncake Store backend.
use_layerwise bool false Set to true to enable layerwise save and load.
load_async bool false Keep false for layerwise loading. Future-layer prefetch is managed by the layerwise transfer pipeline.
layerwise_prefetch_layers int 2 Positive prefetch distance in layers. Larger values submit loads earlier.
layerwise_max_transfer_blocks int 0 Maximum key/block rows per range transfer batch. Also batches session operations and lookup work. 0 adds no block limit.
layerwise_max_transfer_bytes int 0 Maximum size in bytes of an individual contiguous transfer segment. Larger segments are split. This is not a limit on total bytes per API call; 0 adds no segment-size limit.

Transfer batch limits are optional and are omitted from the examples. For roles, placement, tenancy, and other pool options, see Ascend Store configuration.

Tuning Suggestions

  • Start with the default prefetch distance of 2. Increase it if layers wait for remote loads; reduce it if early transfers contend with other communication.
  • Use block and segment limits to control transfer batch size for long prompts. Smaller batches reduce per-call work but increase the number of calls.
  • Evaluate time to first token and throughput with representative prompt lengths and prefix hit rates. Benefits depend on how much transfer the attention computation can hide and on contention with HCCL or expert-parallel traffic.