Skip to content

Decode Context Parallel Guide

Overview

Decode Context Parallel (DCP) shards the KV cache along the sequence dimension across devices in a Tensor Parallel (TP) group. It removes redundant KV-cache copies and can increase the batch size available for long-context decoding.

Prefill Context Parallel is not supported by vLLM Ascend. The upstream prefill_context_parallel_size option must remain at its default value of 1.

DSA-CP is a separate sparse-attention optimization controlled by additional_config.enable_dsa_cp. See Additional Configuration for its configuration and model requirements.

Supported Scenarios

DCP supports eager and graph execution, prefix caching, chunked prefill, speculative decoding, P/D disaggregation, and MLAPO on the model and hardware combinations documented by vLLM Ascend. The following table shows whether each feature can be combined with DCP across devices and attention backends:

Device Attention Backend Chunked Prefill + DCP Prefix Caching + DCP Graph Mode + DCP P/D Disaggregation + DCP MLAPO + DCP Speculative Decoding + DCP
Ascend A2/A3 MLA/GQA 🟢 Supported 🟢 Supported 🟢 Supported 🟢 Supported 🟢 Supported (MLA)
— Not applicable (GQA)		 🟢 P/D disaggregation
🔴 PD-mixed deployment
Ascend A2/A3 SFA 🟢 Supported 🟢 Supported 🟢 Supported 🟢 Supported 🟢 Supported 🟢 Supported
Ascend 950 MLA/GQA 🔵 Experimental 🔵 Experimental 🔵 Experimental 🔵 Experimental 🔵 Experimental (MLA)
— Not applicable (GQA)		 🔵 P/D disaggregation
🔴 PD-mixed deployment
Ascend 950 SFA 🔴 Not supported 🔴 Not supported 🔴 Not supported 🔴 Not supported 🔴 Not supported 🔴 Not supported
  • 🟢 Supported: Combining the feature with DCP is supported.
  • 🔵 Experimental: Combining the feature with DCP is experimentally supported; interfaces and functionality may change.
  • 🔴 Not supported: Combining the feature with DCP is not supported.
  • Not applicable: The feature does not apply to this attention backend.

Usage

Offline example:

from vllm import LLM, SamplingParams

prompts = ["The future of AI is"]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(
    model="deepseek-ai/DeepSeek-V2-Lite",
    tensor_parallel_size=2,
    decode_context_parallel_size=2,
)
outputs = llm.generate(prompts, sampling_params)

Online example:

vllm serve deepseek-ai/DeepSeek-V2-Lite \
    --tensor-parallel-size 2 \
    --decode-context-parallel-size 2

DCP reuses the TP devices and does not increase the world size.

Constraints

  • For an MLA model such as DeepSeek-R1:
    • tensor_parallel_size >= decode_context_parallel_size
    • tensor_parallel_size % decode_context_parallel_size == 0
  • For a GQA model such as Qwen3-235B:
    • (tensor_parallel_size // num_key_value_heads) >= decode_context_parallel_size
    • (tensor_parallel_size // num_key_value_heads) % decode_context_parallel_size == 0

  • In a KV-cache transfer scenario such as KV pooling or P/D disaggregation, set cp_kv_cache_interleave_size to the KV-cache block_size (default: 128):

    vllm serve deepseek-ai/DeepSeek-V2-Lite \
    --tensor-parallel-size 2 \
    --decode-context-parallel-size 2 \
    --cp-kv-cache-interleave-size 128 \
    --kv-transfer-config '{...}'

For implementation details, see the Decode Context Parallel design document.