Scheduler-Managed Paged KV Cache¶

Scheduler-managed paged KV cache stores diffusion DiT key/value tensors in Worker-owned pages. It is useful when a denoising request should keep its stable prompt/reference prefix in device memory while rewriting only the changing image span on later steps. The current HunyuanImage-3.0 integration is request-level only; it does not silently change an unsupported request to dense attention.

Paging changes KV storage and scheduling ownership; it does not change the model's causal/full attention semantics.

Choose a mode¶

Use dense_legacy when the model, attention backend, or offload combination has not been validated for paged KV. HunyuanImage-3.0 paged KV currently accepts request-level execution only; set step_execution: false.

Supported scope¶

Only the logical FLASH_ATTN backend currently advertises paged-KV support. TORCH_SDPA, CUDNN_ATTN, FLASHINFER_ATTN, SAGE, Hub, and other diffusion backends do not transparently fall back to dense execution for a paged request.

Strict Ulysses SP and two-branch CFG parallel are supported in HunyuanImage-3.0 request mode when the selected device topology has been validated. Ring SP, AllGather-KV SP, Hunyuan paged step execution, and independent public-request batching are not supported by this path. DreamZero and LingBot-World use the separate ar_diffusion_kv imported-AR-KV contract.

Configure¶

The diffusion KV fields are stage settings. The two diffusion_kv_* fields are diffusion-specific engine fields and are forwarded through engine_extras; kv_cache_memory_bytes is a standard stage field and is placed directly under the stage.

Set pipeline: hunyuan_image3_dit to select the standalone DiT pipeline, where stage 0 is the diffusion stage.

pipeline: hunyuan_image3_dit

stages:
  - stage_id: 0
    max_num_seqs: 1
    enforce_eager: true
    diffusion_attention_backend: FLASH_ATTN
    step_execution: false
    kv_cache_memory_bytes: 536870912  # 512 MiB per Worker/rank
    engine_extras:
      diffusion_kv_mode: paged_scheduler
      diffusion_kv_max_rows_per_request: 2

kv_cache_memory_bytes describes pool capacity, not live payload. Reserved memory can therefore exceed the memory currently used by active requests. The Scheduler allocates all rows belonging to one request together: a request with CFG disabled has one row, while standard two-branch CFG has two rows. CFG rows are internal branches, not independent public requests.

Paged KV is currently unquantized BF16. diffusion_kv_cache_dtype does not enable quantized paged KV, and model weight quantization has not been validated in combination with this mode. CPU offload, layerwise offload, and distributed layerwise offload (DLO) are currently untested with paged KV. Leave these options disabled for paged KV; use dense_legacy when offload is required.

Hunyuan request-mode example¶

Create a small deploy overlay (for example, hunyuan_image3_paged.yaml) with the stage settings above, then start the validated Hunyuan request-mode path:

vllm serve tencent/HunyuanImage-3.0-Instruct \
  --omni \
  --trust-remote-code \
  --port 8091 \
  --deploy-config /path/to/hunyuan_image3_paged.yaml

The base Hunyuan DiT deploy file can be reused and overridden from the command line when a separate overlay is inconvenient:

vllm serve tencent/HunyuanImage-3.0-Instruct \
  --omni \
  --trust-remote-code \
  --deploy-config vllm_omni/deploy/hunyuan_image3_dit.yaml \
  --stage-overrides \
  '{"0":{"diffusion_kv_mode":"paged_scheduler","diffusion_kv_max_rows_per_request":2,"kv_cache_memory_bytes":536870912,"step_execution":false,"diffusion_attention_backend":"FLASH_ATTN"}}'

After the server is ready, send the normal Hunyuan image-generation request:

curl -s http://localhost:8091/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "modalities": ["image"],
    "messages": [{"role": "user", "content": "A red ceramic vase on a wooden table"}],
    "extra_body": {
      "height": 1024,
      "width": 1024,
      "num_inference_steps": 20,
      "guidance_scale": 5.0,
      "seed": 1234
    }
  }' | jq -r '.choices[0].message.content[0].image_url.url' \
    | cut -d',' -f2- | base64 -d > hunyuan_image3_paged.png

The same configuration is shown in the HunyuanImage-3.0 recipe.

Confirm from startup and worker logs that the resolved backend is FLASH_ATTN and that the paged native path is active. A paged request must fail explicitly if its Worker metadata or native backend is unavailable; it must not silently switch to dense attention.

Parallelism¶

Hunyuan request-mode paged KV can be combined with tensor parallelism, strict Ulysses SP, or two-way CFG parallel when the selected topology has been validated. Configure the topology in the stage's existing parallel_config:

max_num_seqs controls public request concurrency and is independent of the internal CFG row limit. Keep max_num_seqs: 1 for the current Hunyuan paged path; increasing it does not enable independent request batching.

Troubleshooting¶

• paged_scheduler requires diffusion_kv_max_rows_per_request: add a positive stage value and include all CFG rows.
• Unsupported backend error: select FLASH_ATTN; the other attention backends currently do not implement the paged-KV capability.
• Step-mode rejection: remove step_execution: true and use request-level execution, or select dense_legacy for step execution.
• Imported AR-KV rejection: paged_scheduler does not consume the ar_diffusion_kv contract used by DreamZero or LingBot-World.
• Missing image output: include "modalities": ["image"] when using the chat completions endpoint, and decode the returned data URI as shown above.

For the ownership and lifecycle contract, see the Scheduler-managed paged KV design.