.. _tutorial_disagg:

Disaggregated Prefill
=====================

Introduction
------------------------

This tutorial explains how to run the disaggregated prefill system, which splits the model execution into prefill and decode phases across different servers. This approach can improve throughput and resource utilization by separating the initial processing (prefill) from the token generation (decode) phases.

Prerequisites
-------------------------

* A Kubernetes cluster with GPU support and NVLink enabled
* NVIDIA GPUs available (at least 2 GPUs recommended)
* ``kubectl`` configured to talk to your cluster
* Helm installed and initialized locally
* Completion of the following setup tutorials:

  * `00-install-kubernetes-env.md <https://github.com/vllm-project/production-stack/blob/main/tutorials/00-a-install-multinode-kubernetes-env.md>`__
  * `01-minimal-helm-installation.md <https://github.com/vllm-project/production-stack/blob/main/tutorials/01-b-minimal-helm-installation.md>`__

Kubernetes Deployment
-------------------------------

For production environments, you can deploy the disaggregated prefill system using Kubernetes and Helm. This approach provides better scalability, resource management, and high availability.

Step 1: Create Configuration File
++++++++++++++++++++++++++++++++++

Create a configuration file ``values-16-disagg-prefill.yaml`` with the following content:

.. code-block:: yaml

    # Unified configuration for disaggregated prefill setup
    # Unified configuration for disaggregated prefill setup
    servingEngineSpec:
      enableEngine: true
      runtimeClassName: ""
      containerPort: 8000
      modelSpec:
        # Prefill node configuration
        - name: "llama-prefill"
          repository: "lmcache/vllm-openai"
          tag: "2025-05-27-v1"
          modelURL: "meta-llama/Llama-3.1-8B-Instruct"
          replicaCount: 1
          requestCPU: 8
          requestMemory: "30Gi"
          # requestGPU: 1
          pvcStorage: "50Gi"
          vllmConfig:
            enablePrefixCaching: true
            maxModelLen: 32000
            v1: 1
            gpuMemoryUtilization: 0.6
          lmcacheConfig:
            cudaVisibleDevices: "0"
            enabled: true
            kvRole: "kv_producer"
            enableNixl: true
            nixlRole: "sender"
            nixlPeerHost: "vllm-llama-decode-engine-service"
            nixlPeerPort: "55555"
            nixlBufferSize: "1073741824"  # 1GB
            nixlBufferDevice: "cuda"
            nixlEnableGc: true
            enablePD: true
            cpuOffloadingBufferSize: 0
          hf_token: <your-hf-token>
          labels:
            model: "llama-prefill"
        # Decode node configuration
        - name: "llama-decode"
          repository: "lmcache/vllm-openai"
          tag: "2025-05-27-v1"
          modelURL: "meta-llama/Llama-3.1-8B-Instruct"
          replicaCount: 1
          requestCPU: 8
          requestMemory: "30Gi"
          # requestGPU: 1
          pvcStorage: "50Gi"
          vllmConfig:
            enablePrefixCaching: true
            maxModelLen: 32000
            v1: 1
          lmcacheConfig:
            cudaVisibleDevices: "1"
            enabled: true
            kvRole: "kv_consumer"  # Set decode node as consumer
            enableNixl: true
            nixlRole: "receiver"
            nixlPeerHost: "0.0.0.0"
            nixlPeerPort: "55555"
            nixlBufferSize: "1073741824"  # 1GB
            nixlBufferDevice: "cuda"
            nixlEnableGc: true
            enablePD: true
          hf_token: <your-hf-token>
          labels:
            model: "llama-decode"
    routerSpec:
      enableRouter: true
      repository: "lmcache/lmstack-router"
      tag: "pd"
      replicaCount: 1
      containerPort: 8000
      servicePort: 80
      routingLogic: "disaggregated_prefill"
      engineScrapeInterval: 15
      requestStatsWindow: 60
      enablePD: true
      resources:
        requests:
          cpu: "4"
          memory: "16G"
        limits:
          cpu: "4"
          memory: "32G"
      labels:
        environment: "router"
        release: "router"
      extraArgs:
        - "--prefill-model-labels"
        - "llama-prefill"
        - "--decode-model-labels"
        - "llama-decode"


Step 2: Deploy Using Helm
++++++++++++++++++++++++++++++++++

Install the deployment using Helm with the configuration file:

.. code-block:: bash

    helm install pd helm/ -f tutorials/assets/values-16-disagg-prefill.yaml

This will deploy:

* A prefill server with the specified configuration
* A decode server with the specified configuration
* A router to coordinate between them

The configuration includes:

* Resource requests and limits for each component
* NIXL communication settings for LMCache
* Model configurations
* Router settings for disaggregated prefill

Step 3: Verify Deployment
++++++++++++++++++++++++++++++++++

Check the status of your deployment:

.. code-block:: bash

    kubectl get pods
    kubectl get services

You should see pods for:

* The prefill server
* The decode server
* The router

Step 4: Access the Service
++++++++++++++++++++++++++++++++++

First do port forwarding to access the service:

.. code-block:: bash

    kubectl port-forward svc/pd-router-service 30080:80

And then send a request to the router by:

.. code-block:: bash

    curl http://localhost:30080/v1/completions \
        -H "Content-Type: application/json" \
        -d '{
            "model": "meta-llama/Llama-3.1-8B-Instruct",
            "prompt": "Your prompt here",
            "max_tokens": 100
        }'

You should see logs from LMCache like the following on the decoder instance's side:

.. code-block:: console

    [2025-05-26 20:12:21,913] LMCache DEBUG: Scheduled to load 6 tokens for request cmpl-058cf35e022a479f849a60daefbade9e-0 (vllm_v1_adapter.py:299:lmcache.integration.vllm.vllm_v1_adapter)
    [2025-05-26 20:12:21,915] LMCache DEBUG: Retrieved 6 out of 6 out of total 6 tokens (cache_engine.py:330:lmcache.experimental.cache_engine)