Gateway API Inference Extension

Gateway API Inference Extension#

This guide deploys agentgateway as an inference gateway for a pool of vLLM model servers. The Kubernetes Gateway API Inference Extension defines the InferencePool contract, and the llm-d Router Endpoint Picker (EPP) selects a model-server pod for each request.

The request path is:

Client -> agentgateway -> HTTPRoute -> InferencePool
                                      -> llm-d Router EPP -> vLLM pod

The example pins Gateway API v1.6.0, Inference Extension v1.5.0, agentgateway v1.4.1, and llm-d Router v0.9.0.

Prerequisites#

  • A Kubernetes cluster with at least two GPU nodes

  • kubectl configured for the cluster

  • Helm 3

  • A Hugging Face token with access to meta-llama/Llama-3.2-1B-Instruct

Create the model credential#

kubectl create secret generic hf-token \
  --from-literal=token='<YOUR_HF_TOKEN>'

Install the APIs and agentgateway#

Install the pinned Gateway API and Inference Extension CRDs:

export GATEWAY_API_VERSION=v1.6.0
export INFERENCE_EXTENSION_VERSION=v1.5.0

kubectl apply --server-side -f \
  "https://github.com/kubernetes-sigs/gateway-api/releases/download/${GATEWAY_API_VERSION}/standard-install.yaml"

kubectl apply --server-side -f \
  "https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/${INFERENCE_EXTENSION_VERSION}/manifests.yaml"

Install agentgateway with Inference Extension support:

export AGENTGATEWAY_VERSION=v1.4.1

helm upgrade -i --create-namespace \
  --namespace agentgateway-system \
  --version "${AGENTGATEWAY_VERSION}" \
  agentgateway-crds oci://cr.agentgateway.dev/charts/agentgateway-crds

helm upgrade -i \
  --namespace agentgateway-system \
  --version "${AGENTGATEWAY_VERSION}" \
  --set inferenceExtension.enabled=true \
  agentgateway oci://cr.agentgateway.dev/charts/agentgateway

Deploy vLLM#

The example runs two replicas of meta-llama/Llama-3.2-1B-Instruct and labels them with app: vllm-llama3-1b-instruct. The InferencePool uses that label to discover model servers.

kubectl apply -f \
  src/gateway_inference_extension/configs/vllm/gpu-deployment.yaml

kubectl rollout status deployment/vllm-llama3-1b-instruct \
  --timeout=15m

Adjust GPU resources, replica count, and model-server arguments before using the example in production.

Create the agentgateway Gateway#

kubectl apply -f \
  src/gateway_inference_extension/configs/gateway/agentgateway/gateway.yaml

kubectl wait --for=condition=Programmed --timeout=120s \
  gateway/inference-gateway

The Gateway selects the agentgateway GatewayClass:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: inference-gateway
spec:
  gatewayClassName: agentgateway
  listeners:
    - name: http
      port: 80
      protocol: HTTP

Install the llm-d Router#

The llm-d Router Gateway chart creates the current inference.networking.k8s.io/v1 InferencePool, EPP Deployment and Service, RBAC, and an HTTPRoute attached to inference-gateway.

export LLM_D_ROUTER_VERSION=v0.9.0

helm upgrade -i vllm-llama3-1b-instruct \
  oci://ghcr.io/llm-d/charts/llm-d-router-gateway \
  --version "${LLM_D_ROUTER_VERSION}" \
  -f src/gateway_inference_extension/configs/llm-d-router-values.yaml

kubectl rollout status deployment/vllm-llama3-1b-instruct-epp \
  --timeout=120s

The values set provider.name=none because agentgateway processes the chart-created HTTPRoute and InferencePool directly. The values also select the vLLM pods through router.modelServers.matchLabels.

Verify the deployment#

kubectl get gateway inference-gateway
kubectl get httproute vllm-llama3-1b-instruct
kubectl get inferencepool vllm-llama3-1b-instruct
kubectl get deployment vllm-llama3-1b-instruct-epp

For troubleshooting, inspect status conditions and component logs:

kubectl describe gateway inference-gateway
kubectl describe httproute vllm-llama3-1b-instruct
kubectl logs deployment/vllm-llama3-1b-instruct-epp
kubectl logs -n agentgateway-system deployment/agentgateway

Send an inference request#

Port-forward the Gateway Service:

kubectl port-forward service/inference-gateway 8080:80

In a separate terminal, send an OpenAI-compatible completion request:

curl -i http://localhost:8080/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "meta-llama/Llama-3.2-1B-Instruct",
    "prompt": "Write as if you were a critic: San Francisco",
    "max_tokens": 100,
    "temperature": 0.5
  }'

Use agentgateway AI policies#

The default route points directly to the InferencePool. This is appropriate when only endpoint selection is needed. To use token-based rate limiting, guardrails, transformations, or LLM observability, route to an AgentgatewayBackend whose custom provider references the InferencePool. See the agentgateway inference routing guide.

Uninstall#

./src/gateway_inference_extension/delete.sh

The cleanup script retains shared agentgateway, Gateway API, and Inference Extension CRDs. On a dedicated test cluster, remove them explicitly:

DELETE_SHARED_CRDS=true ./src/gateway_inference_extension/delete.sh

Migration from kgateway#

Kgateway’s inference-extension operation without agentgateway was deprecated in kgateway 2.1 and is unsupported in 2.2. The Production Stack inference example no longer includes a kgateway configuration or locally patched EPP.

For an existing installation:

  1. Install agentgateway and change gatewayClassName from kgateway to agentgateway.

  2. Upgrade InferencePool from inference.networking.x-k8s.io/v1alpha2 to inference.networking.k8s.io/v1.

  3. Replace targetPortNumber with targetPorts and extensionRef with endpointPickerRef.

  4. Remove obsolete InferenceModel resources.

  5. Replace the custom EPP Deployment, Service, and RBAC with the llm-d Router Gateway chart.

  6. Remove the kgateway releases only after the agentgateway route reports Accepted=True and an inference request succeeds.

This deprecation does not apply to kgateway used as a generic Gateway API ingress controller.