Welcome to vLLM!

Easy, fast, and cheap LLM serving for everyone

Star Watch Fork

vLLM is a fast and easy-to-use library for LLM inference and serving.

vLLM is fast with:

• State-of-the-art serving throughput

• Efficient management of attention key and value memory with PagedAttention

• Continuous batching of incoming requests

• Fast model execution with CUDA/HIP graph

• Quantization: GPTQ, AWQ, INT4, INT8, and FP8

• Optimized CUDA kernels, including integration with FlashAttention and FlashInfer.

• Speculative decoding

• Chunked prefill

vLLM is flexible and easy to use with:

• Seamless integration with popular HuggingFace models

• High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more

• Tensor parallelism and pipeline parallelism support for distributed inference

• Streaming outputs

• OpenAI-compatible API server

• Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, TPU, and AWS Trainium and Inferentia Accelerators.

• Prefix caching support

• Multi-lora support

For more information, check out the following:

• vLLM announcing blog post (intro to PagedAttention)

• vLLM paper (SOSP 2023)

• How continuous batching enables 23x throughput in LLM inference while reducing p50 latency by Cade Daniel et al.

• vLLM Meetups.

Documentation

Getting Started

• Installation
• Installation with ROCm
• Installation with OpenVINO
• Installation with CPU
• Installation with Neuron
• Installation with TPU
• Installation with XPU
• Quickstart
• Debugging Tips
• Examples

Serving

• OpenAI Compatible Server
• Deploying with Docker
• Distributed Inference and Serving
• Production Metrics
• Environment Variables
• Usage Stats Collection
• Integrations
• Loading Models with CoreWeave’s Tensorizer
• Frequently Asked Questions

Models

• Supported Models
• Model Support Policy
• Adding a New Model
• Enabling Multimodal Inputs
• Engine Arguments
• Using LoRA adapters
• Using VLMs
• Speculative decoding in vLLM
• Performance and Tuning

Quantization

• Supported Hardware for Quantization Kernels
• AutoAWQ
• BitsAndBytes
• GGUF
• INT8 W8A8
• FP8 W8A8
• FP8 E5M2 KV Cache
• FP8 E4M3 KV Cache

Automatic Prefix Caching

• Introduction
• Implementation
• Generalized Caching Policy

Performance benchmarks

• Benchmark suites of vLLM

Developer Documentation

• Sampling Parameters
  • SamplingParams
• Offline Inference
  • LLM Class
  • LLM Inputs
• vLLM Engine
  • LLMEngine
  • AsyncLLMEngine
• vLLM Paged Attention
  • Inputs
  • Concepts
  • Query
  • Key
  • QK
  • Softmax
  • Value
  • LV
  • Output
• Input Processing
  • Guides
  • Module Contents
• Multi-Modality
  • Guides
  • Module Contents
• Dockerfile
• Profiling vLLM
• Example commands and usage:
  • Offline Inference:
  • OpenAI Server:

Community

• vLLM Meetups
• Sponsors

Indices and tables

• Index

• Module Index