Welcome to vLLM

Easy, fast, and cheap LLM serving for everyone

Star Watch Fork

vLLM is a fast and easy-to-use library for LLM inference and serving.

Originally developed in the Sky Computing Lab at UC Berkeley, vLLM has evolved into a community-driven project with contributions from both academia and industry.

vLLM is fast with:

• State-of-the-art serving throughput

• Efficient management of attention key and value memory with PagedAttention

• Continuous batching of incoming requests

• Fast model execution with CUDA/HIP graph

• Quantization: GPTQ, AWQ, INT4, INT8, and FP8

• Optimized CUDA kernels, including integration with FlashAttention and FlashInfer.

• Speculative decoding

• Chunked prefill

vLLM is flexible and easy to use with:

• Seamless integration with popular HuggingFace models

• High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more

• Tensor parallelism and pipeline parallelism support for distributed inference

• Streaming outputs

• OpenAI-compatible API server

• Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs, Gaudi® accelerators and GPUs, IBM Power CPUs, TPU, and AWS Trainium and Inferentia Accelerators.

• Prefix caching support

• Multi-lora support

For more information, check out the following:

• vLLM announcing blog post (intro to PagedAttention)

• vLLM paper (SOSP 2023)

• How continuous batching enables 23x throughput in LLM inference while reducing p50 latency by Cade Daniel et al.

• vLLM Meetups

Documentation

Getting Started

• Installation
• Quickstart
• Examples
• Troubleshooting
• Frequently Asked Questions
• vLLM V1 User Guide

Models

• Supported Models
• Generative Models
• Pooling Models
• Built-in Extensions

Features

• Quantization
• Multimodal Inputs
• Prompt Embedding Inputs
• LoRA Adapters
• Tool Calling
• Reasoning Outputs
• Structured Outputs
• Automatic Prefix Caching
• Disaggregated Prefilling (experimental)
• Speculative Decoding
• Compatibility Matrix

Training

• Transformers Reinforcement Learning
• Reinforcement Learning from Human Feedback

Inference and Serving

• Offline Inference
• OpenAI-Compatible Server
• Server Arguments
• Distributed Inference and Serving
• Production Metrics
• Engine Arguments
• Environment Variables
• Usage Stats Collection
• External Integrations

Deployment

• Security Guide
• Using Docker
• Using Kubernetes
• Using Nginx
• Using other frameworks
• External Integrations

Performance

• Optimization and Tuning
• Benchmark Suites

Design Documents

• Architecture Overview
  • Entrypoints
  • LLM Engine
  • Worker
  • Model Runner
  • Model
  • Class Hierarchy
• Integration with HuggingFace
• vLLM’s Plugin System
  • How Plugins Work in vLLM
  • How vLLM Discovers Plugins
  • Types of supported plugins
  • Guidelines for Writing Plugins
  • Compatibility Guarantee
• vLLM Paged Attention
  • Inputs
  • Concepts
  • Query
  • Key
  • QK
  • Softmax
  • Value
  • LV
  • Output
• Multi-Modal Data Processing
  • Prompt Update Detection
  • Tokenized Prompt Inputs
  • Processor Output Caching
• Automatic Prefix Caching
  • Generalized Caching Policy
• Python Multiprocessing
  • Debugging
  • Introduction
  • Multiprocessing Methods
  • Compatibility with Dependencies
  • Current State (v0)
  • Prior State in v1
  • Alternatives Considered
  • Future Work

V1 Design Documents

• vLLM’s torch.compile integration
  • Compilation Cache
  • Python Code Compilation
  • Computation Graph Processing
  • Computation Graph Compilation
  • Cudagraph Capture
• Automatic Prefix Caching
  • Data Structure
  • Operations
  • Example
• Metrics
  • Objectives
  • Background
  • v1 Design
  • Deprecated Metrics
  • Future Work
  • Tracing - OpenTelemetry

Developer Guide

• Contributing to vLLM
  • Job Board
  • License
  • Developing
  • Testing
  • Issues
  • Pull Requests & Code Reviews
  • Thank You
• Deprecation Policy
  • Overview
  • Deprecation Pipeline
  • Example Timeline
  • Important Guidelines
  • Final Notes
• Profiling vLLM
  • Profile with PyTorch Profiler
  • Profile with NVIDIA Nsight Systems
  • Profiling vLLM Python Code
• Dockerfile
• Adding a New Model
  • Implementing a Basic Model
  • Registering a Model to vLLM
  • Writing Unit Tests
  • Multi-Modal Support
• Vulnerability Management
  • Reporting Vulnerabilities
  • Vulnerability Management Team
  • Slack Discussion
  • Vulnerability Disclosure

API Reference

• Summary
  • Configuration
  • Offline Inference
  • vLLM Engines
  • Inference Parameters
  • Multi-Modality
  • Model Development
• vllm
  • Subpackages
  • Submodules

Community

• vLLM Blog
• vLLM Meetups
• Sponsors

Indices and tables

• Index

• Module Index