Welcome to vLLM!

Easy, fast, and cheap LLM serving for everyone

Star Watch Fork

vLLM is a fast and easy-to-use library for LLM inference and serving.

vLLM is fast with:

• State-of-the-art serving throughput

• Efficient management of attention key and value memory with PagedAttention

• Continuous batching of incoming requests

• Optimized CUDA kernels

vLLM is flexible and easy to use with:

• Seamless integration with popular HuggingFace models

• High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more

• Tensor parallelism support for distributed inference

• Streaming outputs

• OpenAI-compatible API server

For more information, check out the following:

• vLLM announcing blog post (intro to PagedAttention)

• vLLM paper (SOSP 2023)

• How continuous batching enables 23x throughput in LLM inference while reducing p50 latency by Cade Daniel et al.

Documentation

Getting Started

• Installation
• Quickstart

Serving

• Distributed Inference and Serving
• Running on clouds with SkyPilot
• Deploying with NVIDIA Triton
• Deploying with Docker
• Serving with Langchain
• Production Metrics

Models

• Supported Models
• Adding a New Model
• Engine Arguments

Quantization

• AutoAWQ