Skip to main content

Ctrl+K

Getting Started

Installation
Installation with ROCm
Installation with OpenVINO
Installation with CPU
Installation with Neuron
Installation with TPU
Installation with XPU
Quickstart
Debugging Tips
Examples

Serving

OpenAI Compatible Server
Deploying with Docker
Deploying with Kubernetes
Distributed Inference and Serving
Production Metrics
Environment Variables
Usage Stats Collection
Integrations
Loading Models with CoreWeave’s Tensorizer
Compatibility Matrix
Frequently Asked Questions

Models

Supported Models
Adding a New Model
Enabling Multimodal Inputs
Engine Arguments
Using LoRA adapters
Using VLMs
Speculative decoding in vLLM
Performance and Tuning

Quantization

Supported Hardware for Quantization Kernels
AutoAWQ
BitsAndBytes
GGUF
INT8 W8A8
FP8 W8A8
FP8 E5M2 KV Cache
FP8 E4M3 KV Cache

Automatic Prefix Caching

Introduction
Implementation

Performance benchmarks

Benchmark suites of vLLM

Developer Documentation

Sampling Parameters
Offline Inference
- LLM Class
- LLM Inputs
vLLM Engine
- LLMEngine
- AsyncLLMEngine
vLLM Paged Attention
Input Processing
- Input Processing Pipeline
Multi-Modality
- Adding a Multimodal Plugin
Dockerfile
Profiling vLLM

Community

vLLM Meetups
Sponsors

Search

Ctrl+K

By the vLLM Team

© Copyright 2024, vLLM Team.