Skip to main content

Ctrl+K

Getting Started

Installation
Installation with ROCm
Installation with OpenVINO
Installation with CPU
Installation with Intel® Gaudi® AI Accelerators
Installation with Neuron
Installation with TPU
Installation with XPU
Quickstart
Debugging Tips
Examples

Serving

OpenAI Compatible Server
Deploying with Docker
Deploying with Kubernetes
Deploying with Nginx Loadbalancer
Distributed Inference and Serving
Production Metrics
Environment Variables
Usage Stats Collection
Integrations
Loading Models with CoreWeave’s Tensorizer
Compatibility Matrix
Frequently Asked Questions

Models

Supported Models
Adding a New Model
Enabling Multimodal Inputs
Engine Arguments
Using LoRA adapters
Using VLMs
Speculative decoding in vLLM
Performance and Tuning

Quantization

Supported Hardware for Quantization Kernels
AutoAWQ
BitsAndBytes
GGUF
INT8 W8A8
FP8 W8A8
FP8 E5M2 KV Cache
FP8 E4M3 KV Cache

Automatic Prefix Caching

Introduction
Implementation

Performance

Benchmark Suites

Community

vLLM Meetups
Sponsors

API Documentation

Sampling Parameters
Pooling Parameters
Offline Inference
- LLM Class
- LLM Inputs
vLLM Engine
- LLMEngine
- AsyncLLMEngine

Design

vLLM’s Class Hierarchy
Integration with HuggingFace
Input Processing
- Input Processing Pipeline
vLLM Paged Attention
Multi-Modality
- Adding a Multimodal Plugin

For Developers

Contributing to vLLM
Profiling vLLM
Dockerfile

Repository
Suggest edit

.rst

Offline Inference

Offline Inference#

LLM Class
LLM Inputs

previous

Pooling Parameters

next

LLM Class

By the vLLM Team

© Copyright 2024, vLLM Team.