Skip to main content

Ctrl+K

Getting Started

Installation
Installation with ROCm
Installation with OpenVINO
Installation with CPU
Installation with Neuron
Installation with TPU
Installation with XPU
Quickstart
Debugging Tips
Examples

Serving

OpenAI Compatible Server
Deploying with Docker
Distributed Inference and Serving
Production Metrics
Environment Variables
Usage Stats Collection
Integrations
Loading Models with CoreWeave’s Tensorizer
Frequently Asked Questions

Models

Supported Models
Adding a New Model
Enabling Multimodal Inputs
Engine Arguments
Using LoRA adapters
Using VLMs
Speculative decoding in vLLM
Performance and Tuning

Quantization

Supported Hardware for Quantization Kernels
AutoAWQ
FP8
FP8 E5M2 KV Cache
FP8 E4M3 KV Cache

Automatic Prefix Caching

Introduction
Implementation

Developer Documentation

Sampling Parameters
Offline Inference
- LLM Class
- LLM Inputs
vLLM Engine
- LLMEngine
- AsyncLLMEngine
vLLM Paged Attention
Input Processing
- Input Processing Pipeline
Multi-Modality
- Adding a Multimodal Plugin
Dockerfile

Community

vLLM Meetups
Sponsors

Repository
Suggest edit

.rst

Sampling Parameters

Sampling Parameters#

previous

Implementation

next

Offline Inference

By the vLLM Team

© Copyright 2024, vLLM Team.