Skip to main content
Ctrl+K
vLLM - Home

Getting Started

  • Installation
  • Installation with ROCm
  • Installation with Neuron
  • Installation with CPU
  • Quickstart
  • Examples
    • API Client
    • Aqlm Example
    • Gradio OpenAI Chatbot Webserver
    • Gradio Webserver
    • Llava Example
    • LLM Engine Example
    • Lora With Quantization Inference
    • MultiLoRA Inference
    • Offline Inference
    • Offline Inference Arctic
    • Offline Inference Distributed
    • Offline Inference Embedding
    • Offline Inference Neuron
    • Offline Inference With Prefix
    • OpenAI Chat Completion Client
    • OpenAI Completion Client
    • OpenAI Embedding Client
    • Save Sharded State
    • Tensorize vLLM Model

Serving

  • OpenAI Compatible Server
  • Deploying with Docker
  • Distributed Inference and Serving
  • Production Metrics
  • Environment Variables
  • Usage Stats Collection
  • Integrations
    • Deploying and scaling up with SkyPilot
    • Deploying with KServe
    • Deploying with NVIDIA Triton
    • Deploying with BentoML
    • Deploying with LWS
    • Deploying with dstack
    • Serving with Langchain

Models

  • Supported Models
  • Adding a New Model
  • Engine Arguments
  • Using LoRA adapters
  • Using VLMs
  • Performance and Tuning

Quantization

  • AutoAWQ
  • FP8 E5M2 KV Cache
  • FP8 E4M3 KV Cache

Developer Documentation

  • Sampling Parameters
  • Offline Inference
    • LLM Class
    • LLM Inputs
  • vLLM Engine
    • LLMEngine
    • AsyncLLMEngine
  • vLLM Paged Attention
  • Multi-Modality
  • Dockerfile

Community

  • vLLM Meetups
  • Sponsors

Python Module Index

v
 
v
- vllm
    vllm.engine
    vllm.multimodal
    vllm.multimodal.image

By the vLLM Team

© Copyright 2024, vLLM Team.