Welcome to vLLM!

Easy, fast, and cheap LLM serving for everyone

Star Watch Fork

vLLM is a fast and easy-to-use library for LLM inference and serving.

vLLM is fast with:

• State-of-the-art serving throughput

• Efficient management of attention key and value memory with PagedAttention

• Continuous batching of incoming requests

• Fast model execution with CUDA/HIP graph

• Quantization: GPTQ, AWQ, SqueezeLLM, FP8 KV Cache

• Optimized CUDA kernels

vLLM is flexible and easy to use with:

• Seamless integration with popular HuggingFace models

• High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more

• Tensor parallelism support for distributed inference

• Streaming outputs

• OpenAI-compatible API server

• Support NVIDIA GPUs and AMD GPUs

• (Experimental) Prefix caching support

• (Experimental) Multi-lora support

For more information, check out the following:

• vLLM announcing blog post (intro to PagedAttention)

• vLLM paper (SOSP 2023)

• How continuous batching enables 23x throughput in LLM inference while reducing p50 latency by Cade Daniel et al.

• vLLM Meetups.

Documentation

Getting Started

• Installation
• Installation with ROCm
• Installation with OpenVINO
• Installation with CPU
• Installation with Neuron
• Installation with TPU
• Installation with XPU
• Quickstart
• Debugging Tips
• Examples

Serving

• OpenAI Compatible Server
• Deploying with Docker
• Distributed Inference and Serving
• Production Metrics
• Environment Variables
• Usage Stats Collection
• Integrations
• Loading Models with CoreWeave’s Tensorizer
• Frequently Asked Questions

Models

• Supported Models
• Adding a New Model
• Engine Arguments
• Using LoRA adapters
• Using VLMs
• Speculative decoding in vLLM
• Performance and Tuning

Quantization

• Supported Hardware for Quantization Kernels
• AutoAWQ
• FP8
• FP8 E5M2 KV Cache
• FP8 E4M3 KV Cache

Automatic Prefix Caching

• Introduction
• Implementation
• Generalized Caching Policy

Developer Documentation

• Sampling Parameters
  • SamplingParams
    • SamplingParams.clone()
    • SamplingParams.update_from_generation_config()
• Offline Inference
  • LLM Class
    • LLM
      • LLM.DEPRECATE_LEGACY
      • LLM.encode()
      • LLM.generate()
  • LLM Inputs
    • PromptStrictInputs
    • TextPrompt
      • TextPrompt.prompt
      • TextPrompt.multi_modal_data
    • TokensPrompt
      • TokensPrompt.prompt_token_ids
      • TokensPrompt.multi_modal_data
• vLLM Engine
  • LLMEngine
    • LLMEngine
      • LLMEngine.DO_VALIDATE_OUTPUT
      • LLMEngine.abort_request()
      • LLMEngine.add_request()
      • LLMEngine.do_log_stats()
      • LLMEngine.from_engine_args()
      • LLMEngine.get_decoding_config()
      • LLMEngine.get_model_config()
      • LLMEngine.get_num_unfinished_requests()
      • LLMEngine.has_unfinished_requests()
      • LLMEngine.has_unfinished_requests_for_virtual_engine()
      • LLMEngine.step()
  • AsyncLLMEngine
    • AsyncLLMEngine
      • AsyncLLMEngine.abort()
      • AsyncLLMEngine.check_health()
      • AsyncLLMEngine.encode()
      • AsyncLLMEngine.engine_step()
      • AsyncLLMEngine.from_engine_args()
      • AsyncLLMEngine.generate()
      • AsyncLLMEngine.get_decoding_config()
      • AsyncLLMEngine.get_model_config()
      • AsyncLLMEngine.start_background_loop()
• vLLM Paged Attention
  • Inputs
  • Concepts
  • Query
  • Key
  • QK
  • Softmax
    • qk_max and logits
    • exp_sum
  • Value
  • LV
  • Output
• Input Processing
  • Guides
    • Input Processing Pipeline
  • Module Contents
    • LLM Engine Inputs
      • LLMInputs
        • LLMInputs.multi_modal_data
        • LLMInputs.prompt
        • LLMInputs.prompt_token_ids
    • Registry
      • INPUT_REGISTRY
      • DummyDataFactory
      • InputContext
        • InputContext.get_hf_config()
        • InputContext.get_multimodal_config()
        • InputContext.model_config
      • InputProcessor
      • InputRegistry
        • InputRegistry.create_input_processor()
        • InputRegistry.dummy_data_for_profiling()
        • InputRegistry.process_input()
        • InputRegistry.register_dummy_data()
        • InputRegistry.register_input_processor()
• Multi-Modality
  • Guides
    • Adding a New Multimodal Model
      • 1. Set up the base vLLM model
      • 2. Register input mappers
      • 3. Register maximum number of multimodal tokens
      • 4. (Optional) Register dummy data
      • 5. (Optional) Register input processor
  • Module Contents
    • Registry
      • MULTIMODAL_REGISTRY
      • MultiModalRegistry
        • MultiModalRegistry.create_input_mapper()
        • MultiModalRegistry.get_max_multimodal_tokens()
        • MultiModalRegistry.map_input()
        • MultiModalRegistry.register_image_input_mapper()
        • MultiModalRegistry.register_input_mapper()
        • MultiModalRegistry.register_max_image_tokens()
        • MultiModalRegistry.register_max_multimodal_tokens()
    • Base Classes
      • MultiModalDataDict
      • MultiModalInputs
        • MultiModalInputs.batch()
      • MultiModalPlugin
        • MultiModalPlugin.get_data_key()
        • MultiModalPlugin.get_max_multimodal_tokens()
        • MultiModalPlugin.map_input()
        • MultiModalPlugin.register_input_mapper()
        • MultiModalPlugin.register_max_multimodal_tokens()
    • Image Classes
      • ImagePlugin
        • ImagePlugin.get_data_key()
• Dockerfile

Community

• vLLM Meetups
• Sponsors

Indices and tables

• Index

• Module Index