Skip to content
vLLM-Omni
patch
Initializing search
vllm-project/vllm-omni
Home
User Guide
Developer Guide
API Reference
CLI Reference
Community
vLLM-Omni
vllm-project/vllm-omni
Home
User Guide
User Guide
Getting Started
Getting Started
Quickstart
GPU
NPU
Serving
Serving
OpenAI-Compatible API
OpenAI-Compatible API
Diffusion Chat API
Image Generation
Image Edit
Text to Speech
Examples
Examples
Offline Inference
Offline Inference
BAGEL-7B-MoT
CosyVoice3
Fish Speech S2 Pro
GLM-Image Multistage End-to-End Inference
Helios Video Generation
HunyuanImage-3.0 Image-to-Text Inference
Image-To-Image
Image-To-Video
MammothModa2-Preview
MiMo-Audio Offline Inference
Qwen2.5-Omni
Qwen3-Omni
Qwen3-TTS
Text-To-Audio
Text-To-Image
Text-To-Video
Voxtral TTS Offline Inference
X-To-Video-Audio
Online Serving
Online Serving
BAGEL-7B-MoT
vLLM-Omni Helm Chart
Fish Speech S2 Pro
GLM-Image Online Serving
Helios Video Generation via Online API
Image-To-Image
Image-To-Video
Online serving Example of vLLM-Omni for MiMo-Audio
OpenAI Chat Completion Client For Multimodal Generation
Qwen2.5-Omni
Qwen3-Omni
Qwen3-TTS
Text-To-Image
Text-To-Video
General
General
Frequently Asked Questions
Configuration
Configuration
GPU Memory Calculation and Configuration
Prefill-Decode (PD) Disaggregation
Stage configs for vLLM-Omni
Models
Models
Supported Models
Features
Features
Sleep Mode
Diffusion Features
Diffusion Features
Acceleration Overview
TeaCache
Cache-DiT
Quantization
Quantization
Overview
FP8
Int8
GGUF
Step Execution
Parallelism Acceleration
CPU Offloading
LoRA
Hybrid Sharded Data Parallel
Custom Pipeline
ComfyUI
Developer Guide
Developer Guide
General
General
VLLM-Omni PR Reviewer
Metrics
Profiling vLLM-Omni
Model Implementation
Model Implementation
Adding an Omni-Modality Model
Adding a TTS Model
Adding a Diffusion Model
Support FP8 Quantization
CI
CI
Multi-Level Automated Testing System Documentation
CI Failures
Test Guide
Markers for Tests
Test File Structure and Style Guide
Design Documents
Design Documents
Architecture Overview
Feature Design
Feature Design
Disaggregated Inference for Omni-Modality Models
Distributed utils
Omni connectors
Omni connectors
MooncakeStoreConnector
MooncakeTransferEngineConnector
SharedMemoryConnector
YuanrongConnector
CFG-Parallel
Sequence Parallel
Tensor Parallel
Cache-DiT
TeaCache
Async Chunk Design
VAE Patch Parallelism
Diffusion Step Execution
Module Design
Module Design
AutoRegressive (AR) Module
Diffusion Module Architecture Design
Entrypoint module
AsyncOmni Architecture (Qwen3-Omni Example)
Docs Guide
API Reference
API Reference
vllm_omni
vllm_omni
logger
outputs
patch
request
version
config
config
lora
model
stage_config
yaml_util
core
core
sched
sched
omni_ar_scheduler
omni_generation_scheduler
output
diffusion
diffusion
compile
data
diffusion_engine
envs
forward_context
ipc
registry
request
stage_diffusion_client
attention
attention
layer
selector
backends
backends
abstract
flash_attn
registry
ring_flash_attn
ring_pytorch_attn
sage_attn
sdpa
ring
ring
ring_globals
ring_kernels
ring_selector
ring_utils
utils
utils
fa
parallel
parallel
base
factory
ring
ulysses
cache
cache
base
cache_dit_backend
selector
teacache
teacache
backend
coefficient_estimator
config
extractors
hook
state
distributed
distributed
cfg_parallel
comm
group_coordinator
hsdp
parallel_state
sp_plan
sp_sharding
utils
vae_patch_parallel
autoencoders
autoencoders
autoencoder_kl
autoencoder_kl_qwenimage
autoencoder_kl_wan
distributed_vae_executor
executor
executor
abstract
multiproc_executor
hooks
hooks
base
sequence_parallel
layers
layers
adalayernorm
custom_op
rope
lora
lora
manager
utils
layers
layers
base_linear
column_parallel_linear
replicated_linear
row_parallel_linear
model_loader
model_loader
diffusers_loader
gguf_adapters
gguf_adapters
base
flux2_klein
qwen_image
z_image
models
models
interface
progress_bar
bagel
bagel
autoencoder
bagel_transformer
pipeline_bagel
cosyvoice3_audio
cosyvoice3_audio
cosyvoice3_dit
dreamid_omni
dreamid_omni
fusion
pipeline_dreamid_omni
wan2_2
flux
flux
flux_pipeline_mixin
flux_transformer
pipeline_flux
pipeline_flux_kontext
flux2
flux2
flux2_transformer
pipeline_flux2
flux2_klein
flux2_klein
flux2_klein_transformer
pipeline_flux2_klein
glm_image
glm_image
glm_image_transformer
pipeline_glm_image
helios
helios
helios_transformer
pipeline_helios
scheduling_helios
hunyuan_image_3
hunyuan_image_3
autoencoder
hunyuan_fused_moe
hunyuan_image_3_tokenizer
hunyuan_image_3_transformer
pipeline_hunyuan_image_3
hunyuan_video
hunyuan_video
hunyuan_video_15_transformer
pipeline_hunyuan_video_1_5
pipeline_hunyuan_video_1_5_i2v
longcat_image
longcat_image
longcat_image_transformer
pipeline_longcat_image
pipeline_longcat_image_edit
ltx2
ltx2
ltx2_transformer
pipeline_ltx2
pipeline_ltx2_image2video
nextstep_1_1
nextstep_1_1
modeling_flux_vae
modeling_nextstep
modeling_nextstep_heads
modeling_nextstep_llama
pipeline_nextstep_1_1
omnigen2
omnigen2
omnigen2_transformer
pipeline_omnigen2
ovis_image
ovis_image
ovis_image_transformer
pipeline_ovis_image
schedulers
schedulers
base
scheduling_flow_unipc_multistep
sd3
sd3
pipeline_sd3
sd3_transformer
stable_audio
stable_audio
pipeline_stable_audio
stable_audio_transformer
t5_encoder
t5_encoder
t5_encoder
wan2_2
wan2_2
pipeline_wan2_2
pipeline_wan2_2_i2v
pipeline_wan2_2_ti2v
wan2_2_transformer
z_image
z_image
pipeline_z_image
z_image_transformer
offloader
offloader
base
layerwise_backend
module_collector
sequential_backend
profiler
profiler
diffusion_pipeline_profiler
sched
sched
base_scheduler
interface
request_scheduler
utils
utils
hf_utils
network_utils
tf_utils
worker
worker
diffusion_model_runner
diffusion_worker
utils
distributed
distributed
kv_transfer
kv_transfer
monkey_patch
omni_connectors
omni_connectors
adapter
factory
kv_transfer_manager
connectors
connectors
base
mooncake_store_connector
mooncake_transfer_engine_connector
shm_connector
yuanrong_connector
transfer_adapter
transfer_adapter
base
chunk_transfer_adapter
utils
utils
config
initialization
kv_utils
logging
serialization
omni_coordinator
omni_coordinator
load_balancer
messages
omni_coord_client_for_hub
omni_coord_client_for_stage
omni_coordinator
ray_utils
ray_utils
utils
engine
engine
arg_utils
async_omni_engine
mm_outputs
orchestrator
output_modality
output_processor
serialization
stage_engine_core_client
stage_init_utils
entrypoints
entrypoints
async_omni
async_omni_diffusion
cfg_companion_tracker
chat_utils
client_request_state
omni
omni_base
pd_utils
stage_utils
utils
cli
cli
logo
main
serve
benchmark
benchmark
base
main
serve
inputs
inputs
data
preprocess
lora
lora
request
utils
metrics
metrics
stats
utils
model_executor
model_executor
custom_process_mixin
layers
layers
rotary_embedding
rotary_embedding
mrope
model_loader
model_loader
weight_utils
models
models
output_templates
registry
utils
bagel
bagel
bagel
cosyvoice3
cosyvoice3
config
cosyvoice3
cosyvoice3_code2wav
cosyvoice3_talker
tokenizer
utils
code2wav_core
code2wav_core
cfm
hifigan
layers
fish_speech
fish_speech
configuration_fish_speech
dac_encoder
dac_utils
fish_speech_dac_decoder
fish_speech_fast_ar
fish_speech_slow_ar
prompt_utils
glm_image
glm_image
glm_image_ar
hunyuan_image3
hunyuan_image3
autoencoder_kl_3d
hunyuan_image3
siglip2
mammoth_moda2
mammoth_moda2
mammoth_moda2
pipeline_mammothmoda2_dit
mimo_audio
mimo_audio
config_mimo_audio
mimo_audio
mimo_audio_code2wav
mimo_audio_llm
modeling_audio_tokenizer
modeling_rope_utils
quantization
qwen2_5_omni
qwen2_5_omni
audio_length
qwen2_5_omni
qwen2_5_omni_talker
qwen2_5_omni_thinker
qwen2_5_omni_token2wav
qwen2_old
qwen3_omni
qwen3_omni
qwen3_moe
qwen3_omni
qwen3_omni_code2wav
qwen3_omni_moe_code_predictor_mtp
qwen3_omni_moe_talker
qwen3_omni_moe_thinker
qwen3_tts
qwen3_tts
configuration_qwen3_tts
cuda_graph_decoder_wrapper
qwen3_tts_code2wav
qwen3_tts_code_predictor_vllm
qwen3_tts_talker
qwen3_tts_tokenizer
voice_cache_manager
tokenizer_12hz
tokenizer_12hz
configuration_qwen3_tts_tokenizer_v2
modeling_qwen3_tts_tokenizer_v2
tokenizer_25hz
tokenizer_25hz
configuration_qwen3_tts_tokenizer_v1
modeling_qwen3_tts_tokenizer_v1
vq
vq
core_vq
speech_vq
whisper_encoder
stage_configs
stage_configs
stage_input_processors
stage_input_processors
bagel
chunk_size_utils
cosyvoice3
fish_speech
glm_image
mammoth_moda2
mimo_audio
qwen2_5_omni
qwen3_omni
qwen3_tts
tts_utils
voxtral_tts
platforms
platforms
interface
cuda
cuda
platform
npu
npu
platform
profiler
models
models
hunyuan_fused_moe
worker
worker
base
npu_ar_model_runner
npu_ar_worker
npu_generation_model_runner
npu_generation_worker
npu_model_runner
rocm
rocm
platform
xpu
xpu
platform
utils
worker
worker
xpu_ar_model_runner
xpu_ar_worker
xpu_generation_model_runner
xpu_generation_worker
plugins
plugins
profiler
profiler
omni_torch_profiler
sample
sample
tokenizers
tokenizers
mammoth_moda2_tokenizer
transformers_utils
transformers_utils
configs
configs
fish_speech
mammoth_moda2
utils
utils
worker
worker
base
gpu_ar_model_runner
gpu_ar_worker
gpu_generation_model_runner
gpu_generation_worker
gpu_memory_utils
gpu_model_runner
mixins
CLI Reference
CLI Reference
vllm-omni serve
Bench
Bench
vLLM-Omni Benchmark CLI Guide
Community
Community
Governance
Contact Us
Meetups
Volunteers for Bugfix and CI
Slack
Blog
Forum
vllm_omni.patch
¶
Back to top