Architecture Overview¶

This document outlines the architecture of vLLM-Omni.

Design goals¶

The primary goal of vLLM-Omni is to provide a fast and easy-to-use inference and serving engine for omni-modality models. vLLM-Omni extends vLLM's text-oriented autoregressive (AR) runtime with stage-based execution for non-textual outputs and non-autoregressive model components.

The architecture is designed to:

• support text, image, audio, video, and action inputs and outputs;
• compose autoregressive, generation, and diffusion stages in one pipeline;
• reuse vLLM scheduling, cache, distributed-execution, and serving primitives where they fit; and
• keep model topology, deployment placement, runtime lifecycle, transport, and public API concerns in separate layers.

Model execution¶

These four examples show how vLLM-Omni maps model components to stage-local execution policies. Each summary covers the model-owned path, batching, attention, parallelism, quantization, and primary serving targets. The profiles are representative; a model may support multiple PipelineConfig and DeployConfig combinations.

Serving metric vocabulary¶

The target metric follows the output modality, not just the model name.

Qwen3-Omni: streaming AR pipeline¶

Qwen3-Omni illustrates a multi-stage streaming AR pipeline: Thinker → Talker → Code2Wav. The technical report describes the model components, and the figure shows the stage-level streaming schedule used by vLLM-Omni.

Model architecture by stage¶

Representative stage execution¶

For this pipeline, the main serving target is interactive responsiveness: TTFT for text, TTFP for the first audio packet, TPOT for autoregressive decode, E2EL for the complete answer, and RTF/throughput for sustained audio serving. Async chunking and streaming primarily reduce TTFP and improve stage overlap; batching and CUDA Graphs primarily improve E2EL and throughput. See the Qwen3-Omni recipe and async chunk design for the deployment details.

HunyuanImage-3.0: shared multimodal model, split deployment choices¶

HunyuanImage-3.0 illustrates a shared multimodal model with flexible AR, DiT, and split AR → DiT deployment. The official technical report provides the model view; the tables below map it to vLLM-Omni stages.

Model architecture by stage¶

The vLLM-Omni deployment may expose the AR path only, the DiT path only, or both as an AR → DiT pipeline with a connector between them. That is a runtime decomposition of the model's capabilities; it should not be read as a claim that the published framework figure contains two unrelated backbones.

Representative stage execution¶

Image generation is therefore measured mainly with per-image E2EL/denoising latency, peak memory, and images-per-second throughput. TTFT and TPOT remain meaningful for the optional AR understanding path, but they are not the right headline metrics for the DiT path. See the HunyuanImage-3.0 recipe for the validated batching, attention, and quantization combinations.

MiniMax-H3: conditioning, denoising, and decode¶

MiniMax-H3 illustrates logical component and phase boundaries between text encoding, task-specific denoising, and media decoding: Text Encoder → DiT → VAE Decoder. The current registered topology co-locates these components in one diffusion stage. These boundaries are useful for profiling and offload; independent placement would require a future multi-stage PipelineConfig. The stage mapping follows the implementation and serving recipe; the release overview notes that a detailed technical report is forthcoming.

Model architecture by component¶

Both DiT partitions remain behind one task-selection point; task chooses which partition runs. In the current implementation, the three logical components execute inside one diffusion stage. This is different from Qwen3-Omni's three independently scheduled stages, even though both models produce synchronized audio and video-like outputs.

Representative component execution¶

The primary target is video/audio E2EL and media throughput, with the logical component boundaries making prompt encoding, denoising, and VAE decode costs separately measurable. RTF is useful when the audio stream is evaluated independently. TTFT and TPOT do not describe the main H3 generation path because the user-visible output is diffusion media rather than a token stream. The MiniMax-H3 recipe contains the hardware-specific profiles and warmup requirements.

Cosmos3: unified MoT reasoner and generator¶

Cosmos3 illustrates a unified Mixture-of-Transformers (MoT) reasoner and diffusion generator. The official technical report and NVIDIA's technical explanation provide the two-tower model view used below.

Model architecture by stage¶

The reasoner and generator are model-owned components, not two independent vLLM-Omni stages by default. The current Cosmos3 pipeline materializes both inside one diffusion stage; model-level CPU offload can swap the reasoner and generator components between their phases.

Representative stage execution¶

For Cosmos3, generation E2EL and output throughput are the primary targets. Action-policy workloads add a control-loop latency target, while synchronized audio workloads add RTF. TTFT/TPOT are appropriate for a reasoner-only service, but not for the full diffusion generation path. See the Cosmos3 recipe and diffusion execution modes for the current runtime choices.

Why the stage-based architecture follows from these examples¶

The four representative pipelines show that omni-modality serving is not one execution problem. Autoregressive decoding, diffusion denoising, multimodal encoding, and media decoding have different scheduling, attention, parallelism, memory, and quantization requirements. The architecture therefore needs a unit that can be optimized and resourced independently while remaining part of one request pipeline.

This leads to a precise definition of a stage: it is a logical execution unit with its own model component, execution policy, resource ownership, and performance objectives. A stage is not necessarily a separate process. Cosmos3 keeps its reasoner and generator inside one diffusion pipeline, while MiniMax-H3 exposes useful logical boundaries between text encoding, denoising, and VAE decoding. Both are valid mappings of the same abstraction.

The resulting separation can be summarized as:

Model structure       What is computed and how components are related
        ↓
PipelineConfig        Which logical stages exist and how they are connected
        ↓
Stage runtime policy  How each stage batches, attends, parallelizes, and quantizes
        ↓
DeployConfig           Where stages run, with which devices, replicas, and connectors

AsyncOmniEngine and Orchestrator coordinate the pipeline, OmniConnector transports stage outputs, and StageRuntime materializes the selected placement. This separation lets one logical model support multiple serving topologies without coupling model code to process layout or deployment resources.

System architecture¶

The figure shows the main components of the stage-based runtime: entrypoints, the shared AsyncOmniEngine, request orchestration, stage lifecycle management, AR and diffusion execution modules, model/layer operations, connector transport, and multimodal outputs. AsyncOmniEngine is the composition root between the entrypoints and the orchestrator.

flowchart TB
    subgraph omni["vLLM-Omni"]
        subgraph entry["Entry points"]
            api["API server"]
            client["Omni / AsyncOmni"]
        end
        engine["AsyncOmniEngine<br/>composition root and background loop"]
        orchestrator["Orchestrator<br/>request state and cross-stage routing"]
        runtime["StageRuntime<br/>placement, replicas, readiness, and lifecycle"]
        ar["AR module<br/>vLLM engine, scheduler, cache, executor, worker, and model runner"]
        diffusion["Diffusion module<br/>scheduler, worker, model runner, and pipeline"]
        ops["Model / layer / ops<br/>attention, parallelism, and quantization"]
        connector["OmniConnector<br/>payload and KV transport plus synchronization"]
        outputs["Multimodal outputs<br/>streaming chunks and final artifacts"]

        api --> engine
        client --> engine
        engine --> orchestrator
        engine --> runtime
        orchestrator --> runtime
        runtime --> ar
        runtime --> diffusion
        ar -. uses .-> ops
        diffusion -. uses .-> ops
        ar -. transports through .-> connector
        diffusion -. transports through .-> connector
        ar --> outputs
        diffusion --> outputs
    end

Key components¶

Configuration and runtime resolution¶

The control plane has five conceptual layers. Authoring inputs are resolved once into a complete, transport-safe configuration before runtime processes are started. This keeps stage topology and model capabilities distinct from deployment placement and from process-local engine objects.

The figure shows only the primary hand-off object for each layer; the detailed inputs, fields, and ownership rules are described below.

flowchart TB
    layer1["Layer 1 · Authoring inputs<br/>PipelineConfig + DeployConfig"]
    layer2["Layer 2 · Resolve once<br/>StageConfigFactory.create_from_model()<br/>VllmOmniConfig.from_pipeline_config()"]
    layer3["Layer 3 · Transport-safe control plane<br/>VllmOmniConfig"]
    layer4["Layer 4 · Runtime launch planning<br/>StageRuntime"]
    layer5["Layer 5 · Engine materialization<br/>VllmConfig / OmniDiffusionConfig"]

    layer1 --> layer2 --> layer3 --> layer4 --> layer5

The single resolution boundary is implemented by StageConfigFactory.create_from_model() and VllmOmniConfig.from_pipeline_config() in vllm_omni/config. The legacy stage_args YAML path remains only for models that have not yet migrated to PipelineConfig and DeployConfig.

In the typed path, each stage configuration derives from BaseVllmOmniStageConfig and is specialized as VllmOmniARStageConfig, VllmOmniGenerationStageConfig, or VllmOmniDiffusionStageConfig. The request and engine-spec fields belong to the control-plane boundary; the current implementation stores their equivalent projections in the structured stage configuration and materializes backend-specific engine objects during stage initialization.

The important ownership rules are:

1. PipelineConfig is the source of truth for stage topology, execution type, model capabilities, and stage relationships.
2. DeployConfig describes placement, replicas, devices, connectors, and deploy-time defaults; it does not redefine the model graph.
3. CLI and Python overrides are applied at the resolution boundary, with per-stage overrides taking precedence over global values where supported.
4. StageRuntime owns launch planning and replica lifecycle. ReplicaInitPlan is runtime-private state, not a user configuration object.
5. VllmConfig and the enriched OmniDiffusionConfig are materialized in the process that owns the corresponding engine.

Main features¶

The feature surface is grouped to match the feature design documents. This page summarizes each feature's architectural role; the linked design document is the source for configuration, compatibility, and implementation details.

Runtime and stage execution¶

• Disaggregated inference: Logical stages can run in separate processes, devices, or nodes while the orchestrator preserves their declared relationships. OmniConnector implementations transfer stage data and control-plane metadata. See Disaggregated Inference.
• Asynchronous stage and output execution: Async Chunk forwards partial stage outputs as they become available. Async Diffusion Output overlaps device-to-host output packing with the next diffusion request, while Async Omni Output Materialization moves CPU-side payload construction off the AR decode critical path.
• Automatic prefix caching: Automatic Prefix Caching in Omni Models reuses KV-cache-aligned stage outputs and multimodal tensors for requests with common prefixes.

Communication¶

• OmniConnector transport: The connector contract carries tensors, KV-cache data, and transport metadata across stage boundaries. The available implementations cover shared memory and multi-node Mooncake, Mori, and Yuanrong transports; see Disaggregated Inference for the connector choices and configuration model.

Diffusion acceleration¶

• Request and step batching: Diffusion Continuous Batching defines request-batch and step-batch execution, scheduler admission, and the common streaming output path.
• Composable parallelism: Diffusion stages can combine CFG-Parallel, Expert Parallel, HSDP, Pipeline Parallel, Sequence Parallel, Tensor Parallel, and VAE Patch Parallelism according to the pipeline and hardware topology.
• Attention and cache acceleration: Skip-Softmax, Cache-DiT, and TeaCache provide backend and denoising-step optimizations without changing the stage contract.
• Quantization and memory efficiency: Quantization resolves per-pipeline or per-component quantization configurations, while Distributed Layerwise Offload streams diffusion blocks from host memory within the existing parallel topology.

Interfaces¶

The public interfaces map onto the same engine and stage boundaries:

flowchart LR
    offline["Offline Python<br/>Omni.generate()"] --> engine["AsyncOmniEngine"]
    online["OpenAI-compatible serving<br/>vllm serve ... --omni"] --> engine
    duplex["Experimental duplex WebSocket<br/>/v1/duplex or realtime duplex"] --> engine
    engine --> stages["Configured AR and diffusion stages"]
    stages --> result["Streaming or final multimodal output"]

Offline inference¶

The Omni class provides a Python interface for offline batched inference:

from vllm_omni.entrypoints.omni import Omni

omni = Omni(model="Qwen/Qwen3-Omni-30B-A3B-Instruct")

om_inputs = {
    "prompt": prompt,
    "multi_modal_data": {
        "video": video_frames,
        "audio": audio_signal,
    },
}

outputs = omni.generate(om_inputs, sampling_params_list)

Online serving¶

The OpenAI-compatible server uses the same stage configuration and engine boundaries:

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091

For example, a Qwen3-Omni chat request can contain text, image, audio, or video content and a sampling_params_list for its configured stages. See the Qwen3-Omni serving example and the examples for complete requests.

Some pipelines expose additional OpenAI-compatible endpoints, such as joint video/audio generation. Endpoint support remains model-specific; consult the relevant model guide before assuming that every OpenAI route applies to every pipeline.