Skip to content

vllm_omni.outputs.mm_outputs

Multimodal output data structures for vLLM-Omni.

This module defines structured types for multimodal outputs.

logger module-attribute 

logger = init_logger(__name__)

MultimodalCompletionOutput dataclass

Bases: CompletionOutput

CompletionOutput with multimodal support.

Inherits all CompletionOutput fields and adds multimodal_output. As a CompletionOutput subclass, compatible with all existing vLLM consumers.

multimodal_output instance-attribute 

multimodal_output = multimodal_output

MultimodalPayload dataclass

Bases: Mapping

Structured multimodal output payload.

Implements collections.abc.Mapping so that isinstance(payload, dict) style checks in downstream code can be replaced with duck-typing, and payload.get(key), payload[key], key in payload, len(payload) all work seamlessly for both tensors and metadata.

Attributes:

Name Type Description
tensors dict[str, Tensor]

Dictionary mapping modality/key names to their tensors.
metadata dict[str, Any]

Optional dictionary for non-tensor metadata (e.g., sample rate for audio, image dimensions).

is_empty property 

is_empty: bool

Return True if the payload has no tensors and no metadata.

metadata class-attribute instance-attribute 

metadata: dict[str, Any] = field(default_factory=dict)

primary_tensor property 

primary_tensor: Tensor | None

Return the first tensor in the payload, or None if empty.

tensors class-attribute instance-attribute 

tensors: dict[str, Tensor] = field(default_factory=dict)

consolidate_metadata 

consolidate_metadata() -> None

Resolve deferred tensor lists in metadata by keeping the latest value.

Metadata values are per-step snapshots (e.g. sample rate), not content deltas, so the latest value supersedes earlier ones. Nested dicts (unflattened payloads) are resolved one level down.

consolidate_tensors 

consolidate_tensors(
    strategy: TensorAccumulationStrategy,
) -> None

Concatenate deferred tensor lists into single tensors.

Tensors are generated content accumulated as chunks, so lists are concatenated according to strategy (e.g. audio chunks along the time dimension, latent frames along the batch dimension).

from_dict classmethod 

from_dict(
    data: dict[str, Any] | None,
) -> MultimodalPayload | None

Create a MultimodalPayload from a raw dictionary.

Separates torch.Tensor values into tensors and everything else into metadata.

from_raw classmethod 

from_raw(
    payload: Any, modality_key: str
) -> MultimodalPayload | None

Create a MultimodalPayload from a raw producer payload.

Accepts a MultimodalPayload (returned as-is), a dict, or a bare tensor (stored under modality_key). Tensors are moved to CPU. Producer-specific dict keys are remapped to the semantic modality key (e.g. "audio", "latent"): AR runners produce {"hidden": ...} and generation runners produce {"model_outputs": ...}.

get 

get(key: str, default: Any = None) -> Any

Get a value by key, searching tensors first then metadata.

merged_with 

merged_with(
    incoming: MultimodalPayload,
) -> MultimodalPayload

Merge incoming onto this payload and return the result.

Tensor values accumulate into lists for deferred concatenation; non-tensor values are replaced with the latest. When this payload is empty, incoming is returned as-is, so callers should use the return value: accumulated = accumulated.merged_with(incoming).

to_dict 

to_dict() -> dict[str, Any]

Convert back to a plain dict (tensors + metadata merged).