Skip to content

vllm_omni.diffusion.models.kandinsky6

Modules:

Name Description
cache_accel

Kandinsky 6 cache adapters.

kandinsky6_transformer

Generated standalone Kandinsky 6 vLLM-Omni transformer.

modeling_kandinsky6_audio

Generated standalone Kandinsky 6 audio VAE component for vLLM-Omni.

modeling_kandinsky6_vae

Generated standalone Kandinsky 6 video VAE component for vLLM-Omni.

pipeline_kandinsky6

Generated standalone Kandinsky 6 vLLM-Omni pipeline.

scheduling_kandinsky6

Native K6 shifted flow-matching Euler scheduler.

super_resolution

Optional HD/Full-HD refinement after Kandinsky6 video/audio generation.

AutoencoderKLHunyuanVideo

Bases: ModelMixin, ConfigMixin, DistributedVaeMixin

A VAE model with KL loss for encoding videos into latents and decoding latent representations into videos. Introduced in HunyuanVideo.

This model inherits from [ModelMixin]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving).

decoder instance-attribute

decoder = HunyuanVideoDecoder3D(
    in_channels=latent_channels,
    out_channels=out_channels,
    up_block_types=up_block_types,
    block_out_channels=block_out_channels,
    layers_per_block=layers_per_block,
    norm_num_groups=norm_num_groups,
    act_fn=act_fn,
    time_compression_ratio=temporal_compression_ratio,
    spatial_compression_ratio=spatial_compression_ratio,
    mid_block_add_attention=mid_block_add_attention,
)

distributed_executor instance-attribute

distributed_executor = None

encoder instance-attribute

encoder = HunyuanVideoEncoder3D(
    in_channels=in_channels,
    out_channels=latent_channels,
    down_block_types=down_block_types,
    block_out_channels=block_out_channels,
    layers_per_block=layers_per_block,
    norm_num_groups=norm_num_groups,
    act_fn=act_fn,
    double_z=True,
    mid_block_add_attention=mid_block_add_attention,
    temporal_compression_ratio=temporal_compression_ratio,
    spatial_compression_ratio=spatial_compression_ratio,
)

post_quant_conv instance-attribute

post_quant_conv = nn.Conv3d(
    latent_channels, latent_channels, kernel_size=1
)

quant_conv instance-attribute

quant_conv = nn.Conv3d(
    2 * latent_channels, 2 * latent_channels, kernel_size=1
)

spatial_compression_ratio instance-attribute

spatial_compression_ratio = spatial_compression_ratio

temporal_compression_ratio instance-attribute

temporal_compression_ratio = temporal_compression_ratio

tile_sample_min_height instance-attribute

tile_sample_min_height = 256

tile_sample_min_num_frames instance-attribute

tile_sample_min_num_frames = 16

tile_sample_min_width instance-attribute

tile_sample_min_width = 256

tile_sample_stride_height instance-attribute

tile_sample_stride_height = 192

tile_sample_stride_num_frames instance-attribute

tile_sample_stride_num_frames = 12

tile_sample_stride_width instance-attribute

tile_sample_stride_width = 192

tile_size instance-attribute

tile_size = None

time_compression_ratio instance-attribute

time_compression_ratio = temporal_compression_ratio

use_framewise_decoding instance-attribute

use_framewise_decoding = True

use_framewise_encoding instance-attribute

use_framewise_encoding = True

use_slicing instance-attribute

use_slicing = False

use_tiling instance-attribute

use_tiling = True

apply_tiling

apply_tiling(
    tile: tuple[int, int, int, int],
    stride: tuple[int, int, int],
)

Applies tiling.

blend_h

blend_h(a: Tensor, b: Tensor, blend_extent: int) -> Tensor

blend_t

blend_t(a: Tensor, b: Tensor, blend_extent: int) -> Tensor

blend_v

blend_v(a: Tensor, b: Tensor, blend_extent: int) -> Tensor

decode

decode(
    z: Tensor, return_dict: bool = True, **ignore_kwargs
) -> DecoderOutput | Tensor

Decode a batch of images.

Parameters:

Name Type Description Default
z `torch.Tensor`

Input batch of latent vectors.

required
return_dict `bool`, *optional*, defaults to `True`

Whether to return a [~models.vae.DecoderOutput] instead of a plain tuple.

True

Returns:

Type Description
DecoderOutput | Tensor

[~models.vae.DecoderOutput] or tuple: If return_dict is True, a [~models.vae.DecoderOutput] is returned, otherwise a plain tuple is returned.

encode

encode(
    x: Tensor,
    opt_tiling: bool = True,
    return_dict: bool = True,
) -> (
    AutoencoderKLOutput
    | tuple[DiagonalGaussianDistribution]
)

Encode a batch of images into latents.

Parameters:

Name Type Description Default
x `torch.Tensor`

Input batch of images.

required
return_dict `bool`, *optional*, defaults to `True`

Whether to return a [~models.autoencoder_kl.AutoencoderKLOutput] instead of a plain tuple.

True

Returns:

Type Description
AutoencoderKLOutput | tuple[DiagonalGaussianDistribution]

The latent representations of the encoded videos. If return_dict is True, a

AutoencoderKLOutput | tuple[DiagonalGaussianDistribution]

[~models.autoencoder_kl.AutoencoderKLOutput] is returned,

AutoencoderKLOutput | tuple[DiagonalGaussianDistribution]

otherwise a plain tuple is returned.

forward

forward(
    sample: Tensor,
    sample_posterior: bool = False,
    return_dict: bool = True,
    generator: Generator | None = None,
) -> DecoderOutput | Tensor

Parameters:

Name Type Description Default
sample `torch.Tensor`

Input sample.

required
sample_posterior `bool`, *optional*, defaults to `False`

Whether to sample from the posterior.

False
return_dict `bool`, *optional*, defaults to `True`

Whether or not to return a [DecoderOutput] instead of a plain tuple.

True

get_dec_optimal_tiling

get_dec_optimal_tiling(
    shape: list[int], device: device | str | None = None
) -> tuple[tuple[int, int, int, int], tuple[int, int, int]]

Returns optimal tiling for given shape.

get_enc_optimal_tiling

get_enc_optimal_tiling(
    shape: list[int], device: device | str | None = None
) -> tuple[tuple[int, int, int, int], tuple[int, int, int]]

Returns optimal tiling for given shape.

set_parallel_size

set_parallel_size(
    parallel_size: int, mode: str = "tile"
) -> None

tiled_decode

tiled_decode(
    z: Tensor, return_dict: bool = True
) -> DecoderOutput | Tensor

Decode a batch of images using a tiled decoder.

Parameters:

Name Type Description Default
z `torch.Tensor`

Input batch of latent vectors.

required
return_dict `bool`, *optional*, defaults to `True`

Whether or not to return a [~models.vae.DecoderOutput] instead of a plain tuple.

True

Returns:

Type Description
DecoderOutput | Tensor

[~models.vae.DecoderOutput] or tuple: If return_dict is True, a [~models.vae.DecoderOutput] is returned, otherwise a plain tuple is returned.

tiled_encode

tiled_encode(x: Tensor) -> AutoencoderKLOutput

Encode a batch of images using a tiled encoder.

Parameters:

Name Type Description Default
x `torch.Tensor`

Input batch of videos.

required

Returns:

Type Description
AutoencoderKLOutput

torch.Tensor: The latent representation of the encoded videos.

Kandinsky6AudioVAE

Bases: ModelMixin, ConfigMixin

vLLM-Omni-serializable K6 audio VAE, mel frontend, and vocoder.

The conversion-only checkpoint paths are accepted for materializing a bundle, while vocoder_config makes a saved bundle self-contained.

device property

device: device

downsample_factor instance-attribute

downsample_factor = int(self.native.downsample_factor)

dtype property

dtype: dtype

ignore_for_config class-attribute instance-attribute

ignore_for_config = [
    "tod_vae_ckpt",
    "bigvgan_vocoder_ckpt",
    "device",
]

native instance-attribute

native = FeaturesUtils(
    tod_vae_ckpt=None
    if tod_vae_ckpt is None
    else str(tod_vae_ckpt),
    bigvgan_vocoder_ckpt=None
    if bigvgan_vocoder_ckpt is None
    else str(bigvgan_vocoder_ckpt),
    vocoder_config=vocoder_config,
    mode=mode,
    need_vae_encoder=need_vae_encoder,
    need_vae_decoder=need_vae_decoder,
    scaling_factor=scaling_factor,
).eval()

scaling_factor instance-attribute

scaling_factor = float(scaling_factor)

decode

decode(latents: Tensor) -> Tensor

encode_audio

encode_audio(audio: Tensor)

vocode

vocode(mel: Tensor) -> Tensor

wrapped_decode

wrapped_decode(latents: Tensor) -> Tensor

wrapped_encode

wrapped_encode(audio: Tensor) -> Tensor

Kandinsky6TI2VAPipeline

Bases: Module, CFGParallelMixin, ProgressBarMixin, DiffusionPipelineProfilerMixin, SupportsComponentDiscovery, SupportAudioOutput, SupportImageInput

vLLM-Omni pipeline for text/image-to-video-and-audio generation with Kandinsky 6.

Parameters:

Name Type Description Default
transformer Module | None

Multimodal K6 transformer used to denoise video and audio latents (Kandinsky6Transformer3DModel).

None
vae Module | None

Video VAE used to decode generated video latents (AutoencoderKLHunyuanVideo).

None
text_encoder Module | None

Qwen2.5-VL text encoder for token-level embeddings.

None
audio_vae Module | None

Audio VAE used to decode generated audio latents (Kandinsky6AudioVAE). May be None only when sample_audio=False is the caller's own default.

None
scheduler object | None

Flow-matching scheduler used by the denoising loop (KandinskyFlowMatchScheduler).

None
tokenizer object | None

Qwen2.5-VL processor.

None
text_encoder_2 Module | None

CLIP text encoder for pooled embeddings.

None
tokenizer_2 object | None

CLIP tokenizer.

None

audio_downsample_factor instance-attribute

audio_downsample_factor = audio_downsample_factor

audio_sample_rate instance-attribute

audio_sample_rate = audio_sample_rate

audio_vae instance-attribute

audio_vae = audio_vae

default_guidance_scale instance-attribute

default_guidance_scale = float(
    getattr(
        scheduler,
        "default_guidance_scale",
        KandinskyFlowMatchScheduler.default_guidance_scale,
    )
)

default_height instance-attribute

default_height = int(
    getattr(
        scheduler,
        "default_height",
        KandinskyFlowMatchScheduler.default_height,
    )
)

default_num_frames instance-attribute

default_num_frames = int(
    getattr(
        scheduler,
        "default_num_frames",
        KandinskyFlowMatchScheduler.default_num_frames,
    )
)

default_num_inference_steps class-attribute instance-attribute

default_num_inference_steps = int(
    getattr(
        scheduler,
        "default_num_inference_steps",
        self.default_num_inference_steps,
    )
)

default_width instance-attribute

default_width = int(
    getattr(
        scheduler,
        "default_width",
        KandinskyFlowMatchScheduler.default_width,
    )
)

device instance-attribute

device = get_local_device()

dummy_run_num_frames class-attribute instance-attribute

dummy_run_num_frames = 0

max_sequence_length instance-attribute

max_sequence_length = max_sequence_length

od_config instance-attribute

od_config = od_config

sample_fps instance-attribute

sample_fps = sample_fps

scale_factor instance-attribute

scale_factor = scale_factor

scheduler instance-attribute

scheduler = scheduler

support_audio_output class-attribute instance-attribute

support_audio_output = True

support_image_input class-attribute instance-attribute

support_image_input = True

supports_step_execution class-attribute instance-attribute

supports_step_execution = True

text_encoder instance-attribute

text_encoder = text_encoder

text_encoder_2 instance-attribute

text_encoder_2 = text_encoder_2

text_token_padding instance-attribute

text_token_padding = text_token_padding

tokenizer instance-attribute

tokenizer = tokenizer

tokenizer_2 instance-attribute

tokenizer_2 = tokenizer_2

transformer instance-attribute

transformer = transformer

vae instance-attribute

vae = vae

denoise_step

denoise_step(
    input_batch: Any, **kwargs: Any
) -> Tensor | None

One CFG-combined video velocity. Audio velocity is stashed on the request.

diffuse

diffuse(
    *,
    bundle: LatentBundle,
    positive: TextEmbeds,
    negative: TextEmbeds,
    positive_cu: Tensor,
    negative_cu: Tensor,
    positive_mask: Tensor | None,
    negative_mask: Tensor | None,
    visual_rope: Tensor,
    audio_rope: Tensor | None,
    num_inference_steps: int,
    guidance_scale: float,
    first_frames: Tensor | None,
    visual_cond_scheme: str,
    sample_audio: bool,
    visual_token_type_ids: Tensor | None,
    device: device,
) -> LatentBundle

encode_prompt

encode_prompt(
    text: str,
) -> tuple[TextEmbeds, Tensor, Tensor | None]

forward

load_weights

load_weights(
    weights: Iterable[tuple[str, Tensor]],
) -> set[str]

Load each Hub folder into its module.

The framework yields transformer.*, vae.*, text_encoder.*, text_encoder_2.*, and audio_vae.* from weights_sources.

post_decode

post_decode(state: Any, **kwargs: Any) -> DiffusionOutput

Decode the finished video and audio latents.

predict_noise

predict_noise(
    *,
    forward,
    text_embeds,
    pooled,
    rope,
    attn_mask,
    cache_scope,
)

One CFG branch. forward is the denoise loop's DiT call.

prepare_encode

prepare_encode(state: Any, **kwargs: Any) -> Any

Encode one request and store everything the denoise step needs.

prepare_latents

prepare_latents(
    *,
    latent_frames: int,
    height: int,
    width: int,
    dtype: dtype,
    device: device,
    seed: int,
    sample_audio: bool,
) -> LatentBundle

step_scheduler

step_scheduler(
    state: Any, noise_pred: Tensor | None, **kwargs: Any
) -> None

Apply one scheduler update to video and, when present, audio.

Kandinsky6Transformer3DModel

Bases: Module

Kandinsky 6 DiT — handles T2V (is_multimodal=False) and T2VA (is_multimodal=True).

Tensor-parallel port of core/components/dit.py::DiffusionTransformer3D, with submodule names/nesting matching the Diffusers-exported checkpoint layout (see module docstring) so from_pretrained loads a convert_checkpoint.py --use-patched-diffusers bundle directly.

The class is a plain nn.Module. from_diffusers_config reads the bundle config.json. Safetensors from transformer/ are applied later by the pipeline load_weights path, which narrows full checkpoint tensors onto each rank's ColumnParallelLinear / RowParallelLinear shard.

audio_embeddings instance-attribute

audio_embeddings = Kandinsky6TextEmbeddings(
    in_audio_dim,
    model_dim_a,
    quant_config=quant_config,
    prefix=f"{prefix}.audio_embeddings",
)

audio_out_layer instance-attribute

audio_out_layer = Kandinsky6OutLayerAudio(
    model_dim_a,
    time_dim_a,
    in_audio_dim
    if out_audio_dim is None
    else out_audio_dim,
    quant_config=quant_config,
    prefix=f"{prefix}.audio_out_layer",
)

audio_rope_embeddings instance-attribute

audio_rope_embeddings = RoPE1D(
    head_dim_a, freqs_scaling=audio_freqs_scaling
)

in_audio_dim instance-attribute

in_audio_dim = in_audio_dim

in_visual_dim instance-attribute

in_visual_dim = in_visual_dim

is_multimodal instance-attribute

is_multimodal = is_multimodal

out_layer instance-attribute

out_layer = Kandinsky6OutLayer(
    model_dim,
    time_dim,
    out_visual_dim,
    patch_size,
    quant_config=quant_config,
    prefix=f"{prefix}.out_layer",
)

patch_size instance-attribute

patch_size = patch_size

pooled_text_embeddings instance-attribute

pooled_text_embeddings = Kandinsky6TextEmbeddings(
    in_text_dim2,
    time_dim,
    quant_config=quant_config,
    prefix=f"{prefix}.pooled_text_embeddings",
)

text_embeddings instance-attribute

text_embeddings = Kandinsky6TextEmbeddings(
    in_text_dim,
    model_dim,
    quant_config=quant_config,
    prefix=f"{prefix}.text_embeddings",
)

text_rope_embeddings instance-attribute

text_rope_embeddings = RoPE1D(head_dim)

text_token_padding instance-attribute

text_token_padding = text_token_padding

text_transformer_blocks instance-attribute

text_transformer_blocks = nn.ModuleList(
    [
        Kandinsky6TransformerEncoderBlock(
            model_dim,
            time_dim,
            ff_dim,
            head_dim,
            attention_engine,
            text_token_padding,
            quant_config=quant_config,
            prefix=f"{prefix}.text_transformer_blocks.{i}",
        )
        for i in range(num_text_blocks)
    ]
)

time_embeddings instance-attribute

time_embeddings = Kandinsky6TimeEmbeddings(
    model_dim,
    time_dim,
    quant_config=quant_config,
    prefix=f"{prefix}.time_embeddings",
)

visual_cond instance-attribute

visual_cond = visual_cond

visual_embeddings instance-attribute

visual_embeddings = Kandinsky6VisualEmbeddings(
    vis_in_dim,
    model_dim,
    patch_size,
    quant_config=quant_config,
    prefix=f"{prefix}.visual_embeddings",
)

visual_rope_embeddings instance-attribute

visual_rope_embeddings = RoPE3D(axes_dims)

visual_token_type_embeddings instance-attribute

visual_token_type_embeddings = nn.Embedding(
    self.visual_token_type_num_embeddings, model_dim
)

visual_token_type_num_embeddings instance-attribute

visual_token_type_num_embeddings = int(
    visual_token_type_num_embeddings or 0
)

clear_text_proj_cache

clear_text_proj_cache() -> None

Drop cached text/pooled projections (call once per generation).

forward

forward(
    x_video: Tensor | None,
    x_audio: Tensor | None,
    text_embed: Tensor | list[Tensor],
    pooled_text_embed: Tensor | list[Tensor],
    time: Tensor | list[Tensor],
    visual_rope: Tensor | None,
    audio_rope: Tensor | None,
    text_rope: Tensor | list[Tensor],
    sparse_params: dict | None = None,
    attention_mask: Tensor | None = None,
    visual_token_type_ids: Tensor | None = None,
) -> Tensor | tuple[Tensor, Tensor]

RoPE tensors are precomputed outside (pipeline / export); not from positions.

from_diffusers_config classmethod

from_diffusers_config(
    config: dict,
    *,
    quant_config: QuantizationConfig | None = None,
    prefix: str = "",
) -> Kandinsky6Transformer3DModel

Build the DiT from a Diffusers config.json dict.

reset_parameters

reset_parameters() -> None

KandinskyFlowMatchScheduler

Stateful Euler stepper over K6's shifted flow-matching sigma schedule.

timesteps holds the model-scale ([0, 1000]-ish) values fed to the DiT's time input; sigmas holds the same schedule unscaled (num_inference_steps + 1 entries, running 1.0 -> 0.0), needed to derive a step size without consuming a stateful step() call when one modality is stepped implicitly alongside another (see the vllm port's denoise_loop override).

config instance-attribute

config = SimpleNamespace(
    scheduler_scale=self.scheduler_scale
)

default_guidance_scale class-attribute instance-attribute

default_guidance_scale = 5.0

default_height class-attribute instance-attribute

default_height = 480

default_num_frames class-attribute instance-attribute

default_num_frames = 125

default_num_inference_steps class-attribute instance-attribute

default_num_inference_steps = 50

default_width class-attribute instance-attribute

default_width = 864

device instance-attribute

device = device

order class-attribute instance-attribute

order = 1

scheduler_scale instance-attribute

scheduler_scale = float(scheduler_scale)

sigmas instance-attribute

sigmas = torch.empty(0, dtype=torch.float32)

step_index property

step_index: int | None

timesteps instance-attribute

timesteps = torch.empty(0, dtype=torch.float32)

index_for_timestep

index_for_timestep(timestep: Tensor) -> int

set_timesteps

set_timesteps(
    num_inference_steps: int,
    device: device | str | None = None,
    **kwargs,
) -> None

step

step(
    model_output: Tensor,
    timestep: Tensor,
    sample: Tensor,
    return_dict: bool = True,
    **kwargs,
) -> KandinskyFlowMatchSchedulerOutput | tuple[Tensor]

KandinskyPiFlowScheduler

Bases: KandinskyFlowMatchScheduler

Pi-Flow sampler for Kandinsky6 Pro Distil checkpoints.

Distilled checkpoints predict a grid of denoising outputs per modality. Each outer model call uses that grid to construct a DX policy, then rolls the policy forward with the trained Pi-Flow integration schedule.

default_guidance_scale class-attribute instance-attribute

default_guidance_scale = 1.0

default_num_frames class-attribute instance-attribute

default_num_frames = 121

default_num_inference_steps class-attribute instance-attribute

default_num_inference_steps = 10

eps instance-attribute

eps = float(eps)

final_step_size_scale instance-attribute

final_step_size_scale = float(final_step_size_scale)

n_grid instance-attribute

n_grid = int(n_grid)

num_policy_substeps instance-attribute

num_policy_substeps = int(num_policy_substeps)

set_timesteps

set_timesteps(
    num_inference_steps: int,
    device: device | str | None = None,
    **kwargs,
) -> None

step

step(*args, **kwargs)

step_pair

step_pair(
    prediction_video: Tensor,
    prediction_audio: Tensor,
    timestep: Tensor,
    sample_video: Tensor,
    sample_audio: Tensor,
) -> tuple[Tensor, Tensor]

Apply one synchronized Pi-Flow step to the video and audio states.

get_kandinsky6_post_process_func

get_kandinsky6_post_process_func(
    od_config: OmniDiffusionConfig,
)

Factory returning the post-process function registered in vllm_omni/diffusion/registry.py's _DIFFUSION_POST_PROCESS_FUNCS.

Unpacks the (video, audio) pair set on DiffusionOutput.output by Kandinsky6TI2VAPipeline.forward into the flat {"video": ..., "audio": ..., "audio_sample_rate": ..., "fps": ...} payload the framework's shared audio-output plumbing (io_support.py -> output_formatter.py -> media_utils.py PyAV muxing) already expects — the same shape MiniMax H3's post-process function produces. output_formatter.normalize_diffusion_postprocess_output lifts audio_sample_rate/fps from this flat form into metadata; wrapping it in a {"payload", "metadata"} envelope instead would silently drop the sample rate (the envelope path only reads metadata["audio"]["sample_rate"]) and the MP4 would be muxed at the 24 kHz default, i.e. ~1.8x too slow.

postprocess_audio emits int16 PCM; the muxer (and the OpenAI video route) consume float32 in [-1, 1], so the waveform is rescaled here.

get_kandinsky6_pre_process_func

get_kandinsky6_pre_process_func(
    od_config: OmniDiffusionConfig,
)

Materialize checkpoint step defaults before StepScheduler admission.