vllm_omni.diffusion.models.kandinsky6 ¶
Modules:
| Name | Description |
|---|---|
cache_accel | Kandinsky 6 cache adapters. |
kandinsky6_transformer | Generated standalone Kandinsky 6 vLLM-Omni transformer. |
modeling_kandinsky6_audio | Generated standalone Kandinsky 6 audio VAE component for vLLM-Omni. |
modeling_kandinsky6_vae | Generated standalone Kandinsky 6 video VAE component for vLLM-Omni. |
pipeline_kandinsky6 | Generated standalone Kandinsky 6 vLLM-Omni pipeline. |
scheduling_kandinsky6 | Native K6 shifted flow-matching Euler scheduler. |
super_resolution | Optional HD/Full-HD refinement after Kandinsky6 video/audio generation. |
AutoencoderKLHunyuanVideo ¶
Bases: ModelMixin, ConfigMixin, DistributedVaeMixin
A VAE model with KL loss for encoding videos into latents and decoding latent representations into videos. Introduced in HunyuanVideo.
This model inherits from [ModelMixin]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving).
decoder instance-attribute ¶
decoder = HunyuanVideoDecoder3D(
in_channels=latent_channels,
out_channels=out_channels,
up_block_types=up_block_types,
block_out_channels=block_out_channels,
layers_per_block=layers_per_block,
norm_num_groups=norm_num_groups,
act_fn=act_fn,
time_compression_ratio=temporal_compression_ratio,
spatial_compression_ratio=spatial_compression_ratio,
mid_block_add_attention=mid_block_add_attention,
)
encoder instance-attribute ¶
encoder = HunyuanVideoEncoder3D(
in_channels=in_channels,
out_channels=latent_channels,
down_block_types=down_block_types,
block_out_channels=block_out_channels,
layers_per_block=layers_per_block,
norm_num_groups=norm_num_groups,
act_fn=act_fn,
double_z=True,
mid_block_add_attention=mid_block_add_attention,
temporal_compression_ratio=temporal_compression_ratio,
spatial_compression_ratio=spatial_compression_ratio,
)
post_quant_conv instance-attribute ¶
quant_conv instance-attribute ¶
spatial_compression_ratio instance-attribute ¶
temporal_compression_ratio instance-attribute ¶
apply_tiling ¶
Applies tiling.
decode ¶
decode(
z: Tensor, return_dict: bool = True, **ignore_kwargs
) -> DecoderOutput | Tensor
Decode a batch of images.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
z | `torch.Tensor` | Input batch of latent vectors. | required |
return_dict | `bool`, *optional*, defaults to `True` | Whether to return a [ | True |
Returns:
| Type | Description |
|---|---|
DecoderOutput | Tensor | [ |
encode ¶
encode(
x: Tensor,
opt_tiling: bool = True,
return_dict: bool = True,
) -> (
AutoencoderKLOutput
| tuple[DiagonalGaussianDistribution]
)
Encode a batch of images into latents.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x | `torch.Tensor` | Input batch of images. | required |
return_dict | `bool`, *optional*, defaults to `True` | Whether to return a [ | True |
Returns:
| Type | Description |
|---|---|
AutoencoderKLOutput | tuple[DiagonalGaussianDistribution] | The latent representations of the encoded videos. If |
AutoencoderKLOutput | tuple[DiagonalGaussianDistribution] | [ |
AutoencoderKLOutput | tuple[DiagonalGaussianDistribution] | otherwise a plain |
forward ¶
forward(
sample: Tensor,
sample_posterior: bool = False,
return_dict: bool = True,
generator: Generator | None = None,
) -> DecoderOutput | Tensor
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample | `torch.Tensor` | Input sample. | required |
sample_posterior | `bool`, *optional*, defaults to `False` | Whether to sample from the posterior. | False |
return_dict | `bool`, *optional*, defaults to `True` | Whether or not to return a [ | True |
get_dec_optimal_tiling ¶
get_dec_optimal_tiling(
shape: list[int], device: device | str | None = None
) -> tuple[tuple[int, int, int, int], tuple[int, int, int]]
Returns optimal tiling for given shape.
get_enc_optimal_tiling ¶
get_enc_optimal_tiling(
shape: list[int], device: device | str | None = None
) -> tuple[tuple[int, int, int, int], tuple[int, int, int]]
Returns optimal tiling for given shape.
tiled_decode ¶
tiled_decode(
z: Tensor, return_dict: bool = True
) -> DecoderOutput | Tensor
Decode a batch of images using a tiled decoder.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
z | `torch.Tensor` | Input batch of latent vectors. | required |
return_dict | `bool`, *optional*, defaults to `True` | Whether or not to return a [ | True |
Returns:
| Type | Description |
|---|---|
DecoderOutput | Tensor | [ |
tiled_encode ¶
Encode a batch of images using a tiled encoder.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x | `torch.Tensor` | Input batch of videos. | required |
Returns:
| Type | Description |
|---|---|
AutoencoderKLOutput |
|
Kandinsky6AudioVAE ¶
Bases: ModelMixin, ConfigMixin
vLLM-Omni-serializable K6 audio VAE, mel frontend, and vocoder.
The conversion-only checkpoint paths are accepted for materializing a bundle, while vocoder_config makes a saved bundle self-contained.
ignore_for_config class-attribute instance-attribute ¶
native instance-attribute ¶
native = FeaturesUtils(
tod_vae_ckpt=None
if tod_vae_ckpt is None
else str(tod_vae_ckpt),
bigvgan_vocoder_ckpt=None
if bigvgan_vocoder_ckpt is None
else str(bigvgan_vocoder_ckpt),
vocoder_config=vocoder_config,
mode=mode,
need_vae_encoder=need_vae_encoder,
need_vae_decoder=need_vae_decoder,
scaling_factor=scaling_factor,
).eval()
Kandinsky6TI2VAPipeline ¶
Bases: Module, CFGParallelMixin, ProgressBarMixin, DiffusionPipelineProfilerMixin, SupportsComponentDiscovery, SupportAudioOutput, SupportImageInput
vLLM-Omni pipeline for text/image-to-video-and-audio generation with Kandinsky 6.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
transformer | Module | None | Multimodal K6 transformer used to denoise video and audio latents ( | None |
vae | Module | None | Video VAE used to decode generated video latents ( | None |
text_encoder | Module | None | Qwen2.5-VL text encoder for token-level embeddings. | None |
audio_vae | Module | None | Audio VAE used to decode generated audio latents ( | None |
scheduler | object | None | Flow-matching scheduler used by the denoising loop ( | None |
tokenizer | object | None | Qwen2.5-VL processor. | None |
text_encoder_2 | Module | None | CLIP text encoder for pooled embeddings. | None |
tokenizer_2 | object | None | CLIP tokenizer. | None |
default_guidance_scale instance-attribute ¶
default_guidance_scale = float(
getattr(
scheduler,
"default_guidance_scale",
KandinskyFlowMatchScheduler.default_guidance_scale,
)
)
default_height instance-attribute ¶
default_height = int(
getattr(
scheduler,
"default_height",
KandinskyFlowMatchScheduler.default_height,
)
)
default_num_frames instance-attribute ¶
default_num_frames = int(
getattr(
scheduler,
"default_num_frames",
KandinskyFlowMatchScheduler.default_num_frames,
)
)
default_num_inference_steps class-attribute instance-attribute ¶
default_num_inference_steps = int(
getattr(
scheduler,
"default_num_inference_steps",
self.default_num_inference_steps,
)
)
default_width instance-attribute ¶
default_width = int(
getattr(
scheduler,
"default_width",
KandinskyFlowMatchScheduler.default_width,
)
)
denoise_step ¶
One CFG-combined video velocity. Audio velocity is stashed on the request.
diffuse ¶
diffuse(
*,
bundle: LatentBundle,
positive: TextEmbeds,
negative: TextEmbeds,
positive_cu: Tensor,
negative_cu: Tensor,
positive_mask: Tensor | None,
negative_mask: Tensor | None,
visual_rope: Tensor,
audio_rope: Tensor | None,
num_inference_steps: int,
guidance_scale: float,
first_frames: Tensor | None,
visual_cond_scheme: str,
sample_audio: bool,
visual_token_type_ids: Tensor | None,
device: device,
) -> LatentBundle
load_weights ¶
Load each Hub folder into its module.
The framework yields transformer.*, vae.*, text_encoder.*, text_encoder_2.*, and audio_vae.* from weights_sources.
post_decode ¶
post_decode(state: Any, **kwargs: Any) -> DiffusionOutput
Decode the finished video and audio latents.
predict_noise ¶
One CFG branch. forward is the denoise loop's DiT call.
prepare_encode ¶
Encode one request and store everything the denoise step needs.
Kandinsky6Transformer3DModel ¶
Bases: Module
Kandinsky 6 DiT — handles T2V (is_multimodal=False) and T2VA (is_multimodal=True).
Tensor-parallel port of core/components/dit.py::DiffusionTransformer3D, with submodule names/nesting matching the Diffusers-exported checkpoint layout (see module docstring) so from_pretrained loads a convert_checkpoint.py --use-patched-diffusers bundle directly.
The class is a plain nn.Module. from_diffusers_config reads the bundle config.json. Safetensors from transformer/ are applied later by the pipeline load_weights path, which narrows full checkpoint tensors onto each rank's ColumnParallelLinear / RowParallelLinear shard.
audio_embeddings instance-attribute ¶
audio_embeddings = Kandinsky6TextEmbeddings(
in_audio_dim,
model_dim_a,
quant_config=quant_config,
prefix=f"{prefix}.audio_embeddings",
)
audio_out_layer instance-attribute ¶
audio_out_layer = Kandinsky6OutLayerAudio(
model_dim_a,
time_dim_a,
in_audio_dim
if out_audio_dim is None
else out_audio_dim,
quant_config=quant_config,
prefix=f"{prefix}.audio_out_layer",
)
audio_rope_embeddings instance-attribute ¶
audio_rope_embeddings = RoPE1D(
head_dim_a, freqs_scaling=audio_freqs_scaling
)
out_layer instance-attribute ¶
out_layer = Kandinsky6OutLayer(
model_dim,
time_dim,
out_visual_dim,
patch_size,
quant_config=quant_config,
prefix=f"{prefix}.out_layer",
)
pooled_text_embeddings instance-attribute ¶
pooled_text_embeddings = Kandinsky6TextEmbeddings(
in_text_dim2,
time_dim,
quant_config=quant_config,
prefix=f"{prefix}.pooled_text_embeddings",
)
text_embeddings instance-attribute ¶
text_embeddings = Kandinsky6TextEmbeddings(
in_text_dim,
model_dim,
quant_config=quant_config,
prefix=f"{prefix}.text_embeddings",
)
text_transformer_blocks instance-attribute ¶
text_transformer_blocks = nn.ModuleList(
[
Kandinsky6TransformerEncoderBlock(
model_dim,
time_dim,
ff_dim,
head_dim,
attention_engine,
text_token_padding,
quant_config=quant_config,
prefix=f"{prefix}.text_transformer_blocks.{i}",
)
for i in range(num_text_blocks)
]
)
time_embeddings instance-attribute ¶
time_embeddings = Kandinsky6TimeEmbeddings(
model_dim,
time_dim,
quant_config=quant_config,
prefix=f"{prefix}.time_embeddings",
)
visual_embeddings instance-attribute ¶
visual_embeddings = Kandinsky6VisualEmbeddings(
vis_in_dim,
model_dim,
patch_size,
quant_config=quant_config,
prefix=f"{prefix}.visual_embeddings",
)
visual_token_type_embeddings instance-attribute ¶
visual_token_type_num_embeddings instance-attribute ¶
visual_token_type_num_embeddings = int(
visual_token_type_num_embeddings or 0
)
clear_text_proj_cache ¶
Drop cached text/pooled projections (call once per generation).
forward ¶
forward(
x_video: Tensor | None,
x_audio: Tensor | None,
text_embed: Tensor | list[Tensor],
pooled_text_embed: Tensor | list[Tensor],
time: Tensor | list[Tensor],
visual_rope: Tensor | None,
audio_rope: Tensor | None,
text_rope: Tensor | list[Tensor],
sparse_params: dict | None = None,
attention_mask: Tensor | None = None,
visual_token_type_ids: Tensor | None = None,
) -> Tensor | tuple[Tensor, Tensor]
RoPE tensors are precomputed outside (pipeline / export); not from positions.
from_diffusers_config classmethod ¶
from_diffusers_config(
config: dict,
*,
quant_config: QuantizationConfig | None = None,
prefix: str = "",
) -> Kandinsky6Transformer3DModel
Build the DiT from a Diffusers config.json dict.
KandinskyFlowMatchScheduler ¶
Stateful Euler stepper over K6's shifted flow-matching sigma schedule.
timesteps holds the model-scale ([0, 1000]-ish) values fed to the DiT's time input; sigmas holds the same schedule unscaled (num_inference_steps + 1 entries, running 1.0 -> 0.0), needed to derive a step size without consuming a stateful step() call when one modality is stepped implicitly alongside another (see the vllm port's denoise_loop override).
set_timesteps ¶
step ¶
step(
model_output: Tensor,
timestep: Tensor,
sample: Tensor,
return_dict: bool = True,
**kwargs,
) -> KandinskyFlowMatchSchedulerOutput | tuple[Tensor]
KandinskyPiFlowScheduler ¶
Bases: KandinskyFlowMatchScheduler
Pi-Flow sampler for Kandinsky6 Pro Distil checkpoints.
Distilled checkpoints predict a grid of denoising outputs per modality. Each outer model call uses that grid to construct a DX policy, then rolls the policy forward with the trained Pi-Flow integration schedule.
set_timesteps ¶
get_kandinsky6_post_process_func ¶
get_kandinsky6_post_process_func(
od_config: OmniDiffusionConfig,
)
Factory returning the post-process function registered in vllm_omni/diffusion/registry.py's _DIFFUSION_POST_PROCESS_FUNCS.
Unpacks the (video, audio) pair set on DiffusionOutput.output by Kandinsky6TI2VAPipeline.forward into the flat {"video": ..., "audio": ..., "audio_sample_rate": ..., "fps": ...} payload the framework's shared audio-output plumbing (io_support.py -> output_formatter.py -> media_utils.py PyAV muxing) already expects — the same shape MiniMax H3's post-process function produces. output_formatter.normalize_diffusion_postprocess_output lifts audio_sample_rate/fps from this flat form into metadata; wrapping it in a {"payload", "metadata"} envelope instead would silently drop the sample rate (the envelope path only reads metadata["audio"]["sample_rate"]) and the MP4 would be muxed at the 24 kHz default, i.e. ~1.8x too slow.
postprocess_audio emits int16 PCM; the muxer (and the OpenAI video route) consume float32 in [-1, 1], so the waveform is rescaled here.
get_kandinsky6_pre_process_func ¶
get_kandinsky6_pre_process_func(
od_config: OmniDiffusionConfig,
)
Materialize checkpoint step defaults before StepScheduler admission.