Wan VAE Encoder and Decoder Fast Paths¶
Video pipelines built on the Wan 2.1/2.2 causal VAE (Cosmos3, Wan2.2, LingBot, Helios, SANA-Video, LongCat-Video, DreamZero) spend a large share of their end-to-end time decoding latents to pixels. The decoder is a chunked 3D convolutional network whose reference implementation in diffusers moves every activation through memory several times per layer (normalization, activation, causal padding, feature-cache bookkeeping, shortcut upsampling). vLLM-Omni installs a fast path on every loaded Wan VAE that fuses this data movement into a handful of Triton kernels. The opt-in channels_last level also specializes first-frame convolutions that have no history.
The same fast path also covers image models whose VAE is a Wan VAE under other class names: Qwen-Image (and Edit, Anima, Krea2), Qwen-Image-Layered and Qwen-Image-2.1. See Wan-family VAEs.
Decoder levels¶
The fast path is controlled by --vae-fast-path (engine argument vae_fast_path, deploy-config key vae_fast_path):
| Level | Default | Output vs. diffusers | What it does |
|---|---|---|---|
lossless | yes | bit-identical | Fused RMSNorm epilogue, fused causal-conv input assembly and cache refresh, fused shortcut upsampling and residual adds (with the neighbouring convolution biases folded in), single-pass nearest 2x upsampling, preallocated output assembly. |
channels_last | no | within tolerance (PSNR typically > 60 dB) | Everything in lossless, plus channels-last convolution weights, a single-pass channels-last RMSNorm+SiLU kernel that also absorbs the preceding conv1 bias, and Conv2d for supported single-frame causal convolutions without history. |
off | no | bit-identical | Reference diffusers decoder. |
# Default: bit-exact fast path
vllm serve nvidia/Cosmos3-Nano --omni
# Fastest: channels-last decoder (not bit-exact)
vllm serve nvidia/Cosmos3-Nano --omni --vae-fast-path channels_last
# Reference implementation, e.g. to bisect a quality issue
vllm serve nvidia/Cosmos3-Nano --omni --vae-fast-path off
The equivalent per-stage deploy configuration is:
Behavior and limitations¶
- Only CUDA and XPU are supported. On other platforms the reference decoder runs.
- The fast path is installed once per VAE instance when the pipeline is initialized. It rebinds the forwards of the loaded decoder modules; parameter names,
state_dictkeys and weight loading are unchanged. - Installation is skipped, with a logged reason, if a custom forward wrapper or forward hook would be replaced or bypassed by the fused paths. Wrappers and hooks on modules that the fast path still calls normally are preserved.
- Every fused kernel validates its inputs and falls back to the exact PyTorch expression for anything it does not support (unusual dtypes or layouts, CPU tensors, autograd enabled,
torch.compiletracing). - The SiLU activation is folded into the normalization kernel only after an exhaustive self-test over all bf16/fp16 values proves the fused epilogue bit-identical to
F.siluon the running toolkit; otherwise SiLU stays a separate operation. The startup log reportsfused_silu=.... - Spatially sharded VAE decode (
--vae-parallel-mode spatial_shard_heightorspatial_shard_width) is not combined with the fast path; the installer skips the VAE and logs why. Tiled and tile-parallel decode work unchanged. - The
channels_lastlevel changes the order in which cuDNN accumulates convolutions, so outputs differ from the reference in the last bits. Uselosslesswhen bitwise reproducibility against diffusers matters. channels_lastalso skips internal decoder bitwise padding checks to reduce first-decode peak memory. Lossless decoding retains all of its checks.- For a supported causal convolution with one input frame and no history, the temporal input is
[0, 0, x]. Thechannels_lastlevel uses Conv2d with the last temporal weight slice, preserving the convolution formula for finite inputs and the existing cache update. Calls with history retain Conv3d. The weight slice is taken on each call, so weight updates and device moves do not leave a stale packed copy. - For VAEs kept in fp32, cuDNN's channels-last convolution algorithms run in TF32 under PyTorch's default
torch.backends.cudnn.allow_tf32 = True, which dominates the difference to the reference (about 1e-3). Settorch.backends.cudnn.allow_tf32 = Falseif full fp32 convolutions are required.
Cosmos3 encoder¶
--vae-encode-fast-path selects the encoder level independently of the decoder. The engine/deploy-config field is vae_encode_fast_path. The supported target is Cosmos3's residual Wan2.2 VAE: 2x2 patchification, 4x temporal compression, 16x spatial compression, and four residual downsampling stages. The real Cosmos3 encoder uses 160/320/640 channels and produces 48 latent channels. Other Wan architectures retain their reference encoding path.
| Encoder level | Default | Behavior |
|---|---|---|
lossless | yes | Reuses the exact normalization epilogue, causal input/cache assembly and bias/residual fusions. Fuses spatial pad/input assembly and temporal downsampling cache copies. Keeps the reference shortcut averaging reduction. |
channels_last | no | Also converts encoder convolution weights to channels-last, uses single-pass normalization and fuses shortcut averaging with residual addition. Outputs can differ in the last bits. |
off | no | Reference encoding. |
For eligible cached encoder convolutions, normalization/SiLU writes directly into the temporally assembled convolution input and the next two-frame cache. Already-normalized history is copied unchanged; missing history stays zero. lossless retains ATen's reduction and the existing intermediate rounding, and only folds SiLU for dtypes that pass the installation-time exactness probe. channels_last retains its existing single-pass normalization math, except that for BF16 the encoder keeps the reference's intermediate roundings (the decoder keeps fast math); this fusion does not change its quality gates. Unsupported layouts, non-identity or hooked dropout, and rejected spatial-padding probes use the existing separate path. This fusion is encoder-only; decoder dispatch is unchanged.
Untiled encoding patchifies one temporal chunk at a time and writes encoder features into a preallocated buffer. The schedule remains one initial frame, then four frames per chunk. quant_conv runs once after assembly. Tiled encoding retains its existing per-chunk quant_conv, tile coordinates and blending. Both image conditioning and video conditioning use this path through the ordinary vae.encode(...).latent_dist interface.
# Fast encoder, reference decoder
vllm serve nvidia/Cosmos3-Nano --omni \
--vae-encode-fast-path channels_last --vae-fast-path off
# Reference encoder and decoder
vllm serve nvidia/Cosmos3-Nano --omni \
--vae-encode-fast-path off --vae-fast-path off
Encoder installation has independent reports and rollback/uninstall state. It preserves weights and parameter names, and rejects custom forwards or hooks that a fusion would bypass. A substituted RMSNormVAE keeps its own forward and numerical semantics. Autograd and compilation use PyTorch fallbacks. The encoder can coexist with spatial-sharded decoding, since the two installers modify different modules; spatial tiling remains the encoder's parallel mode.
The Python entry points are install_wan_vae_encoder_fastpath, is_encoder_installed, and uninstall_wan_vae_encoder_fastpath, exported from vllm_omni.diffusion.distributed.autoencoders.wan_vae_fastpath. Uninstall restores original forwards and tensor layouts while retaining current weight values.
Encoder benchmarking¶
benchmarks/diffusion/bench_wan_vae_encode.py loads only the VAE. It compares all encoder levels with the same weights, inputs and backend settings, always setting up and validating against off first. It reports startup/first-call time separately from warmed median latency, throughput and peak allocated memory. Reference tensors are kept on CPU during candidate timing. Console output includes timing and per-tensor quality tables with explicit PASS/FAIL/OOM/NO REF statuses; speedups are still shown when numerical validation fails. --json saves the full metrics and per-level errors lists.
# Image conditioning
python benchmarks/diffusion/bench_wan_vae_encode.py \
--model nvidia/Cosmos3-Nano --frames 1 --profile --json encode-image.json
# Video encoding and quality; --revision can pin the checkpoint
python benchmarks/diffusion/bench_wan_vae_encode.py \
--model nvidia/Cosmos3-Nano --frames 93 --check-reconstruction --json encode-video.json
# A preprocessed natural image/video: floating RGB [B, 3, T, H, W] in [-1, 1]
python benchmarks/diffusion/bench_wan_vae_encode.py \
--input pixels.pt --check-reconstruction --json encode-natural.json
# Two GPUs, existing tile-parallel encoding
torchrun --nproc-per-node 2 benchmarks/diffusion/bench_wan_vae_encode.py \
--vae-patch-parallel-size 2 --frames 93 --json encode-tiled.json
The default is BF16, 1280x720, three warmups and ten timed iterations. --tiny uses a reduced four-stage VAE for development; production performance measurements must use real weights. --tf32 off controls FP32 convolution math. The profiler reports CUDA operator times, including padding, copying, normalization, reduction, attention and convolutions.
Power- and thermally-limited GPUs (for example H100 NVL, RTX 5090 and RTX PRO 6000) run the same kernels more than 2x slower after sustained load, so timing the levels one after another favors whichever level runs first. The default --schedule interleaved keeps every level's VAE resident and runs one encode per level in each warmup and timed round, rotating the first level from round to round. Peak memory excludes the other levels' resident weights. --schedule sequential loads and times one level at a time, for GPUs that cannot hold all levels at once. During timed encodes, NVML samples the SM clock, power, temperature and clock-event (throttle) reasons every --telemetry-interval-ms (default 50; 0 disables). They are reported per level and per iteration. The console warns when median SM clocks differ by more than 3% across levels or when the GPU throttled. --lock-sm-clock-mhz pins the SM clock for the run and resets it on exit; this requires administrator rights.
# Comparable timings on a power-limited GPU (clock lock needs root)
python benchmarks/diffusion/bench_wan_vae_encode.py \
--model nvidia/Cosmos3-Nano --frames 93 --lock-sm-clock-mhz 1500 --json encode-locked.json
The benchmark exits unsuccessfully for a lossless bitwise mismatch, nonfinite outputs, or channels-last normalized RMSE above 1% for posterior parameters, mean or log-variance. --check-reconstruction also requires at least 50 dB PSNR between reconstructions of the reference and candidate mean, using the same reference decoder. These are validation gates, not measured encoder results. Single-GPU OOM cases are recorded explicitly; distributed failures terminate the run because a failed collective cannot be retried safely on one rank. Failure messages identify the level, tensor, measured metric and required threshold. OOM messages identify the failing phase (such as timed encoding or reference-decoder reconstruction), input shape and dtype, and retain the original allocation error. If the reference fails, remaining timings are marked NO REF rather than presented as validated results.
Validate on both H100 and GB200, using 256x256, 640x384 and 1280x720 inputs with 1, 5, 33, 93 and 189 frames. Record repeatable gains separately for images and videos, and investigate latency or peak-memory regressions above 5%. The new encoder kernel launch sizes are conservative defaults; architecture-specific tuning and the numerical gates require validation on those GPUs.
Wan-family VAEs¶
The installer finds which registered family a VAE belongs to and binds the same replacement forwards to that family's classes (vllm_omni/diffusion/distributed/autoencoders/wan_vae_fastpath/families.py). --vae-fast-path (decoder) and --vae-encode-fast-path (encoder) apply to all of them:
| Family | VAE class | Pipelines | Notes |
|---|---|---|---|
wan | diffusers AutoencoderKLWan | Wan2.1/2.2, Cosmos3, LingBot, Helios, ... | Everything above; the Cosmos3 encoder path is Wan-only. |
qwen_image | diffusers AutoencoderKLQwenImage | Qwen-Image, Qwen-Image-Edit, Anima, Krea2 | The decoder's up blocks take no first_chunk. |
qwen_image_layered | vLLM-Omni AutoencoderKLQwenImage (models/qwen_image/) | Qwen-Image-Layered | Its RMS norm skips the fp32 upcast, so it keeps its own forward. |
qwen_image_21 | vLLM-Omni AutoencoderKLQwenImage21 | Qwen-Image-2.1 | Causal convs are Conv2d on a single frame; channels_last also replaces SDPA in the mid-block attention (head dim 768/1152; fp32 math on XPU, the memory-efficient kernel on CUDA) with faster chunked bf16 matmuls whose scores are rounded to bf16 before the fp32 softmax. |
For the non-Wan families the encoder fast path binds the causal convolutions, residual blocks and RMS norms; the Cosmos3-specific encoder forwards are not used. A single-frame Conv2d convolution moves its zero padding into the convolution only for shapes where a first-use probe shows identical bits (otherwise it keeps F.pad), like the Conv3d paths.
Another Wan-style VAE gets the fast path by registering its classes, without copying any forward:
from vllm_omni.diffusion.distributed.autoencoders.wan_vae_fastpath import WanVaeFamily, register_family
register_family(
WanVaeFamily(
name="my_vae",
vaes=(MyAutoencoder,),
decoder=MyDecoder3d,
encoder=MyEncoder3d,
causal_conv=MyCausalConv3d,
rms_norm=MyRMSNorm, # None if its numerics differ from WanRMS_norm
resample=MyResample,
upsample=MyUpsample,
residual_block=MyResidualBlock,
residual_up_block=MyResidualUpBlock, # residual VAEs only
dup_up=MyDupUp3D,
attention_block=MyAttentionBlock,
)
)
Module types are matched exactly, and a family's classes must have the Wan forwards' behavior up to names: add a bitwise lossless test, as tests/diffusion/distributed/test_wan_vae_fastpath_families.py does for the built-in families.
A VAE gets the family registered for its most specific class (a family for a subclass of AutoencoderKLWan wins over wan), and the built-in families do not replace a family of the same name registered before them. Wan-style VAEs that are not registered, and so run their own forwards, include models/lance/wan_vae.py and models/magi2/turbo_vae.py.
The first-use bitwise probes allocate full-size scratch tensors. A probe runs only when its scratch fits twice over in the free device memory (counting memory the allocator has reserved but not handed out); otherwise, or if it runs out of memory, that call uses the reference formulation, nothing is recorded, and the shape is probed again on a later call. A declined spatial-padding probe is decided before the time-assembled input is built, so the call never holds more memory than off. probe_stats() (from wan_vae_fastpath) counts recorded verdicts and declined probes.
Measured on one Intel Arc Pro B70 (bf16, real weights, a natural image encoded at each size and its latent decoded; median of 5 after warmup; benchmarks/diffusion/bench_wan_family_vae.py), in ms:
| VAE | Size | Decode off | lossless | channels_last | Encode off | lossless | channels_last |
|---|---|---|---|---|---|---|---|
| Qwen-Image | 1024x1024 | 251.8 | 170.7 | 81.0 (43.6 dB) | 153.0 | 105.3 | 148.5 |
| Qwen-Image | 1328x1328 | 468.8 | 311.6 | 152.5 | 289.4 | 199.4 | 270.7 |
| Qwen-Image-2.1 | 1024x1024 | 308.7 | 233.4 | 237.2 (63.5 dB) | 77.4 | 59.9 | 87.8 |
| Qwen-Image-2.1 | 1328x1328 | 571.0 | 460.4 | 467.1 (64.0 dB) | 142.3 | 117.6 | 161.4 |
| Qwen-Image-2.1 | 2048x2048 | 1286.0 | 986.5 | 942.9 (65.0 dB) | 333.9 | 280.6 | 394.5 |
lossless is bitwise identical to off at 1024x1024 and for Qwen-Image-2.1 at every size. At 1328x1328 on this XPU stack (torch 2.14.0+xpu), the reference decode of the Wan 2.1-structured VAEs (Wan and Qwen-Image alike) is itself wrong: inside a full decode, the upsampler's fp32 round trip (F.interpolate(x.float(), ...) at 664x664 -> 1328x1328) writes wrong values into the first 524288 output elements, while the fast path's bf16 upsample matches ground truth (repeat_interleave). The same call in isolation is correct, so this is a state-dependent XPU issue outside the fast path.
channels_last decode reaches only 43.6 dB for Qwen-Image on this image (58.7 dB for the Wan 2.1 VAE on the same image and card). The figure depends on the image: on tests/assets/qwen_image_edit/qwen_image_edit_2511_test1.png the same benchmark and card give 60.8 dB at 1024x1024 and 61.4 dB at 1328x1328. Prefer lossless for Qwen-Image where fidelity matters. On this card channels_last gives Qwen-Image-2.1 no decode gain and a slower encode.
Fast paths for other VAEs¶
--vae-fast-path and --vae-encode-fast-path are not Wan-specific. At model load, vllm_omni.diffusion.vae_fast_path.apply_vae_fast_paths gives each pipeline VAE the most recently registered VaeFastPath that matches it, and installs its decoder and its encoder at the configured levels. The Wan-family fast path above is the built-in entry (wan_family, CUDA and XPU). A VAE with a different architecture gets its own entry instead of its own switches:
from vllm_omni.diffusion.vae_fast_path import VaeFastPathContext, register_vae_fast_path
class MyVaeFastPath:
name = "my_vae"
def matches(self, vae) -> bool:
return isinstance(vae, MyVideoVAE)
def supported(self, ctx: VaeFastPathContext) -> bool:
return ctx.platform.is_xpu()
def install(self, vae, component, level, ctx):
# component is "decoder" or "encoder"; level is never "off".
# Return an object with `installed` and `reason`.
...
register_vae_fast_path(MyVaeFastPath())
Every entry keeps the level contract: lossless must be bitwise identical to the reference forwards, and anything that changes values (a different attention kernel, torch.compile, a lower-precision weight copy) belongs to channels_last or must stay off. An entry that cannot honour a level reports it as not installed. ctx.compile_allowed() applies the DiT's compile precedence (--enforce-eager vetoes it, then the platform must support torch inductor), so an entry that compiles must check it. An install failure is logged and leaves that component on its reference forwards.
Measured decoder performance¶
Cosmos3-Nano VAE, 1280x720 x 189 frames, bf16, one GB200 GPU, bench_wan_vae_decode.py (best of 2 runs after warmup):
--vae-fast-path | Decode time | Speedup | Output vs. off |
|---|---|---|---|
off | 6.00 s | 1.00x | reference |
lossless | 3.26 s | 1.84x | bit-identical |
channels_last | 2.51 s | 2.39x | PSNR 62.6 dB, max abs diff 4.6e-2 |
Peak decode memory was unchanged (about 13 GiB): it is set by the largest activations and the cuDNN workspace, not by the output assembly.
Decoder benchmarking¶
benchmarks/diffusion/bench_wan_vae_decode.py decodes seeded latents with the real VAE at each level and reports decode time, peak memory, bitwise equality and PSNR against the off level. After --warmup decodes it times --iters decodes and prints their median, min, max, standard deviation and every raw sample; the speedup compares medians. --tiling decodes with the VAE's spatial tiling on one GPU. --profile prints a per-kernel table with the convolution share and any layout-transpose kernels:
python benchmarks/diffusion/bench_wan_vae_decode.py --model nvidia/Cosmos3-Nano \
--size 1280x720 --frames 189 --fast-path off,lossless,channels_last --profile
For a single-GPU comparison that isolates first-frame Conv2d within channels_last, use --first-frame-ablation. It alternates the two paths, disables cuDNN TF32 and autotuning, and prints JSON with every timing sample, summary statistics, peak allocated memory and output comparisons. Per-call weight packing is included in the measured decode time.
python benchmarks/diffusion/bench_wan_vae_decode.py \
--model Wan-AI/Wan2.1-T2V-1.3B-Diffusers --size 832x480 --frames 1 \
--dtype bf16 --first-frame-ablation --warmup 60 --iters 50 --seed 0
The default input is a seeded latent tensor. --latents /path/to/latents.pt can instead load a saved VAE input after latent mean/std scaling; the benchmark selects the temporal prefix requested by --frames and checks its shape.
The same script benchmarks multi-GPU VAE decode when launched with torchrun; the decode is timed across all ranks and rank 0 reports: