Quantization¶

vLLM-Omni exposes quantization through the unified quantization_config path. The same configuration entrypoint is used across diffusion-only models, multi-stage omni/TTS models, and multi-stage diffusion models, but each model type has a different quantization scope.

Quantization Modes¶

Hardware Support¶

Legend: ✅ supported, ❌ unsupported, ⭕ not verified in this guide. FP8 on Ampere may use a weight-only path where available.

Model Type Support¶

Diffusion Model (Qwen-Image, Wan2.2)¶

These models run a diffusion transformer as the primary inference module. The default quantization target is the transformer; tokenizer, scheduler, text encoder, and VAE stay on the base checkpoint unless a method guide says otherwise.

Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)¶

These models combine an AR language model with audio, vision, talker, or TTS stages. Quantization is scoped to the AR language-model stage when the checkpoint contains a supported quantization_config; the non-AR stages stay in BF16 unless the model guide explicitly adds support.

Multi-Stage Diffusion Model (BAGEL, GLM-Image)¶

These models split generation across multiple stages. Quantization must be attached to the intended stage rather than applied globally.

Quantization Scope¶

Diffusion Model (Qwen-Image, Wan2.2)¶

The default target is the diffusion transformer. Component routing is available through build_quant_config():

from vllm_omni.quantization import build_quant_config

config = build_quant_config({
    "transformer": {"method": "fp8"},
    "vae": None,
})

Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)¶

Multi-Stage Diffusion Model (BAGEL, GLM-Image)¶

Python API¶

build_quant_config() accepts strings, dictionaries, per-component dictionaries, existing QuantizationConfig objects, or None.

from vllm_omni.quantization import build_quant_config

build_quant_config("fp8")
build_quant_config({"method": "fp8", "activation_scheme": "static"})
build_quant_config("auto-round", bits=4, group_size=128)
build_quant_config({"method": "gguf", "gguf_model": "/path/to/model.gguf"})
build_quant_config({"transformer": {"method": "fp8"}, "vae": None})
build_quant_config(None)

Output Similarity Comparison Tool¶

Use vllm_omni.quantization.tools.compare_diffusion_trajectory_similarity to compare a reference diffusion run with a quantized candidate run using the same prompt, seed, resolution, scheduler settings, and inference steps. The tool compares final decoded images or video frames, and also reports generation latency and worker-reported peak memory when available.

This is useful when validating whether online quantization, an offline pre-quantized checkpoint, or a new ignored_layers choice keeps generation quality close to the BF16 reference.

Online Quantization Example¶

python -m vllm_omni.quantization.tools.compare_diffusion_trajectory_similarity \
  --task t2i \
  --model Qwen/Qwen-Image \
  --candidate-quantization fp8 \
  --ignored-layers img_mlp \
  --prompt "a cup of coffee on the table" \
  --height 512 --width 512 \
  --num-inference-steps 20 \
  --seed 142 \
  --output-json /tmp/qwen_image_fp8_similarity/result.json \
  --save-output-dir /tmp/qwen_image_fp8_similarity/images \
  --enforce-eager

Offline Checkpoint Example¶

Use --candidate-model when the candidate is already quantized or lives at a different model path:

python -m vllm_omni.quantization.tools.compare_diffusion_trajectory_similarity \
  --task t2i \
  --reference-model Qwen/Qwen-Image \
  --candidate-model /path/to/qwen-image-fp8-checkpoint \
  --prompt "a cup of coffee on the table" \
  --height 512 --width 512 \
  --num-inference-steps 20 \
  --seed 142 \
  --output-json /tmp/qwen_image_fp8_checkpoint_similarity/result.json

If the checkpoint does not include a loadable quantization config, pass one explicitly:

--candidate-quantization-config-json '{"method":"fp8"}'

Output Metrics¶

The output JSON includes output_metrics, reference_generation, and candidate_generation.

Recommended starting thresholds for same-seed diffusion comparisons:

These thresholds are heuristics. Tune them by model family, task, resolution, quantization method, and deployment tolerance. For release gating, pair the numeric report with visual inspection of saved reference and candidate outputs.

The tool intentionally reports separate quality, latency, and memory metrics instead of a single consolidated similarity score. A single score can hide important tradeoffs, for example a candidate with good PSNR but a meaningful memory regression, or a candidate with low average error but localized visual artifacts. If you need a project-specific pass/fail gate, define it as an explicit policy over the individual metrics.

Pixel-level metrics do not measure semantic consistency. For higher-cost evaluation, you can complement this report with a vision-language judge that describes the reference and candidate outputs and compares those descriptions. Keep that semantic check separate from this lightweight tool so users can choose whether the additional model cost and latency are appropriate.