LoRA (Low-Rank Adaptation) Guide¶

LoRA (Low-Rank Adaptation) enables fine-tuning diffusion models by adding trainable low-rank matrices to existing model weights. vLLM-Omni supports two LoRA backends: PEFT for PEFT-style adapters, and Distill usually for few-steps inference. PEFT backend allows you to customize model behavior without modifying the base model weights. Distill backend fuses LoRA weights into the base model at initialization.

Overview¶

LoRA adapters are lightweight, model-specific fine-tuning weights that can be applied to diffusion models in two ways:

• PEFT backend (--lora-backend peft, default): Loads a PEFT-format adapter folder via DiffusionLoRAManager. Adapters are cached (LRU) and activated per request via LoRARequest. It uses a unified LoRA handling mechanisms similar to vLLM with LRU cache management.
• Distill backend (--lora-backend distill): Calls pipeline.load_lora_weights once at initialization to fuse one or more concrete checkpoint files directly into the base weights. Typically used for distilled few-step LoRAs (e.g. Lightning, LightX2V).

Global merged policy¶

--lora-application merged with --lora-backend peft merges a single startup --lora-path into unquantized dense weights. Requests use the merged policy without attaching a LoRARequest; per-request adapter selection is rejected. Removing the adapter restores the exact base weights. Runtime application is the default. Fixed-model adapters, MoE, HSDP and inference weight offloading are not supported by this generic path.

For adapter-only RL updates and memory/validation requirements, see engine-side LoRA merge.

LoRA Adapter Format¶

PEFT (Parameter-Efficient Fine-Tuning) format (default)¶

A typical PEFT-format LoRA adapter directory structure:

lora_adapter/
├── adapter_config.json
└── adapter_model.safetensors

The adapter_config.json file contains metadata about the LoRA adapter, including:

• r: LoRA rank
• lora_alpha: LoRA alpha scaling factor
• target_modules: List of module names to apply LoRA to

Every module supplied by a PEFT adapter must bind successfully when the adapter is activated. If any module cannot bind, activation raises an error naming the unbound modules and resets the LoRA layers to avoid leaving the adapter partly active. An adapter may still target only some projections of a fused layer, such as Q and V without K, as long as all supplied modules bind.

Previously, some adapters activated even when unmatched modules were silently dropped. Those adapters now fail at activation. For example, in-memory tensor adapters for Qwen-Image, Boogu-Image, and Wan can contain checkpoint keys such as to_out.0 where the engine expects to_out. PEFT checkpoints with extra unbound keys also fail at activation if those keys pass the loader's earlier validation; unsupported module suffixes may already fail during loading. Checkpoint-to-engine name mapping is tracked in #8001; until it is available, adapter keys must resolve to supported engine modules.

LoRA scale semantics¶

The request-level lora_scale is a multiplier applied on top of the adapter's own PEFT scaling, not a replacement for it. The applied weight delta is:

delta = lora_scale × (lora_alpha / r) × (B @ A)

At load time the lora_alpha / r factor is folded into the adapter weights (upstream LoRALayerWeights.optimize); lora_scale is applied when the adapter is activated. This is the same convention as diffusers' cross_attention_kwargs={"scale": s} and LightX2V's --lora-scale: to reproduce a reference invocation, pass the same numeric value here.

Do not pass the already-multiplied effective strength scale × alpha / rank. That applies alpha / r a second time and shrinks the adapter by that factor.

Quick Start¶

Offline Inference¶

PEFT backend: pre-loaded LoRA¶

Load a PEFT-format LoRA adapter at initialization. The adapter is pre-loaded into the cache and can be activated per request:

from vllm_omni import Omni
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
from vllm_omni.lora.request import LoRARequest

lora_path = "/path/to/lora_adapter"

omni = Omni(
    model="stabilityai/stable-diffusion-3.5-medium",
    lora_path=lora_path,
    lora_backend="peft",  # default, can be omitted
)

lora_request = LoRARequest(
    lora_name="preloaded",
    lora_int_id=1,
    lora_path=lora_path
)

outputs = omni.generate(
    "A piece of cheesecake",
    OmniDiffusionSamplingParams(
        lora_request=lora_request,
        lora_scale=2.0,  # optional arg, default 1.0
    ),
)

PEFT backend: stacking multiple LoRAs in one request¶

A single request can activate several LoRA adapters at once, each with its own scale. The adapters are composed into a single adapter whose per-layer rank is the sum of the individual ranks; the composed adapter is cached and activated like any other adapter, and requests with different LoRA combinations (or different scales) are scheduled into separate batches.

Constraints:

• The summed rank of the stacked adapters must not exceed 512 per layer. Two or three typical LoRAs (rank 4-64) fit comfortably.
• Adapters with a scale of approximately 0 are skipped.
• Packed-projection stacking requires a compatible loader and packed-module mapping; MiniMax-H3 supports its fused QKV and MLP projections.
• MiniMax-H3 accepts ordinary adapter stacks and at most one compatible Turbo adapter. Native distilled adapters such as FlashGen cannot be composed with other adapters.
• FastH3 Omni accepts ordinary adapters with independent scales, including under distributed layerwise offload; Turbo and native distilled adapters remain unsupported with this checkpoint.

Offline: pass a list of LoRARequest objects together with a matching list of scales:

outputs = omni.generate(
    prompt="A piece of cheesecake",
    lora_request=[lora_request_a, lora_request_b],
    lora_scale=[0.8, 0.6],
)

Online: the lora request field accepts an array of objects instead of a single object (a single object remains valid):

"lora": [
  {"name": "style", "local_path": "/path/to/style.safetensors", "scale": 0.8, "int_id": 1},
  {"name": "character", "local_path": "/path/to/character.safetensors", "scale": 0.6, "int_id": 2}
]

int_id is optional; when omitted it is derived from local_path.

Distill backend: fuse distilled LoRA at init¶

For distilled few-step LoRAs, pass lora_backend="distill" together with one or more concrete .safetensors files. The weights are fused into the base model once at init; subsequent generate() calls do not need a LoRARequest.

With tensor parallelism, the fusion loader selects each rank's portion of the adapter delta to match the base linear weights. Packed QKV and merged column projections are split per projection before being packed for each rank. Replicated weights and row-parallel biases keep the full delta. Unloading an adapter uses the same partitioning.

Supported pipelines¶

For Qwen-Image and Wan pipelines, the distill backend calls pipeline.load_lora_weights(...) during worker initialization.

Other diffusion pipelines are not currently listed as supporting distilled LoRA. Use the PEFT backend for request-time adapters, or bake converted weights into a local Diffusers directory before serving.

Single-file example (Qwen-Image-Lightning):

from vllm_omni import Omni

omni = Omni(
    model="Qwen/Qwen-Image-2512",
    lora_path="/path/to/Qwen-Image-2512-Lightning.safetensors",
    lora_backend="distill",
)

outputs = omni.generate("A piece of cheesecake")

Multi-file example (Wan2.2 MoE, high + low noise):

from vllm_omni import Omni

omni = Omni(
    model="Wan-AI/Wan2.2-T2V-A14B-Diffusers",
    lora_path=[
        "/path/to/wan2.2_high_noise_lora.safetensors",   # -> transformer
        "/path/to/wan2.2_low_noise_lora.safetensors",    # -> transformer_2
    ],
    lora_backend="distill",
)

The CLI examples under examples/offline_inference/ accept the same flags, e.g.:

python examples/offline_inference/text_to_video/text_to_video.py \
  --model Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --lora-backend distill \
  --lora-path /path/to/high.safetensors /path/to/low.safetensors \
  --prompt "A cat playing with yarn"

Online serving¶

Distilled LoRAs can also be fused when an online diffusion server starts. The adapter remains active for the lifetime of that server, so requests do not pass a per-request lora field.

vllm serve Qwen/Qwen-Image-2512 \
  --omni \
  --port 8091 \
  --lora-backend distill \
  --lora-path /path/to/Qwen-Image-2512-Lightning-4steps.safetensors

Then send a normal generation request with the sampling settings expected by the distilled checkpoint:

curl -s http://localhost:8091/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "A Chinese female college student, around 20 years old, with a very short haircut that conveys a gentle, artistic vibe. Her hair naturally falls to partially cover her cheeks, projecting a tomboyish yet charming demeanor. She has cool-toned fair skin and delicate features, with a slightly shy yet subtly confident expression—her mouth crooked in a playful, youthful smirk. She wears an off-shoulder top, revealing one shoulder, with a well-proportioned figure. The image is framed as a close-up selfie: she dominates the foreground, while the background clearly shows her dormitory—a neatly made bed with white linens on the top bunk, a tidy study desk with organized stationery, and wooden cabinets and drawers. The photo is captured on a smartphone under soft, even ambient lighting, with natural tones, high clarity, and a bright, lively atmosphere full of youthful, everyday energy."}
    ],
    "extra_body": {
      "height": 1024,
      "width": 1024,
      "num_inference_steps": 4,
      "true_cfg_scale": 1.0,
      "seed": 42
    }
  }' | jq -r '.choices[0].message.content[0].image_url.url' | cut -d',' -f2 | base64 -d > qwen_image_distill_lora_output.png

For Wan2.2 MoE serving, pass the high-noise checkpoint first and the low-noise checkpoint second to --lora-path.

Wan2.2 LightX2V Offline Assembly¶

This workflow is LoRA-adjacent: it uses external LightX2V conversion plus Wan2.2-Distill-Loras to bake converted Wan2.2 I2V checkpoints into a local Diffusers directory, instead of loading LoRA adapters at runtime.

Required assets¶

• Base model: Wan-AI/Wan2.2-I2V-A14B
• Diffusers skeleton: Wan-AI/Wan2.2-I2V-A14B-Diffusers
• Optional external converter from the LightX2V project (not shipped in this repository)
• Optional LoRA weights: lightx2v/Wan2.2-Distill-Loras

Step 1: Optional - convert high/low-noise DiT weights with LightX2V¶

Install or clone LightX2V from the upstream repository (https://github.com/ModelTC/LightX2V). After cloning, the converter used below is available at <lightx2v_root>/tools/convert/converter.py.

python /path/to/lightx2v/tools/convert/converter.py \
  --source /path/to/Wan2.2-I2V-A14B/high_noise_model \
  --output /tmp/wan22_lightx2v/high_noise_out \
  --output_ext .safetensors \
  --output_name diffusion_pytorch_model \
  --model_type wan_dit \
  --direction forward \
  --lora_path /path/to/wan2.2_i2v_A14b_high_noise_lora_rank64_lightx2v_4step_1022.safetensors \
  --lora_key_convert auto \
  --single_file

python /path/to/lightx2v/tools/convert/converter.py \
  --source /path/to/Wan2.2-I2V-A14B/low_noise_model \
  --output /tmp/wan22_lightx2v/low_noise_out \
  --output_ext .safetensors \
  --output_name diffusion_pytorch_model \
  --model_type wan_dit \
  --direction forward \
  --lora_path /path/to/wan2.2_i2v_A14b_low_noise_lora_rank64_lightx2v_4step_1022.safetensors \
  --lora_key_convert auto \
  --single_file

If you are not using LightX2V, skip this step and either keep the original Diffusers weights from the skeleton or point Step 2 at any other converted transformer/ and transformer_2/ checkpoints.

Step 2: Assemble a final Diffusers-style directory¶

python tools/wan22/assemble_wan22_i2v_diffusers.py \
  --diffusers-skeleton /path/to/Wan2.2-I2V-A14B-Diffusers \
  --transformer-weight /tmp/wan22_lightx2v/high_noise_out \
  --transformer-2-weight /tmp/wan22_lightx2v/low_noise_out \
  --output-dir /path/to/Wan2.2-I2V-A14B-Custom-Diffusers \
  --asset-mode symlink \
  --overwrite

--transformer-weight and --transformer-2-weight are optional. If you omit them, the tool keeps the original weights from the Diffusers skeleton.

Step 3: Run offline inference¶

python examples/offline_inference/image_to_video/image_to_video.py \
  --model /path/to/Wan2.2-I2V-A14B-Custom-Diffusers \
  --image /path/to/input.jpg \
  --prompt "A cat playing with yarn" \
  --num-frames 81 \
  --num-inference-steps 4 \
  --tensor-parallel-size 4 \
  --height 480 \
  --width 832 \
  --flow-shift 12 \
  --sample-solver euler \
  --guidance-scale 1.0 \
  --guidance-scale-high 1.0 \
  --boundary-ratio 0.875

Notes:

• This route avoids runtime LoRA loading changes in vLLM-Omni when you choose to bake converted weights into a local Diffusers directory.
• Output quality and speed depend on the replacement checkpoints and sampling params you choose.
• If you only need to fuse distilled LoRAs into a Wan2.2 checkpoint at load time (without the full LightX2V convert + assemble pipeline), you can instead pass them directly via --lora-backend distill --lora-path <high>.safetensors <low>.safetensors. See the Distill backend section above.

MiniMax-H3 adapter-declared schedules¶

MiniMax-H3 counts num_inference_steps as denoising updates: requesting 8 produces eight updates using nine sigma boundaries, including the terminal zero. Earlier uniform-schedule requests counted boundaries instead; to reproduce an old N-point schedule, request N-1 steps. This also changes Turbo requests from 5/9 to 4/8, preserving their sampling grids. Checkpoint-pinned and native LoRA schedules already count updates and are unchanged.

MiniMax-H3 supports the following few-step mechanisms:

• Checkpoint-pinned schedule: a merged release writes base_schedule into model_index.json. Requests must pass num_inference_steps as the interval count (for example 4 for [1.0, 0.7, 0.4, 0.15, 0.0]).
• Runtime Turbo LoRA: each LightX2V Turbo artifact carries its own contract, read from its filename. A four-step artifact requests num_inference_steps=4 and an eight-step one requests 8; the scheduler adds the terminal sigma boundary. The 768p retrains enforce flow_shift=6 and the 544p artifacts flow_shift=12. audio_flow_shift=3 across the family. A request that does not match the loaded artifact is rejected by name. Alpha comes from the artifact's metadata, or 8 when it declares none. ref2v artifacts require --task-type ref2va; fl2v artifacts serve T2VA and FL2VA. The ComfyUI exports use a fused-QKV layout and are not supported.
• Runtime native LoRA: FlashGen-style artifacts declare key_format=minimax-h3-native and embed base_schedule in safetensors metadata. When active, the adapter schedule overrides the base checkpoint and requests must use the interval-count contract (num_inference_steps=4). Request-mode generation may omit the field and take the count from the adapter schedule; step execution requires it explicitly, because the step scheduler derives the total step count from the request before the adapter schedule is known.

Native artifacts also declare qkv_layout=grouped. The H3 loader reorders fused qkv_proj LoRA rows with the same _reorder_grouped_qkv_to_qkv helper used for base-weight loading, then binds the packed Q/K/V slices through the legacy PEFT manager without modifying DiffusionLoRAManager.

Both runtime adapters run with distributed layerwise offload in request-mode generation, where the manager keeps the LoRA A/B buffers resident on the compute device while DLO streams the base blocks. MiniMax-H3 step execution still rejects DLO, so --step-execution cannot be combined with --enable-distributed-layerwise-offload. Model-level CPU offload and standard layerwise offload are rejected in every mode, because the dynamic LoRA tensors are neither parameters nor registered buffers and therefore do not participate in those weight lifecycles.

See Also¶

• Text-to-Image Offline Example - Complete offline LoRA example
• Text-to-Image Online Example - Complete online LoRA example