Qwen2.5-Omni¶
Source https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/qwen2_5_omni.
Setup¶
Please refer to the stage configuration documentation to configure memory allocation appropriately for your hardware setup.
Run examples¶
Multiple Prompts¶
Get into the example folder
Then run the command below. Note: for processing large volume data, it uses py_generator mode, which will return a python generator from Omni class.Single Prompt¶
Get into the example folder
Then run the command below.Modality control¶
If you want to control output modalities, e.g. only output text, you can run the command below:
Using Local Media Files¶
The end2end.py script supports local media files (audio, video, image) via CLI arguments:
# Use single local media files
python end2end.py --query-type use_image --image-path /path/to/image.jpg
python end2end.py --query-type use_video --video-path /path/to/video.mp4
python end2end.py --query-type use_audio --audio-path /path/to/audio.wav
# Combine multiple local media files
python end2end.py --query-type mixed_modalities \
--video-path /path/to/video.mp4 \
--image-path /path/to/image.jpg \
--audio-path /path/to/audio.wav
# Use audio from video file
python end2end.py --query-type use_audio_in_video --video-path /path/to/video.mp4
If media file paths are not provided, the script will use default assets. Supported query types: - use_image: Image input only - use_video: Video input only - use_audio: Audio input only - mixed_modalities: Audio + image + video - use_audio_in_video: Extract audio from video - text: Text-only query
Composable parallelism (strategy configs)¶
You can shard or replicate a stage with a small composable-parallel strategy.yaml overlaid onto the bundled default deploy config -- no bespoke deploy YAML required. Supply the strategy with --strategy-config and any matching device layout via --stage-overrides (explicit kwargs forwarded to Omni).
Both examples below require >= 3 GPUs (the thinker uses 2, talker + code2wav share a third):
# Tensor-parallel the thinker (TP=2)
python end2end.py --query-type text --num-prompts 6 \
--strategy-config strategy_tp2.yaml \
--stage-overrides '{"0": {"devices": "0,1"}, "1": {"devices": "2"}, "2": {"devices": "2"}}'
# Replicate the thinker across 2 engines (round-robin load balancing)
python end2end.py --query-type text --num-prompts 6 \
--strategy-config strategy_stage_replica.yaml \
--stage-overrides '{"0": {"devices": "0,1"}, "1": {"devices": "2"}, "2": {"devices": "2"}}'
Example materials¶
end2end.py
Large file omitted from the rendered docs. View it on GitHub: https://github.com/vllm-project/vllm-omni/blob/main/examples/offline_inference/qwen2_5_omni/end2end.py.
extract_prompts.py
#!/usr/bin/env python3
import argparse
def extract_prompt(line: str) -> str | None:
# Extract the content between the first '|' and the second '|'
i = line.find("|")
if i == -1:
return None
j = line.find("|", i + 1)
if j == -1:
return None
return line[i + 1 : j].strip()
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--input", "-i", required=True, help="Input .lst file path")
parser.add_argument("--output", "-o", required=True, help="Output file path")
parser.add_argument(
"--topk",
"-k",
type=int,
default=100,
help="Extract the top K prompts (default: 100)",
)
args = parser.parse_args()
prompts = []
with open(args.input, encoding="utf-8", errors="ignore") as f:
for line in f:
if len(prompts) >= args.topk:
break
p = extract_prompt(line.rstrip("\n"))
if p:
prompts.append(p)
with open(args.output, "w", encoding="utf-8") as f:
for p in prompts:
f.write(p + "\n")
if __name__ == "__main__":
main()
run_multiple_prompts.sh
run_single_prompt.sh
strategy_stage_replica.yaml
# Composable-parallel strategy for Qwen2.5-Omni.
#
# Replicate the thinker stage across 2 independent engines and let the omni
# StagePool balance requests round-robin across them. The thinker's `devices`
# entry acts as a per-replica template, so omni expands it to one GPU per replica.
# The `routing: round_robin` below auto-derives the StagePool load-balancer policy,
# so --omni-lb-policy is not needed.
#
# Rather than hand-write a bespoke deploy YAML, run on the bundled default deploy
# config and pass the matching 2-GPU device template via --stage-overrides (explicit
# kwargs): the thinker gets a two-GPU pool while talker + code2wav share a third
# card. Requires >= 3 GPUs.
#
# Usage:
# python3 end2end.py --query-type text --num-prompts 6 \
# --strategy-config strategy_stage_replica.yaml \
# --stage-overrides '{"0": {"devices": "0,1"}, "1": {"devices": "2"}, "2": {"devices": "2"}}'
strategies:
thinker:
- axis: stage_replica
size: 2
routing: round_robin
strategy_tp2.yaml
# Composable-parallel strategy for Qwen2.5-Omni: tensor-parallel the thinker.
#
# Shard the thinker stage across 2 GPUs with tensor parallelism (TP=2). TP routing
# is fixed (broadcast), so no `routing` / load-balancer policy is involved -- this
# is the simplest axis and a good starting point for the mental model.
#
# The strategy sets tensor_parallel_size=2 on the thinker. Rather than hand-write a
# bespoke deploy YAML, run on the bundled default deploy config and pass a matching
# 2-GPU device layout via --stage-overrides (explicit kwargs), so the thinker gets a
# two-GPU pool while talker + code2wav share a third card. Requires >= 3 GPUs.
#
# Usage:
# python3 end2end.py --query-type text --num-prompts 6 \
# --strategy-config strategy_tp2.yaml \
# --stage-overrides '{"0": {"devices": "0,1"}, "1": {"devices": "2"}, "2": {"devices": "2"}}'
strategies:
thinker:
- axis: tp
size: 2