Quickstart¶
This guide will help you quickly get started with vLLM-Omni to perform:
- Offline batched inference
- Online serving using OpenAI-compatible server
Prerequisites¶
- OS: Linux
- Python: 3.12
Installation¶
For installation on GPU from source:
uv venv --python 3.12 --seed
source .venv/bin/activate
# On CUDA (Linux x86_64, CUDA 13.0)
uv pip install --torch-backend=cu130 \
"vllm @ https://wheels.vllm.ai/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm-0.32.0-cp38-abi3-manylinux_2_28_x86_64.whl"
git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
uv pip install -e .
These commands install the vLLM 0.32.0 release-branch wheel at the exact tested tag commit, with PyTorch 2.13.0 and CUDA 13.0. The vLLM 0.32.0 release, PyPI wheel and matching ROCm wheel are published. ROCm runtime validation with this checkout remains pending. A vLLM-Omni 0.32 wheel is not yet published; install Omni from source as shown above. For published vLLM-Omni wheels and additional installation methods, see the installation guide.
Note
It is important to install the same major & minor version of vLLM and vLLM Omni, otherwise things may not work as expected. If the versions are misaligned, you will see a warning when you import vLLM Omni.
If the vllm command does not handle --omni correctly, verify that the active environment contains the matching vLLM release and vLLM-Omni installation. vLLM owns the CLI entrypoint and delegates --omni requests to vLLM-Omni.
Offline Inference¶
Text-to-image generation quickstart with vLLM-Omni:
from vllm_omni.entrypoints.omni import Omni
if __name__ == "__main__":
omni = Omni(model="Tongyi-MAI/Z-Image-Turbo")
prompt = "a cup of coffee on the table"
outputs = omni.generate(prompt)
images = outputs[0].images
images[0].save("coffee.png")
You can pass a list of prompts and wait for the independent requests to finish, as shown below.
Info
For diffusion pipelines, each prompt becomes a separate logical request. The runtime may automatically batch compatible in-flight requests through the scheduler and runner.
from vllm_omni.entrypoints.omni import Omni
if __name__ == "__main__":
omni = Omni(
model="Tongyi-MAI/Z-Image-Turbo",
# deploy_config="./deploy-config.yaml", # Optional deploy override
)
prompts = [
"a cup of coffee on a table",
"a toy dinosaur on a sandy beach",
"a fox waking up in bed and yawning",
]
omni_outputs = omni.generate(prompts)
for i_prompt, prompt_output in enumerate(omni_outputs):
this_images = prompt_output.images
for i_image, image in enumerate(this_images):
image.save(f"p{i_prompt}-img{i_image}.jpg")
print("saved to", f"p{i_prompt}-img{i_image}.jpg")
# saved to p0-img0.jpg
# saved to p1-img0.jpg
# saved to p2-img0.jpg
Info
For diffusion request batching, step execution, and streaming controls, see Diffusion Execution Modes.
For more usages, please refer to offline inference
Online Serving with the API Server¶
Text-to-image generation quickstart with vLLM-Omni:
curl -s http://localhost:8091/v1/images/generations \
-H "Content-Type: application/json" \
-d '{
"prompt": "a cup of coffee on the table",
"size": "1024x1024",
"response_format": "b64_json",
"seed": 42
}' | jq -r '.data[0].b64_json' | base64 -d > coffee.png
See the API Server guide to choose an endpoint. For model-specific details, refer to the text-to-image online serving example.