Skip to content

GPU

vLLM-Omni is a Python library that supports the following GPU variants. The library itself mainly contains python implementations for framework and models.

Requirements

  • OS: Linux
  • Python: 3.12

Note

vLLM-Omni is currently not natively supported on Windows.

  • GPU: compute capability 7.5 or higher (e.g., T4, RTX20xx, A100, L4, H100, etc.)
  • GPU: Validated on gfx942 (It should be supported on the AMD GPUs that are supported by vLLM.)
  • GPU: Validated on Intel® Arc™ B-Series.
  • GPU: Moore Threads GPU with MUSA SDK installed (validated on MTT S5000)

Set up using Python

Create a new Python environment

It's recommended to use uv, a very fast Python environment manager, to create and manage Python environments. Please follow the documentation to install uv. After installing uv, you can create a new Python environment using the following commands:

uv venv --python 3.12 --seed
source .venv/bin/activate

Pre-built wheels

Note: Pre-built wheels are currently available for vLLM-Omni 0.11.0rc1, 0.12.0rc1, 0.14.0rc1, 0.14.0, 0.16.0, 0.18.0, 0.20.0, 0.21.0, 0.22.0, 0.23.0, 0.24.0, 0.25.0, 0.26.0, 0.28.0, and 0.30.0. If you need a newer unreleased revision, please build from source.

Installation of vLLM

These pre-built wheel instructions install the published vLLM-Omni 0.30.0 release.

vLLM-Omni is built based on vLLM. Please install it with command below.

uv pip install vllm==0.30.0 --torch-backend=auto

Installation of vLLM-Omni

uv pip install vllm-omni==0.30.0

To run Gradio demos, also install the optional extras:

uv pip install 'vllm-omni[demo]==0.30.0'

Installation of vLLM

These pre-built wheel instructions install the published vLLM-Omni 0.30.0 release.

vLLM-Omni is built based on vLLM. Please install it with command below.

uv pip install vllm==0.30.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.30.0/rocm723

Installation of vLLM-Omni

# we need to add --no-build-isolation as the torch
# is not obtained from pypi, we have to install using the
# torch installed in our environment
uv pip install vllm-omni==0.30.0

# Optional if want to run Qwen3 TTS
uv pip uninstall onnxruntime # should be removed before we can install onnxruntime-rocm
uv pip install onnxruntime-rocm

Build wheel from source

Installation of vLLM

If you do not need to modify source code of vLLM, install the release-branch 0.32.0 wheel directly from the tested commit:

uv pip install --torch-backend=cu130 \
  "vllm @ https://wheels.vllm.ai/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm-0.32.0-cp38-abi3-manylinux_2_28_x86_64.whl"

This command selects the Linux x86_64 CUDA 13.0 wheel and matching PyTorch backend. It requires a CUDA 13 compatible driver. For another architecture, CUDA variant or an existing incompatible PyTorch installation, build vLLM from source instead. The vLLM 0.32.0 release and its commit-index and PyPI wheels are available. The PyPI x86_64 wheel embeds the same release commit and CUDA/PyTorch requirements. The command above retains the exact tested artifact. A vLLM-Omni 0.32 wheel is not yet published; install Omni from this source checkout.

Installation of vLLM-Omni

Since vllm-omni is rapidly evolving, it's recommended to install it from source

git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
uv pip install -e .

To run Gradio demos, install with optional extras:

uv pip install -e '.[demo]'

(Optional) Installation of vLLM from source If you want to check, modify or debug with source code of vLLM, install the library from source with the following instructions:

git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout 5df63a7203cdb9328d824a27b682e77c7b9928be
Set up environment variables to get pre-built wheels. If there are internet problems, just download the whl file manually. And set VLLM_PRECOMPILED_WHEEL_LOCATION as your local absolute path of whl file.
# CUDA 13.0 release-branch wheel for the exact source commit on x86_64
export VLLM_PRECOMPILED_WHEEL_LOCATION=https://wheels.vllm.ai/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm-0.32.0-cp38-abi3-manylinux_2_28_x86_64.whl
Install vllm with command below (If you have no existing PyTorch).
VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=cu130
Install vllm with command below (If you already have PyTorch).
python use_existing_torch.py
uv pip install -r requirements/build/cuda.txt
uv pip install --no-build-isolation --editable .

Installation of vLLM

The published 0.32.0+rocm723 wheel is available from https://wheels.vllm.ai/rocm/0.32.0/rocm723 for Linux x86_64 and Python 3.12. It requires the matching ROCm 7.2.3 PyTorch/Triton builds, rather than the CUDA packages. Its version file omits the source commit ID, so it has not been verified as the exact source pin below. The older 0.31.0 ROCm wheel is not a validated substitute. An experimental source build must use that exact release commit and upstream's matching ROCm/PyTorch requirements; both paths still need ROCm runtime validation with this checkout.

Installation of vLLM-Omni

Since vllm-omni is rapidly evolving, it's recommended to install it from source

git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
VLLM_OMNI_TARGET_DEVICE=rocm uv pip install -e .
# OR
uv pip install -e . --no-build-isolation

(Optional) Installation of vLLM from source If you want to check, modify or debug with source code of vLLM, install the library from source with the following instructions:

git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout 5df63a7203cdb9328d824a27b682e77c7b9928be
python3 -m pip install -r requirements/rocm.txt
python3 setup.py develop

Prerequisites

  • MUSA SDK: Download from MUSA SDK Download
  • torchada: CUDA→MUSA compatibility layer for PyTorch (pip install torchada)
  • mthreads-ml-py: MTML Python bindings (pip install mthreads-ml-py)
  • MATE: MUSA AI Tensor Engine (GitHub)

Installation of vLLM-MUSA

git clone https://github.com/MooreThreads/vllm-musa.git
cd vllm-musa
git checkout v0.18.0-dev
pip install . --no-build-isolation -v

Installation of vLLM-Omni

git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
VLLM_OMNI_TARGET_DEVICE=musa pip install -e . --no-build-isolation

For Gradio demos:

pip install -e '.[demo]' --no-build-isolation

Environment Variables

export MUSA_VISIBLE_DEVICES=0,1
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_MUSA_CUSTOM_OP_USE_NATIVE=false

Set up using Docker

Pre-built images

vLLM-Omni offers an official docker image for deployment. These images are built on top of vLLM docker images and available on Docker Hub as vllm/vllm-omni. The version of vLLM-Omni indicates which release of vLLM it is based on.

Nightly CUDA images built from the latest main are also available:

Use a commit-pinned tag when you need a reproducible nightly environment.

Here's an example deployment command that has been verified on 2 x H100's:

docker run --runtime nvidia --gpus 2 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=$HF_TOKEN" \
    -p 8091:8091 \
    --ipc=host \
    vllm/vllm-omni:v0.30.0 \
    vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091

Tip

The CUDA image does not define a default entrypoint, so include vllm serve ... --omni after the image name.

vLLM-Omni offers an official docker image for deployment. These images are built on top of vLLM docker images and available on Docker Hub as vllm/vllm-omni-rocm. The version of vLLM-Omni indicates which release of vLLM it is based on.

The prebuilt ROCm image is published separately from the release pipeline: v0.28.0 is the latest tag currently available on Docker Hub, and newer-tag availability is tracked in #7405.

Launch vLLM-Omni Server

Here's an example deployment command that has been verified on 2 x MI300's:

docker run --rm \
  --group-add=video \
  --ipc=host \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  --device /dev/kfd \
  --device /dev/dri \
  -v <path/to/model>:/app/model \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 8091:8091 \
  vllm/vllm-omni-rocm:v0.28.0 \
  --model Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091

Launch an interactive terminal with prebuilt docker image

If you want to run in dev environment you can launch the docker image as follows:

docker run --rm -it \
  --network=host \
  --group-add=video \
  --ipc=host \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  --device /dev/kfd \
  --device /dev/dri \
  -v <path/to/model>:/app/model \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  --entrypoint bash \
  vllm/vllm-omni-rocm:v0.28.0

Build your own docker image

Build docker image

The production Dockerfile defaults to the published vllm/vllm-openai:v0.32.0 CUDA 13 image, which identifies release commit 5df63a7203cdb9328d824a27b682e77c7b9928be:

DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.cuda -t vllm-omni-cuda .

The CI Dockerfile pins the final CUDA base by digest and installs the exact 0.32.0 commit-index wheel used by the tested local runtime. Full CI validation of this final image stack remains pending:

DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.ci -t vllm-omni-cuda-ci .

Launch the docker image

Launch with OpenAI API Server

Note

The model Qwen/Qwen3-Omni-30B-A3B-Instruct requires significant GPU memory. The example below has been verified on 2 x H100's.

docker run --runtime nvidia --gpus 2 \
  -v ${HF_HOME:-$HOME/.cache/huggingface}:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 8091:8091 \
  --ipc=host \
  vllm-omni-cuda \
  vllm serve --omni --model Qwen/Qwen3-Omni-30B-A3B-Instruct --port 8091

By default, this mounts $HOME/.cache/huggingface as the model cache directory. To use a custom location, set the HF_HOME environment variable before running the command (e.g., export HF_HOME=/data/models).

Launch with interactive session for development
docker run --runtime nvidia --gpus all -it --rm \
  -v ${HF_HOME:-$HOME/.cache/huggingface}:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 8091:8091 \
  --ipc=host \
  --entrypoint bash \
  vllm-omni-cuda

Build docker image

The source build defaults to the published upstream base image vllm/vllm-openai-rocm:v0.32.0, matching this checkout's target release. The optional vLLM source rebuild also defaults to v0.32.0. ROCm runtime validation with this checkout remains pending. Older or custom bases must provide the vLLM APIs checked by the Dockerfile's image-build canary. This upstream base is distinct from the prebuilt vllm/vllm-omni-rocm images discussed below; their published-tag availability is tracked in #7405.

DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm -t vllm-omni-rocm .

To select the upstream ROCm base explicitly, pass BASE_IMAGE:

DOCKER_BUILDKIT=1 docker build \
  -f docker/Dockerfile.rocm \
  --build-arg BASE_IMAGE=vllm/vllm-openai-rocm:v0.32.0 \
  -t vllm-omni-rocm .

Launch the docker image

Launch with OpenAI API Server
docker run --rm \
--group-add=video \
--ipc=host \
--cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
--device /dev/kfd \
--device /dev/dri \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=$HF_TOKEN" \
-p 8091:8091 \
--ipc=host \
vllm-omni-rocm \
--model Qwen/Qwen3-Omni-30B-A3B-Instruct --port 8091
Launch with interactive session for development
docker run --rm -it \
--network=host \
--group-add=video \
--ipc=host \
--cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
--device /dev/kfd \
--device /dev/dri \
-v <path/to/model>:/app/model \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--entrypoint bash \
vllm-omni-rocm

Build docker image

export VLLM_VERSION=v0.32.0
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.xpu -t vllm-omni-xpu --shm-size=4g \
  --build-arg VLLM_VERSION="$VLLM_VERSION" .

This layers vLLM-Omni on top of the published vllm/vllm-openai-xpu:<VLLM_VERSION> base image, which Docker pulls automatically. The Dockerfile defaults to v0.32.0. Its matching vLLM wheel requires PyTorch 2.14.0 and triton==3.8.0+xpu. The Dockerfile retains the upstream runtime instead of replacing Triton with the older triton-xpu package, and checks installed dependency requirements after adding Omni. XPU runtime validation with this checkout remains pending. To target a different vLLM release, pass --build-arg VLLM_VERSION=<tag>. If that tag has not been published yet, build the base from upstream's own Dockerfile first and point the build at it:

git clone --depth 1 --branch "$VLLM_VERSION" https://github.com/vllm-project/vllm /tmp/vllm
DOCKER_BUILDKIT=1 docker build -t vllm-openai-xpu:local --target vllm-openai \
  -f /tmp/vllm/docker/Dockerfile.xpu /tmp/vllm
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.xpu -t vllm-omni-xpu --shm-size=4g \
  --build-arg VLLM_BASE=vllm-openai-xpu:local .

Launch the docker image

Launch with OpenAI API Server
docker run -it -d --shm-size 10g \
  --name {container_name} \
  --net=host \
  --ipc=host \
  --privileged \
  -v /dev/dri/by-path:/dev/dri/by-path \
  --device /dev/dri:/dev/dri \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  vllm-omni-xpu \
  --model Qwen/Qwen2.5-Omni-3B --port 8091