Installation Guide¶

This document describes how to install vllm-ascend manually.

Requirements¶

Environment requirements¶

Operating system: Linux
Python: >= 3.10, < 3.13
Hardware equipped with Ascend NPUs. This guide supports the following devices:

Type	Common products
Atlas A2 series products	Atlas 800T A2, Atlas 900 A2 PoD, Atlas 200T A2 Box16, Atlas 300T A2, Atlas 800I A2, and others
Atlas A3 series products	Atlas 800T A3, Atlas 900 A3 SuperPoD, Atlas 9000 A3 SuperPoD, Atlas 800I A3, and others
Atlas inference series products	Atlas 300I DUO and Atlas 200I Pro
Atlas 950DT series products	Atlas 950DT

Hardware and software stack¶

vLLM Ascend runs on a layered hardware and software stack. The following overview shows how the main runtime components fit together.

Hardware and software stack overview

vLLM Ascend hardware and software stack

The following hardware and software stack is validated together for this release:

Validated hardware and software stack

Layer	Component	Validated version / requirement	Role
Runtime environment	Python	`3.12`	Python version used by the validated release image
Host enablement	Ascend HDK	`26.0.RC1`	Driver and firmware requirements for the selected CANN release
Ascend runtime	CANN Toolkit + Ops	`9.1.0`	Ascend user-space runtime, including the CANN Toolkit and hardware-specific Ops packages
Ascend runtime	NNAL	`9.1.0`	Provides `libatb.so` and ATB runtime capabilities
Framework	PyTorch	`2.10.0`	Tensor framework used by vLLM
Framework	TorchNPU	`2.10.0.post4`	Connects PyTorch to the Ascend runtime
Kernel acceleration	Triton Ascend	`3.2.2`	Used on A2, A3, and 950DT; not used on Atlas 300I DUO or Atlas 200I Pro
Inference engine	vLLM	`0.23.0`	Model inference engine
Hardware plugin	vLLM Ascend	`0.23.0`	Connects vLLM to the Ascend software stack

The versions and requirements in the table above are validated as one compatibility set. Do not arbitrarily mix versions from different releases.

For another release, select a complete row from Versioning Policy > Release compatibility matrix.

Installation¶

Set up the hardware environment¶

First, run the following command to confirm that the Ascend NPU firmware and driver are installed correctly:

npu-smi info

For more information, see the CANN installation resources.

Set up the software environment¶

Choose one complete path based on your requirements. Container-based paths require Docker; see the Docker installation guide if needed.

Requirement	Recommended method	Intended users
Get a working vLLM Ascend environment as quickly as possible	Use a prebuilt image	First-time users or users who want a quick deployment
Install vLLM Ascend on an existing CANN environment	Install in a CANN environment	Users of a CANN image or a host/container where CANN is already installed
Install and manage the complete software stack manually	Install from a base environment	Advanced users who need custom CANN or Python dependencies, development, or debugging

Components by installation method

Status: ✓ Already available · ○ Installed during this path

Component	Prebuilt image	CANN environment	Base environment
CANN Toolkit + Ops	✓	✓	○
NNAL	✓	✓	○
PyTorch + TorchNPU	✓	○	○
vLLM + vLLM Ascend	✓	○	○
Triton Ascend	✓	○	○

PyTorch and TorchNPU: Installed as dependencies during the Install vLLM and vLLM Ascend step; no separate installation step is required.
Triton Ascend: Installed only for A2, A3, and 950DT; it is not used on Atlas 300I DUO or Atlas 200I Pro.

Prebuilt imageCANN environmentBase environment

The host needs only a working Ascend driver and firmware, plus Docker. The image includes the CANN user-space environment, PyTorch/TorchNPU, vLLM, and vLLM Ascend.

Choose an official image for the fastest setup, or build an image manually when you need to customize it.

Select an official imageBuild an image manually

Select your hardware and operating system, then pull the official image, start the container, and verify the environment.

Official vLLM Ascend images

Image name	Hardware	OS
`vllm-ascend:v0.23.0`	Ascend A2 series products	Ubuntu
`vllm-ascend:v0.23.0-openeuler`	Ascend A2 series products	openEuler
`vllm-ascend:v0.23.0-a3`	Ascend A3 series products	Ubuntu
`vllm-ascend:v0.23.0-a3-openeuler`	Ascend A3 series products	openEuler
`vllm-ascend:v0.23.0-310p`	Atlas 300I DUO / Atlas 200I Pro	Ubuntu
`vllm-ascend:v0.23.0-310p-openeuler`	Atlas 300I DUO / Atlas 200I Pro	openEuler
`vllm-ascend:v0.23.0-a5`	Ascend 950DT series products	Ubuntu
`vllm-ascend:v0.23.0-a5-openeuler`	Ascend 950DT series products	openEuler

A2A3Atlas 300I DUOAtlas 200I Pro950DT

Pull the image¶

If image downloads are slow

vLLM Ascend images are downloaded from quay.io by default. If direct access is slow, use one of the following registry mirrors to accelerate the download.

For example, the original image address is:

quay.io/ascend/vllm-ascend:<TAG>

You can replace it with:

# Replace with tag you want to pull
TAG=v0.23.0
# use
docker pull m.daocloud.io/quay.io/ascend/vllm-ascend:$TAG
# or
docker pull quay.nju.edu.cn/ascend/vllm-ascend:$TAG

Replace only the registry prefix and preserve the complete original image tag, including suffixes such as -a3, -310p, -950dt, and -openeuler.

UbuntuopenEuler

export IMAGE=quay.io/ascend/vllm-ascend:v0.23.0
docker pull "$IMAGE"

export IMAGE=quay.io/ascend/vllm-ascend:v0.23.0-openeuler
docker pull "$IMAGE"

Start the container¶

UbuntuopenEuler

export DEVICE=/dev/davinci0
export MODEL_CACHE="${HOME}/.cache"

mkdir -p "$MODEL_CACHE"

docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --device "$DEVICE" \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v "$MODEL_CACHE:/root/.cache" \
    -p 8000:8000 \
    -it "$IMAGE" bash

export DEVICE=/dev/davinci0
export MODEL_CACHE="${HOME}/.cache"

mkdir -p "$MODEL_CACHE"

docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --device "$DEVICE" \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v "$MODEL_CACHE:/root/.cache" \
    -p 8000:8000 \
    -it "$IMAGE" bash

Verify the container environment¶

Run the following commands in the container. The container is ready when the output includes vLLM Ascend environment: OK.

npu-smi info

python3 - <<'PY'
import torch
import vllm
import vllm_ascend

assert torch.npu.is_available(), "No available Ascend NPU detected in the container"
print("vLLM Ascend environment: OK")
PY

Pull the image¶

If image downloads are slow

vLLM Ascend images are downloaded from quay.io by default. If direct access is slow, use one of the following registry mirrors to accelerate the download.

For example, the original image address is:

quay.io/ascend/vllm-ascend:<TAG>

You can replace it with:

# Replace with tag you want to pull
TAG=v0.23.0
# use
docker pull m.daocloud.io/quay.io/ascend/vllm-ascend:$TAG
# or
docker pull quay.nju.edu.cn/ascend/vllm-ascend:$TAG

Replace only the registry prefix and preserve the complete original image tag, including suffixes such as -a3, -310p, -950dt, and -openeuler.

UbuntuopenEuler

export IMAGE=quay.io/ascend/vllm-ascend:v0.23.0-a3
docker pull "$IMAGE"

export IMAGE=quay.io/ascend/vllm-ascend:v0.23.0-a3-openeuler
docker pull "$IMAGE"

Start the container¶

A3 container startup requirements

A3 uses a dual-DIE design and requires two Ascend device nodes, such as /dev/davinci0 and /dev/davinci1.

UbuntuopenEuler

export DEVICE0=/dev/davinci0
export DEVICE1=/dev/davinci1
export MODEL_CACHE="${HOME}/.cache"

mkdir -p "$MODEL_CACHE"

docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --device "$DEVICE0" \
    --device "$DEVICE1" \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v "$MODEL_CACHE:/root/.cache" \
    -p 8000:8000 \
    -it "$IMAGE" bash

export DEVICE0=/dev/davinci0
export DEVICE1=/dev/davinci1
export MODEL_CACHE="${HOME}/.cache"

mkdir -p "$MODEL_CACHE"

docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --device "$DEVICE0" \
    --device "$DEVICE1" \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v "$MODEL_CACHE:/root/.cache" \
    -p 8000:8000 \
    -it "$IMAGE" bash

Verify the container environment¶

Run the following commands in the container. The container is ready when the output includes vLLM Ascend environment: OK.

npu-smi info

python3 - <<'PY'
import torch
import vllm
import vllm_ascend

assert torch.npu.is_available(), "No available Ascend NPU detected in the container"
print("vLLM Ascend environment: OK")
PY

Pull the image¶

If image downloads are slow

vLLM Ascend images are downloaded from quay.io by default. If direct access is slow, use one of the following registry mirrors to accelerate the download.

For example, the original image address is:

quay.io/ascend/vllm-ascend:<TAG>

You can replace it with:

# Replace with tag you want to pull
TAG=v0.23.0
# use
docker pull m.daocloud.io/quay.io/ascend/vllm-ascend:$TAG
# or
docker pull quay.nju.edu.cn/ascend/vllm-ascend:$TAG

Replace only the registry prefix and preserve the complete original image tag, including suffixes such as -a3, -310p, -950dt, and -openeuler.

UbuntuopenEuler

export IMAGE=quay.io/ascend/vllm-ascend:v0.23.0-310p
docker pull "$IMAGE"

export IMAGE=quay.io/ascend/vllm-ascend:v0.23.0-310p-openeuler
docker pull "$IMAGE"

Start the container¶

UbuntuopenEuler

export DEVICE=/dev/davinci0
export MODEL_CACHE="${HOME}/.cache"

mkdir -p "$MODEL_CACHE"

docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --device "$DEVICE" \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v "$MODEL_CACHE:/root/.cache" \
    -p 8000:8000 \
    -it "$IMAGE" bash

export DEVICE=/dev/davinci0
export MODEL_CACHE="${HOME}/.cache"

mkdir -p "$MODEL_CACHE"

docker run --rm \
    --name vllm-ascend \
    --shm-size=1g \
    --device "$DEVICE" \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v "$MODEL_CACHE:/root/.cache" \
    -p 8000:8000 \
    -it "$IMAGE" bash

Verify the container environment¶

Run the following commands in the container. The container is ready when the output includes vLLM Ascend environment: OK.

npu-smi info

python3 - <<'PY'
import torch
import vllm
import vllm_ascend

assert torch.npu.is_available(), "No available Ascend NPU detected in the container"
print("vLLM Ascend environment: OK")
PY

Pull the image¶

If image downloads are slow

vLLM Ascend images are downloaded from quay.io by default. If direct access is slow, use one of the following registry mirrors to accelerate the download.

For example, the original image address is:

quay.io/ascend/vllm-ascend:<TAG>

You can replace it with:

# Replace with tag you want to pull
TAG=v0.23.0
# use
docker pull m.daocloud.io/quay.io/ascend/vllm-ascend:$TAG
# or
docker pull quay.nju.edu.cn/ascend/vllm-ascend:$TAG

Replace only the registry prefix and preserve the complete original image tag, including suffixes such as -a3, -310p, -950dt, and -openeuler.

UbuntuopenEuler

export IMAGE=quay.io/ascend/vllm-ascend:v0.23.0-310p
docker pull "$IMAGE"

export IMAGE=quay.io/ascend/vllm-ascend:v0.23.0-310p-openeuler
docker pull "$IMAGE"

Start the container¶

Atlas 200I Pro container startup requirements

Atlas 200I Pro requires additional device nodes, driver libraries, and host configuration files. Before starting the container, make sure that all host paths mounted by the command below exist.

UbuntuopenEuler

export MODEL_CACHE="${HOME}/.cache"

mkdir -p "$MODEL_CACHE"

docker run --rm \
    --privileged \
    --name vllm-ascend \
    --shm-size=10g \
    --device=/dev/davinci0:/dev/davinci0 \
    --device=/dev/davinci_manager \
    --device=/dev/ascend_manager \
    --device=/dev/user_config \
    -v /etc/sys_version.conf:/etc/sys_version.conf \
    -v /etc/ld.so.conf.d/mind_so.conf:/etc/ld.so.conf.d/mind_so.conf \
    -v /etc/hdcBasic.cfg:/etc/hdcBasic.cfg \
    -v /var/dmp_daemon:/var/dmp_daemon \
    -v /usr/lib64/libmmpa.so:/usr/lib64/libmmpa.so \
    -v /usr/lib64/libcrypto.so.1.1:/usr/lib64/libcrypto.so.1.1 \
    -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
    -v /usr/lib64/libstackcore.so:/usr/lib64/libstackcore.so \
    -v /usr/lib/aarch64-linux-gnu/libyaml-0.so.2:/usr/lib64/libyaml-0.so.2 \
    -v /etc/slog.conf:/etc/slog.conf \
    -v /var/slogd:/var/slogd \
    -v /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64 \
    -v /usr/lib64/libtensorflow.so:/usr/lib64/libtensorflow.so \
    -v "$MODEL_CACHE:/root/.cache" \
    -p 8000:8000 \
    -it "$IMAGE" bash

export MODEL_CACHE="${HOME}/.cache"

mkdir -p "$MODEL_CACHE"

docker run --rm \
    --privileged \
    --name vllm-ascend \
    --shm-size=10g \
    --device=/dev/davinci0:/dev/davinci0 \
    --device=/dev/davinci_manager \
    --device=/dev/ascend_manager \
    --device=/dev/user_config \
    -v /etc/sys_version.conf:/etc/sys_version.conf \
    -v /etc/ld.so.conf.d/mind_so.conf:/etc/ld.so.conf.d/mind_so.conf \
    -v /etc/hdcBasic.cfg:/etc/hdcBasic.cfg \
    -v /var/dmp_daemon:/var/dmp_daemon \
    -v /usr/lib64/libsemanage.so.2:/usr/lib64/libsemanage.so.2 \
    -v /usr/lib64/libmmpa.so:/usr/lib64/libmmpa.so \
    -v /usr/lib64/libcrypto.so.1.1:/usr/lib64/libcrypto.so.1.1 \
    -v /usr/lib64/libyaml-0.so.2.0.9:/usr/lib64/libyaml-0.so.2 \
    -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
    -v /usr/lib64/libstackcore.so:/usr/lib64/libstackcore.so \
    -v /etc/slog.conf:/etc/slog.conf \
    -v /var/slogd:/var/slogd \
    -v /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64 \
    -v /usr/lib64/libtensorflow.so:/usr/lib64/libtensorflow.so \
    -v "$MODEL_CACHE:/root/.cache" \
    -p 8000:8000 \
    -it "$IMAGE" bash

Verify the container environment¶

Run the following commands in the container. The container is ready when the output includes vLLM Ascend environment: OK.

npu-smi info

python3 - <<'PY'
import torch
import vllm
import vllm_ascend

assert torch.npu.is_available(), "No available Ascend NPU detected in the container"
print("vLLM Ascend environment: OK")
PY

Pull the image¶

If image downloads are slow

vLLM Ascend images are downloaded from quay.io by default. If direct access is slow, use one of the following registry mirrors to accelerate the download.

For example, the original image address is:

quay.io/ascend/vllm-ascend:<TAG>

You can replace it with:

# Replace with tag you want to pull
TAG=v0.23.0
# use
docker pull m.daocloud.io/quay.io/ascend/vllm-ascend:$TAG
# or
docker pull quay.nju.edu.cn/ascend/vllm-ascend:$TAG

Replace only the registry prefix and preserve the complete original image tag, including suffixes such as -a3, -310p, -950dt, and -openeuler.

UbuntuopenEuler

export IMAGE=quay.io/ascend/vllm-ascend:v0.23.0-a5
docker pull "$IMAGE"

export IMAGE=quay.io/ascend/vllm-ascend:v0.23.0-a5-openeuler
docker pull "$IMAGE"

Start the container¶

UbuntuopenEuler

export MODEL_CACHE="${HOME}/.cache"

mkdir -p "$MODEL_CACHE"

docker run --rm \
    --name vllm-ascend \
    --net=host \
    --shm-size=1g \
    --device /dev/davinci0 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v "$MODEL_CACHE:/root/.cache" \
    -it "$IMAGE" bash

export MODEL_CACHE="${HOME}/.cache"

mkdir -p "$MODEL_CACHE"

docker run --rm \
    --name vllm-ascend \
    --net=host \
    --shm-size=1g \
    --device /dev/davinci0 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v "$MODEL_CACHE:/root/.cache" \
    -it "$IMAGE" bash

Verify the container environment¶

Run the following commands in the container. The container is ready when the output includes vLLM Ascend environment: OK.

npu-smi info

python3 - <<'PY'
import torch
import vllm
import vllm_ascend

assert torch.npu.is_available(), "No available Ascend NPU detected in the container"
print("vLLM Ascend environment: OK")
PY

Run the command for your hardware and operating system:

A2A3Atlas 300I DUO / Atlas 200I Pro950DT

UbuntuopenEuler

git clone --depth 1 --branch v0.23.0 \
    https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
docker build -t vllm-ascend-dev:latest -f Dockerfile .

git clone --depth 1 --branch v0.23.0 \
    https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
docker build -t vllm-ascend-dev:latest -f Dockerfile.openEuler .

UbuntuopenEuler

git clone --depth 1 --branch v0.23.0 \
    https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
docker build -t vllm-ascend-dev:latest -f Dockerfile.a3 .

git clone --depth 1 --branch v0.23.0 \
    https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
docker build -t vllm-ascend-dev:latest -f Dockerfile.a3.openEuler .

UbuntuopenEuler

git clone --depth 1 --branch v0.23.0 \
    https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
docker build -t vllm-ascend-dev:latest -f Dockerfile.310p .

git clone --depth 1 --branch v0.23.0 \
    https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
docker build -t vllm-ascend-dev:latest -f Dockerfile.310p.openEuler .

UbuntuopenEuler

git clone --depth 1 --branch v0.23.0 \
    https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
docker build -t vllm-ascend-dev:latest -f Dockerfile.a5 .

git clone --depth 1 --branch v0.23.0 \
    https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
docker build -t vllm-ascend-dev:latest -f Dockerfile.a5.openEuler .

This path applies to an official CANN base image or to CANN that is already installed on the host or in an existing container.

Prepare the CANN environment¶

Use a CANN base imageUse an existing CANN installation

Recommended CANN base images

Hardware	Ubuntu	openEuler
Ascend A2 series products	`quay.io/ascend/cann:9.1.0-910b-ubuntu22.04-py3.12`	`quay.io/ascend/cann:9.1.0-910b-openeuler24.03-py3.12`
Ascend A3 series products	`quay.io/ascend/cann:9.1.0-a3-ubuntu22.04-py3.12`	`quay.io/ascend/cann:9.1.0-a3-openeuler24.03-py3.12`
Atlas 300I DUO	`quay.io/ascend/cann:9.1.0-310p-ubuntu22.04-py3.12`	`quay.io/ascend/cann:9.1.0-310p-openeuler24.03-py3.12`
Atlas 200I Pro	`quay.io/ascend/cann:9.1.0-310p-ubuntu22.04-py3.12`	`quay.io/ascend/cann:9.1.0-310p-openeuler24.03-py3.12`
Ascend 950DT series products	`quay.io/ascend/cann:9.1.0-950-ubuntu22.04-py3.12`	`quay.io/ascend/cann:9.1.0-950-openeuler24.03-py3.12`

The CANN base image already includes the Toolkit, the operator package for the target hardware, and NNAL. You do not need to reinstall CANN in the container. For other operating systems and tags, see the CANN Container Images Overview.

Select your hardware and operating system, then start the CANN container:

A2A3Atlas 300I DUOAtlas 200I Pro950DT

Pull the image¶

UbuntuopenEuler

export IMAGE=quay.io/ascend/cann:9.1.0-910b-ubuntu22.04-py3.12
docker pull "$IMAGE"

export IMAGE=quay.io/ascend/cann:9.1.0-910b-openeuler24.03-py3.12
docker pull "$IMAGE"

Start the container¶

UbuntuopenEuler

export DEVICE=/dev/davinci0
export MODEL_CACHE="${HOME}/.cache"

mkdir -p "$MODEL_CACHE"

docker run --rm \
    --name vllm-ascend-cann \
    --shm-size=1g \
    --device "$DEVICE" \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v "$MODEL_CACHE:/root/.cache" \
    -p 8000:8000 \
    -it "$IMAGE" bash

export DEVICE=/dev/davinci0
export MODEL_CACHE="${HOME}/.cache"

mkdir -p "$MODEL_CACHE"

docker run --rm \
    --name vllm-ascend-cann \
    --shm-size=1g \
    --device "$DEVICE" \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v "$MODEL_CACHE:/root/.cache" \
    -p 8000:8000 \
    -it "$IMAGE" bash

Pull the image¶

UbuntuopenEuler

export IMAGE=quay.io/ascend/cann:9.1.0-a3-ubuntu22.04-py3.12
docker pull "$IMAGE"

export IMAGE=quay.io/ascend/cann:9.1.0-a3-openeuler24.03-py3.12
docker pull "$IMAGE"

Start the container¶

A3 container startup requirements

A3 uses a dual-DIE design and requires two Ascend device nodes, such as /dev/davinci0 and /dev/davinci1.

UbuntuopenEuler

export DEVICE0=/dev/davinci0
export DEVICE1=/dev/davinci1
export MODEL_CACHE="${HOME}/.cache"

mkdir -p "$MODEL_CACHE"

docker run --rm \
    --name vllm-ascend-cann \
    --shm-size=1g \
    --device "$DEVICE0" \
    --device "$DEVICE1" \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v "$MODEL_CACHE:/root/.cache" \
    -p 8000:8000 \
    -it "$IMAGE" bash

export DEVICE0=/dev/davinci0
export DEVICE1=/dev/davinci1
export MODEL_CACHE="${HOME}/.cache"

mkdir -p "$MODEL_CACHE"

docker run --rm \
    --name vllm-ascend-cann \
    --shm-size=1g \
    --device "$DEVICE0" \
    --device "$DEVICE1" \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v "$MODEL_CACHE:/root/.cache" \
    -p 8000:8000 \
    -it "$IMAGE" bash

Pull the image¶

UbuntuopenEuler

export IMAGE=quay.io/ascend/cann:9.1.0-310p-ubuntu22.04-py3.12
docker pull "$IMAGE"

export IMAGE=quay.io/ascend/cann:9.1.0-310p-openeuler24.03-py3.12
docker pull "$IMAGE"

Start the container¶

UbuntuopenEuler

export DEVICE=/dev/davinci0
export MODEL_CACHE="${HOME}/.cache"

mkdir -p "$MODEL_CACHE"

docker run --rm \
    --name vllm-ascend-cann \
    --shm-size=1g \
    --device "$DEVICE" \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v "$MODEL_CACHE:/root/.cache" \
    -p 8000:8000 \
    -it "$IMAGE" bash

export DEVICE=/dev/davinci0
export MODEL_CACHE="${HOME}/.cache"

mkdir -p "$MODEL_CACHE"

docker run --rm \
    --name vllm-ascend-cann \
    --shm-size=1g \
    --device "$DEVICE" \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v "$MODEL_CACHE:/root/.cache" \
    -p 8000:8000 \
    -it "$IMAGE" bash

Pull the image¶

UbuntuopenEuler

export IMAGE=quay.io/ascend/cann:9.1.0-310p-ubuntu22.04-py3.12
docker pull "$IMAGE"

export IMAGE=quay.io/ascend/cann:9.1.0-310p-openeuler24.03-py3.12
docker pull "$IMAGE"

Start the container¶

Atlas 200I Pro container startup requirements

Atlas 200I Pro requires additional device nodes, driver libraries, and host configuration files. Before starting the container, make sure that all host paths mounted by the command below exist.

UbuntuopenEuler

export MODEL_CACHE="${HOME}/.cache"

mkdir -p "$MODEL_CACHE"

docker run --rm \
    --privileged \
    --name vllm-ascend-cann \
    --shm-size=10g \
    --device=/dev/davinci0:/dev/davinci0 \
    --device=/dev/davinci_manager \
    --device=/dev/ascend_manager \
    --device=/dev/user_config \
    -v /etc/sys_version.conf:/etc/sys_version.conf \
    -v /etc/ld.so.conf.d/mind_so.conf:/etc/ld.so.conf.d/mind_so.conf \
    -v /etc/hdcBasic.cfg:/etc/hdcBasic.cfg \
    -v /var/dmp_daemon:/var/dmp_daemon \
    -v /usr/lib64/libmmpa.so:/usr/lib64/libmmpa.so \
    -v /usr/lib64/libcrypto.so.1.1:/usr/lib64/libcrypto.so.1.1 \
    -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
    -v /usr/lib64/libstackcore.so:/usr/lib64/libstackcore.so \
    -v /usr/lib/aarch64-linux-gnu/libyaml-0.so.2:/usr/lib64/libyaml-0.so.2 \
    -v /etc/slog.conf:/etc/slog.conf \
    -v /var/slogd:/var/slogd \
    -v /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64 \
    -v /usr/lib64/libtensorflow.so:/usr/lib64/libtensorflow.so \
    -v "$MODEL_CACHE:/root/.cache" \
    -p 8000:8000 \
    -it "$IMAGE" bash

export MODEL_CACHE="${HOME}/.cache"

mkdir -p "$MODEL_CACHE"

docker run --rm \
    --privileged \
    --name vllm-ascend-cann \
    --shm-size=10g \
    --device=/dev/davinci0:/dev/davinci0 \
    --device=/dev/davinci_manager \
    --device=/dev/ascend_manager \
    --device=/dev/user_config \
    -v /etc/sys_version.conf:/etc/sys_version.conf \
    -v /etc/ld.so.conf.d/mind_so.conf:/etc/ld.so.conf.d/mind_so.conf \
    -v /etc/hdcBasic.cfg:/etc/hdcBasic.cfg \
    -v /var/dmp_daemon:/var/dmp_daemon \
    -v /usr/lib64/libsemanage.so.2:/usr/lib64/libsemanage.so.2 \
    -v /usr/lib64/libmmpa.so:/usr/lib64/libmmpa.so \
    -v /usr/lib64/libcrypto.so.1.1:/usr/lib64/libcrypto.so.1.1 \
    -v /usr/lib64/libyaml-0.so.2.0.9:/usr/lib64/libyaml-0.so.2 \
    -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
    -v /usr/lib64/libstackcore.so:/usr/lib64/libstackcore.so \
    -v /etc/slog.conf:/etc/slog.conf \
    -v /var/slogd:/var/slogd \
    -v /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64 \
    -v /usr/lib64/libtensorflow.so:/usr/lib64/libtensorflow.so \
    -v "$MODEL_CACHE:/root/.cache" \
    -p 8000:8000 \
    -it "$IMAGE" bash

Pull the image¶

UbuntuopenEuler

export IMAGE=quay.io/ascend/cann:9.1.0-950-ubuntu22.04-py3.12
docker pull "$IMAGE"

export IMAGE=quay.io/ascend/cann:9.1.0-950-openeuler24.03-py3.12
docker pull "$IMAGE"

Start the container¶

UbuntuopenEuler

export MODEL_CACHE="${HOME}/.cache"

mkdir -p "$MODEL_CACHE"

docker run --rm \
    --name vllm-ascend-cann \
    --net=host \
    --shm-size=1g \
    --device /dev/davinci0 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v "$MODEL_CACHE:/root/.cache" \
    -it "$IMAGE" bash

export MODEL_CACHE="${HOME}/.cache"

mkdir -p "$MODEL_CACHE"

docker run --rm \
    --name vllm-ascend-cann \
    --net=host \
    --shm-size=1g \
    --device /dev/davinci0 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v "$MODEL_CACHE:/root/.cache" \
    -it "$IMAGE" bash

Verify the NNAL environment

Confirm that /usr/local/Ascend/nnal/atb/set_env.sh and libatb.so are available. If CANN is installed elsewhere, source the corresponding set_env.sh. If a "libatb.so not found" error occurs at runtime, make sure that the manual installation steps installed NNAL correctly.

source /usr/local/Ascend/ascend-toolkit/set_env.sh

if [ -f /usr/local/Ascend/nnal/atb/set_env.sh ]; then
    source /usr/local/Ascend/nnal/atb/set_env.sh
fi

export ASCEND_TOOLKIT_HOME="${ASCEND_TOOLKIT_HOME:-/usr/local/Ascend/ascend-toolkit/latest}"
npu-smi info

Install vLLM and vLLM Ascend¶

First, install the system dependencies and configure the pip mirror.

# Using apt-get with mirror
sed -i 's|ports.ubuntu.com|mirrors.tuna.tsinghua.edu.cn|g' /etc/apt/sources.list
apt-get update -y && apt-get install -y gcc g++ cmake ninja-build libnuma-dev wget git curl jq
# Or using yum
# yum update -y && yum install -y gcc g++ cmake ninja-build numactl-devel wget git curl jq
# Config pip mirror, only versions 0.11.0 and earlier are supported, if using a version later than 0.11.0, do not execute this command
pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple

Optional: If you are working on an x86 machine or using a TorchNPU development version, configure pip's extra-index:

# For TorchNPU dev version or x86 machine
pip config set global.extra-index-url "https://download.pytorch.org/whl/cpu/"

Choose one of the following methods to install vllm and vllm-ascend. PyTorch and TorchNPU are installed automatically as dependencies during this step using the compatible versions for the selected release.

Installation method comparison

Method	Package type	Hardware handling	Recommended for
pip	Prebuilt wheel	Installs the standard `vllm-ascend` wheel, which currently targets A2	A2 users who want the simplest installation
uv	Prebuilt wheel	Uses WheelNext wheel variants to select a compatible build for the detected hardware	Users who want a prebuilt wheel selected for their hardware
Source	Source code	Builds and installs vLLM Ascend for the current environment	Development, customization, or environments without a suitable prebuilt wheel

pip installationuv installationsource installation

This installation method currently supports only A2 builds and does not automatically support A3, Atlas 300I DUO, Atlas 200I Pro, or 950DT. For other hardware, use a prebuilt image, WheelNext, or a source installation.

pip install "vllm==0.23.0"
pip install \
    --extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi \
    "vllm-ascend==0.23.0"

If pip cannot verify the Huawei Cloud mirror certificate

The Huawei Cloud package index uses HTTPS, so --trusted-host is not normally required. If pip reports an SSL certificate verification or untrusted-host error for mirrors.huaweicloud.com, first update pip and the operating system CA certificates, or configure the CA bundle required by your network.

As a temporary workaround on a trusted network, retry the vLLM Ascend installation command with --trusted-host mirrors.huaweicloud.com.

This option tells pip to trust the host even when HTTPS validation fails, which weakens protection against man-in-the-middle attacks. Use it only when you trust the network and cannot fix the certificate configuration.

Check the device build type:

python - <<'PY'
from vllm_ascend._build_info import __device_type__

print("vLLM Ascend wheel device type:", __device_type__)
assert __device_type__ == "A2", __device_type__
PY

WheelNext selects a vLLM Ascend wheel that matches the hardware from the variant index. First, install and verify uv:

# install uv-wheelnext
curl -LsSf https://astral.sh/uv/install.sh | sed 's/verify_checksum "$_file"/true/' | INSTALLER_DOWNLOAD_URL=https://wheelnext.astral.sh sh
source $HOME/.local/bin/env

# Install vllm-project/vllm. The newest supported version is v0.23.0.
pip install vllm==0.23.0

# Install vllm-project/vllm-ascend from wheelnext index.
uv pip install --system \
    --extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi/variant \
    --index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple \
    --find-links https://mirrors.huaweicloud.com/ascend/repos/pypi/triton-ascend/ \
    vllm-ascend==0.23.0

Clear the cache if uv installation fails

If uv pip install fails because of a corrupted cache or stale package data, clear the uv cache before running the installation command again:

uv cache clean

Check the device build type:

python - <<'PY'
from vllm_ascend._build_info import __device_type__
print("vLLM Ascend wheel device type:", __device_type__)
PY

A3 source installation

When installing vLLM Ascend from source for A3, run git submodule update --init --recursive manually or make sure that the environment has internet access so the required submodules can be downloaded automatically.

Install vLLM first, then install vLLM Ascend:

# Install vLLM.
git clone --depth 1 --branch v0.23.0 https://github.com/vllm-project/vllm
cd vllm
VLLM_TARGET_DEVICE=empty pip install -e .
cd ..

# Install vLLM Ascend.
git clone --depth 1 --branch v0.23.0 https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
# git submodule update --init --recursive
export ASCEND_INDEX_URL=https://mirrors.huaweicloud.com/ascend/repos/pypi
pip install -e . --extra-index-url "${ASCEND_INDEX_URL}"
cd ..

Finally, handle triton and triton-ascend according to the hardware:

A2 / A3 / 950DTAtlas 300I DUO / Atlas 200I Pro

To ensure that Triton Ascend matches the current vLLM Ascend version, remove any existing triton and triton-ascend packages before installing the corresponding version of triton-ascend:

pip uninstall -y triton triton-ascend

pip install triton-ascend==3.2.2 \
    --extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi

Can community Triton and Triton Ascend coexist?

For Triton Ascend 3.2.0 and earlier, uninstall community Triton before installing Triton Ascend.

For Triton Ascend 3.2.1 and later, installing Triton Ascend first installs the required community Triton package and then provides its Ascend-specific implementation over the shared installation paths. If Triton is reinstalled or upgraded afterward, it may overwrite the Triton Ascend files. In that case, uninstall both packages and reinstall Triton Ascend.

For more Triton Ascend installation guidance, see Triton-Ascend > FAQ.

When installing another version of vllm-ascend, check the repository's requirements.txt for the corresponding triton-ascend version.

Atlas 300I DUO and Atlas 200I Pro do not support triton or triton-ascend. If either package is already installed in the environment, uninstall it and do not reinstall triton-ascend:

pip uninstall -y triton triton-ascend

This path is intended for advanced users who need to manage the user-space software stack themselves. You can install on an existing Linux host or start from a minimal Linux container.

Install CANN manually¶

Please refer to CANN Installation Resources or the following code to complete the installation.

The commands below use the default CANN and NNAL installation paths. If you install either component in a non-default directory, source the corresponding set_env.sh from the actual installation directory.

Code

# Create a virtual environment.
python -m venv vllm-ascend-env
source vllm-ascend-env/bin/activate

# Install required Python packages.
python -m pip install --upgrade pip
pip3 install attrs numpy decorator sympy cffi pyyaml pathlib2 psutil protobuf scipy requests absl-py wheel typing_extensions

# Download and install the CANN package.
wget --header="Referer: https://www.hiascend.com/" https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%209.1.0/Ascend-cann-toolkit_9.1.0_linux-"$(uname -i)".run
chmod +x ./Ascend-cann-toolkit_9.1.0_linux-"$(uname -i)".run
./Ascend-cann-toolkit_9.1.0_linux-"$(uname -i)".run --full
source /usr/local/Ascend/ascend-toolkit/set_env.sh
export ASCEND_TOOLKIT_HOME="${ASCEND_TOOLKIT_HOME:-/usr/local/Ascend/ascend-toolkit/latest}"

wget --header="Referer: https://www.hiascend.com/" https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%209.1.0/Ascend-cann-910b-ops_9.1.0_linux-"$(uname -i)".run
chmod +x ./Ascend-cann-910b-ops_9.1.0_linux-"$(uname -i)".run
./Ascend-cann-910b-ops_9.1.0_linux-"$(uname -i)".run --install

wget --header="Referer: https://www.hiascend.com/" https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%209.1.0/Ascend-cann-nnal_9.1.0_linux-"$(uname -i)".run
chmod +x ./Ascend-cann-nnal_9.1.0_linux-"$(uname -i)".run
./Ascend-cann-nnal_9.1.0_linux-"$(uname -i)".run --install

source /usr/local/Ascend/nnal/atb/set_env.sh

Install vLLM and vLLM Ascend¶

First, install the system dependencies and configure the pip mirror.

# Using apt-get with mirror
sed -i 's|ports.ubuntu.com|mirrors.tuna.tsinghua.edu.cn|g' /etc/apt/sources.list
apt-get update -y && apt-get install -y gcc g++ cmake ninja-build libnuma-dev wget git curl jq
# Or using yum
# yum update -y && yum install -y gcc g++ cmake ninja-build numactl-devel wget git curl jq
# Config pip mirror, only versions 0.11.0 and earlier are supported, if using a version later than 0.11.0, do not execute this command
pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple

Optional: If you are working on an x86 machine or using a TorchNPU development version, configure pip's extra-index:

# For TorchNPU dev version or x86 machine
pip config set global.extra-index-url "https://download.pytorch.org/whl/cpu/"

Choose one of the following methods to install vllm and vllm-ascend. PyTorch and TorchNPU are installed automatically as dependencies during this step using the compatible versions for the selected release.

Installation method comparison

Method	Package type	Hardware handling	Recommended for
pip	Prebuilt wheel	Installs the standard `vllm-ascend` wheel, which currently targets A2	A2 users who want the simplest installation
uv	Prebuilt wheel	Uses WheelNext wheel variants to select a compatible build for the detected hardware	Users who want a prebuilt wheel selected for their hardware
Source	Source code	Builds and installs vLLM Ascend for the current environment	Development, customization, or environments without a suitable prebuilt wheel

pip installationuv installationsource installation

This installation method currently supports only A2 builds and does not automatically support A3, Atlas 300I DUO, Atlas 200I Pro, or 950DT. For other hardware, use a prebuilt image, WheelNext, or a source installation.

pip install "vllm==0.23.0"
pip install \
    --extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi \
    "vllm-ascend==0.23.0"

If pip cannot verify the Huawei Cloud mirror certificate

The Huawei Cloud package index uses HTTPS, so --trusted-host is not normally required. If pip reports an SSL certificate verification or untrusted-host error for mirrors.huaweicloud.com, first update pip and the operating system CA certificates, or configure the CA bundle required by your network.

As a temporary workaround on a trusted network, retry the vLLM Ascend installation command with --trusted-host mirrors.huaweicloud.com.

This option tells pip to trust the host even when HTTPS validation fails, which weakens protection against man-in-the-middle attacks. Use it only when you trust the network and cannot fix the certificate configuration.

Check the device build type:

python - <<'PY'
from vllm_ascend._build_info import __device_type__

print("vLLM Ascend wheel device type:", __device_type__)
assert __device_type__ == "A2", __device_type__
PY

WheelNext selects a vLLM Ascend wheel that matches the hardware from the variant index. First, install and verify uv:

# install uv-wheelnext
curl -LsSf https://astral.sh/uv/install.sh | sed 's/verify_checksum "$_file"/true/' | INSTALLER_DOWNLOAD_URL=https://wheelnext.astral.sh sh
source $HOME/.local/bin/env

# Install vllm-project/vllm. The newest supported version is v0.23.0.
pip install vllm==0.23.0

# Install vllm-project/vllm-ascend from wheelnext index.
uv pip install --system \
    --extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi/variant \
    --index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple \
    --find-links https://mirrors.huaweicloud.com/ascend/repos/pypi/triton-ascend/ \
    vllm-ascend==0.23.0

Clear the cache if uv installation fails

If uv pip install fails because of a corrupted cache or stale package data, clear the uv cache before running the installation command again:

uv cache clean

Check the device build type:

python - <<'PY'
from vllm_ascend._build_info import __device_type__
print("vLLM Ascend wheel device type:", __device_type__)
PY

A3 source installation

When installing vLLM Ascend from source for A3, run git submodule update --init --recursive manually or make sure that the environment has internet access so the required submodules can be downloaded automatically.

Install vLLM first, then install vLLM Ascend:

# Install vLLM.
git clone --depth 1 --branch v0.23.0 https://github.com/vllm-project/vllm
cd vllm
VLLM_TARGET_DEVICE=empty pip install -e .
cd ..

# Install vLLM Ascend.
git clone --depth 1 --branch v0.23.0 https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
# git submodule update --init --recursive
export ASCEND_INDEX_URL=https://mirrors.huaweicloud.com/ascend/repos/pypi
pip install -e . --extra-index-url "${ASCEND_INDEX_URL}"
cd ..

Finally, handle triton and triton-ascend according to the hardware:

A2 / A3 / 950DTAtlas 300I DUO / Atlas 200I Pro

To ensure that Triton Ascend matches the current vLLM Ascend version, remove any existing triton and triton-ascend packages before installing the corresponding version of triton-ascend:

pip uninstall -y triton triton-ascend

pip install triton-ascend==3.2.2 \
    --extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi

Can community Triton and Triton Ascend coexist?

For Triton Ascend 3.2.0 and earlier, uninstall community Triton before installing Triton Ascend.

For Triton Ascend 3.2.1 and later, installing Triton Ascend first installs the required community Triton package and then provides its Ascend-specific implementation over the shared installation paths. If Triton is reinstalled or upgraded afterward, it may overwrite the Triton Ascend files. In that case, uninstall both packages and reinstall Triton Ascend.

For more Triton Ascend installation guidance, see Triton-Ascend > FAQ.

When installing another version of vllm-ascend, check the repository's requirements.txt for the corresponding triton-ascend version.

Atlas 300I DUO and Atlas 200I Pro do not support triton or triton-ascend. If either package is already installed in the environment, uninstall it and do not reinstall triton-ascend:

pip uninstall -y triton triton-ascend

Verify the installation¶

Go to Quick Start > Inference and run a simple inference test to verify the installation.

Additional guides¶

CPU-only build verification¶

CPU-only build verification checks whether the Python package can be built without a visible Ascend device. It does not verify NPU runtime loading, inference examples, custom kernels, or NPU-specific tests. The build process needs access to CANN Toolkit headers and libraries, so CANN Toolkit must still be installed.

First, install the Python build backend and native build tools. Editable installations use setuptools-scm directly. If no compatible wheel is available, arctic-inference also requires CMake and Ninja:

python -m pip install --upgrade \
    pip "setuptools>=64" "setuptools-scm>=8" wheel \
    attrs googleapis-common-protos \
    "cmake>=3.26" ninja

This workflow verifies only the build and therefore does not install vLLM. To continue testing vLLM and vLLM Ascend together on the main branch, use the exact vLLM commit recorded in .github/vllm-main-verified.commit and verify the combined environment as described below.

In an x86 environment, install the CPU version of PyTorch from the PyTorch CPU index before installing the remaining Ascend dependencies:

python -m pip install \
    --index-url https://download.pytorch.org/whl/cpu/ \
    torch==2.10.0 torchvision==0.25.0 torchaudio==2.10.0
python -m pip install \
    --extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi \
    torch-npu==2.10.0.post4 triton-ascend==3.2.2
python -m pip install \
    --extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi \
    -r requirements.txt

Before building vLLM Ascend, explicitly set the build target and disable automatic device backend loading:

Set the correct SOC_VERSION when no NPU is visible

If npu-smi is unavailable in the current environment, set SOC_VERSION for the target hardware before running pip install -e .:

A2: export SOC_VERSION=ascend910b1
A3: export SOC_VERSION=ascend910_9391
Atlas 300I DUO / Atlas 200I Pro: export SOC_VERSION=ascend310p1
950DT: export SOC_VERSION=ascend950dt_9582

Enable batch invariance

To enable batch invariance, set VLLM_BATCH_INVARIANT=1 before building vLLM Ascend so that the custom operator library for batch invariance is installed during installation. For usage instructions, see Batch Invariance.

export ASCEND_TOOLKIT_HOME="${ASCEND_TOOLKIT_HOME:-/usr/local/Ascend/ascend-toolkit/latest}"
export TORCH_DEVICE_BACKEND_AUTOLOAD=0
export COMPILE_CUSTOM_KERNELS=0
export SOC_VERSION=ascend910b1  # A2
python -m pip install \
    --no-build-isolation \
    --no-deps \
    --extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi \
    -e .

The explicit build dependencies above and requirements.txt provide the complete build-system dependencies before the non-isolated editable build starts. --no-build-isolation only reuses packages in the current build environment; it cannot make incompatible vLLM, PyTorch, and TorchNPU versions compatible. Before using the environment for actual workloads, run python -m pip check and resolve all reported conflicts. If no device is available, skip the inference examples and NPU-specific tests.

Note

Building custom operators requires gcc/g++ later than version 8 and C++17 or later. If you encounter a TorchNPU version conflict when running pip install -e ., use pip install --no-build-isolation -e . instead to build in the system environment.

If you encounter other compilation issues, an unexpected compiler may be in use. Before compiling, set CXX_COMPILER and C_COMPILER to the locations of g++ and gcc, respectively.

Multi-node deployment¶

Check the physical links, the status of each node, and inter-node connectivity in order.

Physical link requirements¶

The physical machines must be on the same LAN and able to communicate with each other.
All NPUs must be connected through optical modules, and all connections must be healthy.

950DT server precheck

This precheck applies only to 950DT servers. Other server series can skip it.

Prepare the HiXLEP configuration paths:

When deploying a 950DT inference service, confirm on each server that /lib/route.conf, /etc/hccl_rootinfo.json, and the /etc/hixlep directory that describes the UB link topology exist and are configured correctly. If any item is missing or misconfigured, follow the HiXLEP configuration file generation guide to generate the required content. Select the "D2D scenario" when generating /etc/hixlep.

Check each node¶

Run the following commands on each node in order. The command results should be success, and the link status should be UP:

A2A3950DT

# Check the remote switch ports
for i in {0..7}; do hccn_tool -i $i -lldp -g | grep Ifname; done
# Get the link status of the Ethernet ports (UP or DOWN)
for i in {0..7}; do hccn_tool -i $i -link -g ; done
# Check the network health status
for i in {0..7}; do hccn_tool -i $i -net_health -g ; done
# View the network detected IP configuration
for i in {0..7}; do hccn_tool -i $i -netdetect -g ; done
# View gateway configuration
for i in {0..7}; do hccn_tool -i $i -gateway -g ; done
# View NPU network configuration
cat /etc/hccn.conf

# Check the remote switch ports
for i in {0..15}; do hccn_tool -i $i -lldp -g | grep Ifname; done
# Get the link status of the Ethernet ports (UP or DOWN)
for i in {0..15}; do hccn_tool -i $i -link -g ; done
# Check the network health status
for i in {0..15}; do hccn_tool -i $i -net_health -g ; done
# View the network detected IP configuration
for i in {0..15}; do hccn_tool -i $i -netdetect -g ; done
# View gateway configuration
for i in {0..15}; do hccn_tool -i $i -gateway -g ; done
# View NPU network configuration
cat /etc/hccn.conf

# Check the remote switch ports
for i in {0..7}; do hccn_tool -i $i -lldp -g | grep Ifname; done
# Get the link status of the Ethernet ports (UP or DOWN)
for i in {0..7}; do hccn_tool -i $i -link -g ; done
# Check the network health status
for i in {0..7}; do hccn_tool -i $i -net_health -g ; done
# View the network detected IP configuration
for i in {0..7}; do hccn_tool -i $i -netdetect -g ; done
# View gateway configuration
for i in {0..7}; do hccn_tool -i $i -gateway -g ; done
# View NPU network configuration
cat /etc/hccn.conf

Verify inter-node connectivity¶

Obtain NPU IP addresses¶

A2A3950DT

for i in {0..7}; do hccn_tool -i $i -ip -g | grep ipaddr; done

for i in {0..15}; do hccn_tool -i $i -ip -g | grep ipaddr; done

for i in {0..7}; do hccn_tool -i $i -ip -g | grep ipaddr; done

Run a cross-node ping test¶

# Execute on the target node (replace with actual IP)
hccn_tool -i 0 -ping -g address x.x.x.x

Start containers on each node¶

Use the official vLLM Ascend containers described in Quick Start > Installation to quickly prepare consistent multi-node runtime environments.
Commands for multi-node model serving are outside the scope of this installation guide. Continue with the relevant Feature Tutorial or Model Tutorial.