快速入门

快速入门#

简介#

本节将引导您完成基于容器的环境搭建和大模型推理，以 Qwen3-0.6B 离线单卡推理脚本为例。

有关使用不同模型的详细信息，请参见"模型教程"目录中的相应模型教程，例如 Qwen3-30B-A3B。
有关使用不同功能的详细信息，请参见"功能教程"目录中的相应功能教程，例如 Prefill-Decode Disaggregation (Deepseek)。

先决条件#

支持的设备#

Atlas A2 训练系列（Atlas 800T A2、Atlas 900 A2 PoD、Atlas 200T A2 Box16、Atlas 300T A2）
Atlas 800I A2 推理系列（Atlas 800I A2）
Atlas A3 训练系列（Atlas 800T A3、Atlas 900 A3 SuperPoD、Atlas 9000 A3 SuperPoD）
Atlas 800I A3 推理系列（Atlas 800I A3）
[实验性] Atlas 300I 推理系列（Atlas 300I Duo）

Requirements#

OS: Linux
Python: >= 3.10, < 3.13
Hardware with Ascend NPUs. It's usually the Atlas 800 A2 series.

Software:

Software	Supported version	Note
Ascend HDK	Refer to the documentation CANN 9.0.0	Required for CANN
CANN	== 9.0.0	Required for vllm-ascend and torch-npu
torch-npu	== 2.10.0	Required for vllm-ascend, No need to install manually, it will be auto installed in below steps
torch	== 2.10.0	Required for torch-npu and vllm, No need to install manually, it will be auto installed in below steps
NNAL	== 9.0.0	Required for libatb.so, enables advanced tensor operations

使用容器设置环境#

在使用容器之前，请确保您的系统已安装 Docker。如果尚未安装 Docker，请参考 Docker 安装指南进行安装。

Ubuntu

# Update DEVICE according to your device (/dev/davinci[0-7])
export DEVICE=/dev/davinci0
# Update the vllm-ascend image
# Atlas A2:
# export IMAGE=quay.io/ascend/vllm-ascend:v0.18.0
# Atlas A3:
# export IMAGE=quay.io/ascend/vllm-ascend:v0.18.0-a3
export IMAGE=quay.io/ascend/vllm-ascend:v0.18.0
docker run --rm \
--name vllm-ascend \
--shm-size=1g \
--device $DEVICE \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-p 8000:8000 \
-it $IMAGE bash
# Install curl
apt-get update -y && apt-get install -y curl

openEuler

# Update DEVICE according to your device (/dev/davinci[0-7])
export DEVICE=/dev/davinci0
# Update the vllm-ascend image
# Atlas A2:
# export IMAGE=quay.io/ascend/vllm-ascend:v0.18.0-openeuler
# Atlas A3:
# export IMAGE=quay.io/ascend/vllm-ascend:v0.18.0-a3-openeuler
export IMAGE=quay.io/ascend/vllm-ascend:v0.18.0-openeuler
docker run --rm \
--name vllm-ascend \
--shm-size=1g \
--device $DEVICE \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-p 8000:8000 \
-it $IMAGE bash
# Install curl
yum update -y && yum install -y curl

默认工作目录为 /workspace，vLLM 和 vLLM Ascend 代码位于 /vllm-workspace 目录下，并以开发模式（pip install -e）安装，以便开发者能够即时生效更改，而无需重新安装。

用法#

您可以使用 ModelScope 镜像来加速下载：

export VLLM_USE_MODELSCOPE=True

在昇腾 NPU 上启动 vLLM 有两种方式：

离线批量推理

安装 vLLM 后，您可以开始为一系列输入提示生成文本（即离线批量推理）。

Create and run a simple inference test. The example.py can be like:

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
# The first run will take about 3-5 mins (10 MB/s) to download models
llm = LLM(model="Qwen/Qwen3-0.6B")

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Then run:

python example.py

If you encounter a connection error with Hugging Face (e.g., We couldn't connect to 'https://huggingface.co' to load the files, and couldn't find them in the cached files.), run the following commands to use ModelScope as an alternative:

export VLLM_USE_MODELSCOPE=True
pip install modelscope
python example.py

This section shows ascend platform is successfully detected in vllm:

INFO 05-27 11:40:38 [__init__.py:44] Available plugins for group vllm.platform_plugins:
INFO 05-27 11:40:38 [__init__.py:46] - ascend -> vllm_ascend:register
INFO 05-27 11:40:38 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 05-27 11:40:38 [__init__.py:238] Platform plugin ascend is activated

This section shows the final output:

Prompt: 'Hello, my name is', Generated text: ' Lucy and I am an 8 year old who loves to draw and write stories'
Prompt: 'The president of the United States is', Generated text: " a key leader in the federal government, and the president's role in the executive"
Prompt: 'The capital of France is', Generated text: ' a city. What is the capital of France? The capital of France is Paris'
Prompt: 'The future of AI is', Generated text: ' a topic that is being discussed in various contexts. In the business world, AI'

This section shows process exits after offline inference, and does not affect actual inference:

(EngineCore pid=970) INFO 05-12 11:36:00 [core.py:1201] Shutdown initiated (timeout=0)
(EngineCore pid=970) INFO 05-12 11:36:00 [core.py:1224] Shutdown complete
ERROR 05-12 11:36:01 [core_client.py:704] Engine core proc EngineCore died unexpectedly, shutting down client.
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute

OpenAI Completions API

vLLM 也可以部署为实现 OpenAI API 协议的服务器。运行以下命令，使用 Qwen/Qwen3-0.6B 模型启动 vLLM 服务器：

# Deploy vLLM server (The first run will take about 3-5 mins (10 MB/s) to download models)
vllm serve Qwen/Qwen3-0.6B &

如果您看到如下日志：

INFO:     Started server process [3594]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

恭喜，您已成功启动 vLLM 服务器！

您可以查询模型列表：

curl http://localhost:8000/v1/models | python3 -m json.tool

您也可以通过输入提示来查询模型：

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen3-0.6B",
        "prompt": "Beijing is a",
        "max_completion_tokens": 5,
        "temperature": 0
    }' | python3 -m json.tool

vLLM 正作为后台进程运行，您可以使用 kill -2 $VLLM_PID 来优雅地停止后台进程，这类似于使用 Ctrl-C 停止前台 vLLM 进程：

  VLLM_PID=$(pgrep -f "vllm serve")
  kill -2 "$VLLM_PID"

输出如下：

INFO:     Shutting down FastAPI HTTP server.
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.

最后，您可以通过按 ctrl-D 退出容器。