快速入门

简介

本节以 Qwen3-0.6B 离线单卡推理脚本为例，指导您完成基于容器的环境搭建和大模型推理。

• For details on using different models, see the corresponding model tutorial in the "Model Tutorials" directory, for example, Qwen3-30B-A3B.
• For details on using different functions, see the corresponding function tutorial in the "Function Tutorials" directory, for example, Prefill-Decode Disaggregation (Deepseek).

前提条件

支持的设备

• Atlas A2 训练系列（Atlas 800T A2, Atlas 900 A2 PoD, Atlas 200T A2 Box16, Atlas 300T A2）
• Atlas 800I A2 推理系列（Atlas 800I A2）
• Atlas A3 训练系列（Atlas 800T A3, Atlas 900 A3 SuperPoD, Atlas 9000 A3 SuperPoD）
• Atlas 800I A3 推理系列（Atlas 800I A3）
• 【实验性】Atlas 300I 推理系列（Atlas 300I Duo）

要求

• 操作系统：Linux
• Python：>= 3.10, < 3.13
• 配备昇腾 NPU 的硬件，通常为 Atlas 800 A2 系列。

• 软件：

  软件	支持的版本	说明
  Ascend HDK	请参考文档 CANN 9.0.0	CANN 所需
  CANN	== 9.0.0	vllm-ascend 和 torch-npu 所需
  torch-npu	== 2.10.0	vllm-ascend 所需，无需手动安装，将在以下步骤中自动安装
  torch	== 2.10.0	torch-npu 和 vllm 所需，无需手动安装，将在以下步骤中自动安装
  NNAL	== 9.0.0	libatb.so 所需，支持高级张量操作

使用容器设置环境

使用容器前，请确保系统已安装 Docker。如果未安装 Docker，请参考 Docker 安装指南 进行安装。

UbuntuopenEuler

# Update DEVICE according to your device (/dev/davinci[0-7])
export DEVICE=/dev/davinci0
# Update the vllm-ascend image
# Atlas A2:
# export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1
# Atlas A3:
# export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1-a3
export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1
docker run --rm \
--name vllm-ascend \
--shm-size=1g \
--device $DEVICE \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-p 8000:8000 \
-it $IMAGE bash
# Install curl
apt-get update -y && apt-get install -y curl

# Update DEVICE according to your device (/dev/davinci[0-7])
export DEVICE=/dev/davinci0
# Update the vllm-ascend image
# Atlas A2:
# export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1-openeuler
# Atlas A3:
# export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1-a3-openeuler
export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1-openeuler
docker run --rm \
--name vllm-ascend \
--shm-size=1g \
--device $DEVICE \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-p 8000:8000 \
-it $IMAGE bash
# Install curl
yum update -y && yum install -y curl

The default workdir is /workspace, vLLM and vLLM Ascend code are placed in /vllm-workspace and installed in development mode (pip install -e）安装，以帮助开发者无需重新安装即可使修改立即生效。

使用方法

您可以使用 ModelScope 镜像来加速下载：

export VLLM_USE_MODELSCOPE=True

在昇腾 NPU 上启动 vLLM 有两种方式：

离线批量推理OpenAI Completions API

安装 vLLM 后，您可以开始为输入提示列表生成文本（即离线批量推理）。

创建并运行一个简单的推理测试。example.py 内容如下：

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
# The first run will take about 3-5 mins (10 MB/s) to download models
llm = LLM(model="Qwen/Qwen3-0.6B")

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Then run:

python example.py

If you encounter a connection error with Hugging Face (e.g., We couldn't connect to 'https://huggingface.co' to load the files, and couldn't find them in the cached files.), run the following commands to use ModelScope as an alternative:

export VLLM_USE_MODELSCOPE=True
pip install modelscope
python example.py

This section shows ascend platform is successfully detected in vllm:

INFO 05-27 11:40:38 [__init__.py:44] Available plugins for group vllm.platform_plugins:
INFO 05-27 11:40:38 [__init__.py:46] - ascend -> vllm_ascend:register
INFO 05-27 11:40:38 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 05-27 11:40:38 [__init__.py:238] Platform plugin ascend is activated

This section shows the final output:

Prompt: 'Hello, my name is', Generated text: ' Lucy and I am an 8 year old who loves to draw and write stories'
Prompt: 'The president of the United States is', Generated text: " a key leader in the federal government, and the president's role in the executive"
Prompt: 'The capital of France is', Generated text: ' a city. What is the capital of France? The capital of France is Paris'
Prompt: 'The future of AI is', Generated text: ' a topic that is being discussed in various contexts. In the business world, AI'

This section shows process exits after offline inference, and does not affect actual inference:

(EngineCore pid=970) INFO 05-12 11:36:00 [core.py:1201] Shutdown initiated (timeout=0)
(EngineCore pid=970) INFO 05-12 11:36:00 [core.py:1224] Shutdown complete
ERROR 05-12 11:36:01 [core_client.py:704] Engine core proc EngineCore died unexpectedly, shutting down client.
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute

vLLM can also be deployed as a server that implements the OpenAI API protocol. Run the following command to start the vLLM server with the Qwen/Qwen3-0.6B model:

# Deploy vLLM server (The first run will take about 3-5 mins (10 MB/s) to download models)
vllm serve Qwen/Qwen3-0.6B &

If you see a log as below:

INFO:     Started server process [3594]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Congratulations, you have successfully started the vLLM server!

You can query the list of models:

curl http://localhost:8000/v1/models | python3 -m json.tool

You can also query the model with input prompts:

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen3-0.6B",
        "prompt": "Beijing is a",
        "max_completion_tokens": 5,
        "temperature": 0
    }' | python3 -m json.tool

vLLM is serving as a background process, you can use kill -2 $VLLM_PID to stop the background process gracefully, which is similar to Ctrl-C for stopping the foreground vLLM process:

  VLLM_PID=$(pgrep -f "vllm serve")
  kill -2 "$VLLM_PID"

The output is as below:

INFO:     Shutting down FastAPI HTTP server.
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.

Finally, you can exit the container by using ctrl-D 退出容器。