Atlas 300I DUO / Atlas 200I Pro¶

This tutorial covers vLLM deployment on Ascend 310P inference hardware, including Atlas 300I DUO and Atlas 200I Pro acceleration modules.

Note

Atlas 300I DUO 不支持 triton 或 triton-ascend。

在 Atlas 300I DUO 上运行 vLLM¶

安装说明¶

如果从源码安装，vllm 和 vllm-ascend 可能会自动引入 triton 和 triton-ascend 依赖，这可能在 Atlas 300I DUO 上导致意外问题。请在 Atlas 300I DUO 上运行前卸载它们：

pip uninstall -y triton-ascend triton

图模式说明¶

Warning

当前版本在 Atlas 300I DUO 设备上支持 FULL_DECODE_ONLY 图模式，但由于硬件事件ID资源限制，存在以下限制：

当启用多个张量并行（TP）rank时，可捕获的图数量有限，且取决于模型深度。例如，Qwen3-32B 可以捕获并重放 2 个图。
当 TP=1 时，没有此限制。
我们已联系相关专家寻求解决方案。基于软件的修复被认为是可行的，但完全支持需要额外时间。感谢您的理解。

部署¶

根据您的硬件形态选择启动命令。

Atlas 300I DUO¶

运行 Docker 容器：

# Use the vllm-ascend image
export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1-310p

docker run --rm \
--name vllm-ascend \
--shm-size=10g \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
--device /dev/davinci3 \
--device /dev/davinci4 \
--device /dev/davinci5 \
--device /dev/davinci6 \
--device /dev/davinci7 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-p 8080:8080 \
-it $IMAGE bash

Atlas 200I Pro 加速模块¶

When starting a container on Atlas 200I Pro, mount additional driver libraries and configuration files required by npu-smi. Without these mounts, npu-smi commands may fail inside the container. Use the command below for your container OS.

Note

Atlas 200I Pro also uses Ascend 310P. Adjust --device=/dev/davinci0 according to the NPU ID you want to use. You can query available devices with ll /dev/ | grep davinci.

Ubuntu 24.04openEuler 24.03

export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1-310p

docker run --rm \
--privileged \
--name vllm-ascend \
--shm-size=10g \
--device=/dev/davinci0:/dev/davinci0 \
--device=/dev/davinci_manager \
--device=/dev/ascend_manager \
--device=/dev/user_config \
-v /etc/sys_version.conf:/etc/sys_version.conf \
-v /etc/ld.so.conf.d/mind_so.conf:/etc/ld.so.conf.d/mind_so.conf \
-v /etc/hdcBasic.cfg:/etc/hdcBasic.cfg \
-v /var/dmp_daemon:/var/dmp_daemon \
-v /usr/lib64/libmmpa.so:/usr/lib64/libmmpa.so \
-v /usr/lib64/libcrypto.so.1.1:/usr/lib64/libcrypto.so.1.1 \
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
-v /usr/lib64/libstackcore.so:/usr/lib64/libstackcore.so \
-v /usr/lib/aarch64-linux-gnu/libyaml-0.so.2:/usr/lib64/libyaml-0.so.2 \
-v /etc/slog.conf:/etc/slog.conf \
-v /var/slogd:/var/slogd \
-v /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64 \
-v /usr/lib64/libtensorflow.so:/usr/lib64/libtensorflow.so \
-v /root/.cache:/root/.cache \
-p 8080:8080 \
-it $IMAGE bash

export IMAGE=quay.io/ascend/vllm-ascend:v0.22.1rc1-310p-openeuler

docker run --rm \
--privileged \
--name vllm-ascend \
--shm-size=10g \
--device=/dev/davinci0:/dev/davinci0 \
--device=/dev/davinci_manager \
--device=/dev/ascend_manager \
--device=/dev/user_config \
-v /etc/sys_version.conf:/etc/sys_version.conf \
-v /etc/ld.so.conf.d/mind_so.conf:/etc/ld.so.conf.d/mind_so.conf \
-v /etc/hdcBasic.cfg:/etc/hdcBasic.cfg \
-v /var/dmp_daemon:/var/dmp_daemon \
-v /usr/lib64/libsemanage.so.2:/usr/lib64/libsemanage.so.2 \
-v /usr/lib64/libmmpa.so:/usr/lib64/libmmpa.so \
-v /usr/lib64/libcrypto.so.1.1:/usr/lib64/libcrypto.so.1.1 \
-v /usr/lib64/libyaml-0.so.2.0.9:/usr/lib64/libyaml-0.so.2 \
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
-v /usr/lib64/libstackcore.so:/usr/lib64/libstackcore.so \
-v /etc/slog.conf:/etc/slog.conf \
-v /var/slogd:/var/slogd \
-v /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64 \
-v /usr/lib64/libtensorflow.so:/usr/lib64/libtensorflow.so \
-v /root/.cache:/root/.cache \
-p 8080:8080 \
-it $IMAGE bash

设置环境变量：

export VLLM_USE_MODELSCOPE=True

NPU 在线推理¶

Warning

For Atlas 300I DUO (310P), do not rely on max-model-len auto detection (that is, do not omit the --max-model-len argument), because it may cause OOM.

原因（基于当前 310P 注意力路径）：

AscendAttentionMetadataBuilder310 passes model_config.max_model_len to AttentionMaskBuilder310.
AttentionMaskBuilder310 builds a full causal mask with shape [max_model_len, max_model_len] in float16, then converts it to FRACTAL_NZ.
In the 310P attention_v1 prefill/chunked-prefill path (_npu_flash_attention / _npu_paged_attention_splitfuse), this explicit mask tensor is consumed directly, and there is currently no compressed-mask path.

If auto detection resolves to a large context length, the mask allocation (O(max_model_len^2)) may exceed NPU memory and trigger OOM. Always set an explicit and conservative value, for example --max-model-len 16384.

运行以下命令在 NPU 上为 Qwen3 Dense 系列启动 vLLM 服务器。

准备模型权重¶

使用 Eco-Tech 官方模型Scope 仓库中的 W8A8SC 量化权重。

模型	模型Scope 链接
Qwen3-8B-W8A8SC-310	Eco-Tech/Qwen3-8B-w8a8sc-310-vllm
Qwen3-14B-W8A8SC-310	Eco-Tech/Qwen3-14B-w8a8sc-310-vllm
Qwen3-32B-W8A8SC-310	Eco-Tech/Qwen3-32B-w8a8sc-310-vllm

Qwen3-8B-W8A8SCQwen3-14B-W8A8SCQwen3-32B-W8A8SC

vllm serve Eco-Tech/Qwen3-8B-w8a8sc-310-vllm/TP1/Qwen3-8B-w8a8sc-310-vllm-tp1 \
    --host 127.0.0.1 \
    --port 8080 \
    --tensor-parallel-size 1 \
    --gpu_memory_utilization 0.90 \
    --max_num_seqs 32 \
    --served_model_name qwen \
    --dtype float16 \
    --additional-config '{"ascend_compilation_config": {"fuse_norm_quant": false}}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [1,2,4,8,16,32]}' \
    --quantization ascend \
    --max_model_len 16384 \
    --no-enable-prefix-caching \
    --load_format sharded_state

vllm serve Eco-Tech/Qwen3-14B-w8a8sc-310-vllm/TP1/Qwen3-14B-w8a8sc-310-vllm-tp1 \
    --host 127.0.0.1 \
    --port 8080 \
    --tensor-parallel-size 1 \
    --gpu_memory_utilization 0.90 \
    --max_num_seqs 16 \
    --served_model_name qwen \
    --dtype float16 \
    --additional-config '{"ascend_compilation_config": {"fuse_norm_quant": false}}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [1,2,4,8,16]}' \
    --quantization ascend \
    --max_model_len 16384 \
    --no-enable-prefix-caching \
    --load_format sharded_state

export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3

vllm serve Eco-Tech/Qwen3-32B-w8a8sc-310-vllm/TP4/Qwen3-32B-w8a8sc-310-vllm-tp4 \
    --host 127.0.0.1 \
    --port 8080 \
    --tensor-parallel-size 4 \
    --gpu_memory_utilization 0.90 \
    --max_num_seqs 32 \
    --served_model_name qwen \
    --dtype float16 \
    --additional-config '{"ascend_compilation_config": {"fuse_norm_quant": false}}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [16,32]}' \
    --quantization ascend \
    --max_model_len 20480 \
    --no-enable-prefix-caching \
    --load_format sharded_state

服务器启动后，您可以使用输入提示词查询模型。

curl http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "The future of AI is",
    "max_completion_tokens": 64,
    "temperature": 0.0
  }'

如果脚本运行成功，您将看到生成的结果。

离线推理¶

运行以下脚本（example.py）在 NPU 上执行离线推理。

Qwen3-8B-W8A8SCQwen3-14B-W8A8SCQwen3-32B-W8A8SC

import gc
import torch

from vllm import LLM, SamplingParams
from vllm.distributed.parallel_state import (
    destroy_distributed_environment,
    destroy_model_parallel,
)

def clean_up():
    destroy_model_parallel()
    destroy_distributed_environment()
    gc.collect()
    torch.npu.empty_cache()

prompts = [
    "Hello, my name is",
    "The future of AI is",
]

sampling_params = SamplingParams(
    max_completion_tokens=100,
    temperature=0.0,
)

llm = LLM(
    model="Eco-Tech/Qwen3-8B-w8a8sc-310-vllm/TP1/Qwen3-8B-w8a8sc-310-vllm-tp1",
    tensor_parallel_size=1,
    max_model_len=16384,
    dtype="float16",
    quantization="ascend",
    load_format="sharded_state",
    additional_config={
        "ascend_compilation_config": {
            "fuse_norm_quant": False,
        }
    },
    compilation_config={
        "cudagraph_mode": "FULL_DECODE_ONLY",
        "cudagraph_capture_sizes": [1, 2, 4, 8, 16, 32],
    },
    enable_prefix_caching=False,
)

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

del llm
clean_up()

import gc
import torch

from vllm import LLM, SamplingParams
from vllm.distributed.parallel_state import (
    destroy_distributed_environment,
    destroy_model_parallel,
)

def clean_up():
    destroy_model_parallel()
    destroy_distributed_environment()
    gc.collect()
    torch.npu.empty_cache()

prompts = [
    "Hello, my name is",
    "The future of AI is",
]

sampling_params = SamplingParams(
    max_completion_tokens=100,
    temperature=0.0,
)

llm = LLM(
    model="Eco-Tech/Qwen3-14B-w8a8sc-310-vllm/TP1/Qwen3-14B-w8a8sc-310-vllm-tp1",
    tensor_parallel_size=1,
    max_model_len=16384,
    dtype="float16",
    quantization="ascend",
    load_format="sharded_state",
    additional_config={
        "ascend_compilation_config": {
            "fuse_norm_quant": False,
        }
    },
    compilation_config={
        "cudagraph_mode": "FULL_DECODE_ONLY",
        "cudagraph_capture_sizes": [1, 2, 4, 8, 16],
    },
    enable_prefix_caching=False,
)

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

del llm
clean_up()

import gc
import os
import torch

from vllm import LLM, SamplingParams
from vllm.distributed.parallel_state import (
    destroy_distributed_environment,
    destroy_model_parallel,
)

def clean_up():
    destroy_model_parallel()
    destroy_distributed_environment()
    gc.collect()
    torch.npu.empty_cache()

os.environ["ASCEND_RT_VISIBLE_DEVICES"] = "0,1,2,3"

prompts = [
    "Hello, my name is",
    "The future of AI is",
]

sampling_params = SamplingParams(
    max_completion_tokens=100,
    temperature=0.0,
)

llm = LLM(
    model="Eco-Tech/Qwen3-32B-w8a8sc-310-vllm/TP4/Qwen3-32B-w8a8sc-310-vllm-tp4",
    tensor_parallel_size=4,
    max_model_len=20480,
    dtype="float16",
    quantization="ascend",
    load_format="sharded_state",
    additional_config={
        "ascend_compilation_config": {
            "fuse_norm_quant": False,
        }
    },
    compilation_config={
        "cudagraph_mode": "FULL_DECODE_ONLY",
        "cudagraph_capture_sizes": [16, 32],
    },
    enable_prefix_caching=False,
)

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

del llm
clean_up()

运行脚本：

python example.py

如果脚本运行成功，您将看到生成的结果。

结束语¶

如需提前体验 Qwen3-MoE、Qwen3-VL，以及 Qwen3.5 和 Qwen3.6 的预览支持（含性能加速），请关注 #7394 获取更新的部署指南。