安装#
本文档介绍了如何手动安装 vllm-ascend。
要求#
操作系统：Linux
Python：>= 3.10, < 3.12
配备昇腾 NPU 的硬件设备，通常为 Atlas 800 A2 系列。
软件：
软件
支持的版本
说明
Ascend HDK
参考文档 CANN 9.0.0
CANN 运行必需
CANN
== 9.0.0
vllm-ascend 和 torch-npu 运行必需
torch-npu
== 2.10.0
vllm-ascend 必需，无需手动安装，后续步骤会自动安装。
torch
== 2.10.0
torch-npu 和 vllm 运行必需
NNAL
== 9.0.0
libatb.so 必需，用于启用高级张量算子加速
有两种安装方式：
使用 pip：首先手动或通过 CANN 镜像准备环境，然后使用 pip 安装
vllm-ascend。使用 Docker：直接使用
vllm-ascend预构建的 Docker 镜像。
配置昇腾 CANN 环境#
在安装之前，请确保固件/驱动和 CANN 已正确安装，更多详情请参考 昇腾环境搭建指南。
配置硬件环境#
要验证 Ascend NPU 固件和驱动程序是否正确安装，请运行：
npu-smi info
更多详情请参考昇腾环境搭建指南。
配置软件环境#
准备软件环境最简单的方法是直接使用 CANN 官方镜像：
备注
CANN 预构建镜像已包含 NNAL（昇腾神经网络加速库），它为高级张量操作提供 libatb.so 支持。使用预构建镜像时无需额外安装。
# Update DEVICE according to your device (/dev/davinci[0-7])
export DEVICE=/dev/davinci7
# Update the vllm-ascend image
export IMAGE=quay.io/ascend/cann:9.0.0-910b-ubuntu22.04-py3.11
docker run --rm \
--name vllm-ascend-env \
--shm-size=1g \
--device $DEVICE \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-it $IMAGE bash
如果您使用的是 vllm-ascend 预构建的 Docker 镜像，则无需额外步骤。
完成上述步骤后，即可开始安装 vllm 和 vllm-ascend。
使用 Python 安装#
首先安装系统依赖并配置 pip 镜像源：
# Using apt-get with mirror
sed -i 's|ports.ubuntu.com|mirrors.tuna.tsinghua.edu.cn|g' /etc/apt/sources.list
apt-get update -y && apt-get install -y gcc g++ cmake libnuma-dev wget git curl jq
# Or using yum
# yum update -y && yum install -y gcc g++ cmake numactl-devel wget git curl jq
# Config pip mirror,only versions 0.11.0 and earlier are supported, if using a version later than 0.11.0, do not execute this command
pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
[可选] 如果您在 x86 架构机器上操作或使用 torch-npu 开发版本，请配置 pip 的额外索引 (extra-index)：
# For torch-npu dev version or x86 machine
pip config set global.extra-index-url "https://download.pytorch.org/whl/cpu/"
接着，您可以从预构建的 wheel 包安装 vllm 和 vllm-ascend：
# Install vllm-project/vllm. The newest supported version is v0.20.2.
pip install vllm==0.20.2
# Install vllm-project/vllm-ascend.
pip install \
--extra-index-url https://mirrors.huaweicloud.com/repository/pypi/simple \
vllm-ascend==0.20.2rc1
uv-wheelnext 安装方式仅下载基于 vllm 的增量部分，从而减少下载量。首先安装 uv-wheelnext 以支持增量 wheel 包：
# install uv-wheelnext
curl -LsSf https://astral.sh/uv/install.sh | sed 's/verify_checksum "$_file"/true/' | INSTALLER_DOWNLOAD_URL=https://wheelnext.astral.sh sh
source $HOME/.local/bin/env
# Install vllm-project/vllm. The newest supported version is v0.20.2.
pip install vllm==0.20.2
# Install vllm-project/vllm-ascend from wheelnext index.
uv pip install --system \
--extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi/variant \
vllm-ascend==0.20.2rc1
点击此处查看“从源代码构建”
或者从源代码构建：
备注
要安装 triton-ascend，请运行：
pip install triton-ascend==3.2.1 --extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi
如果您通过 uv 安装，请确保在所有其他包安装完成之后再安装 triton-ascend，以避免依赖解析冲突。
# Install vLLM.
git clone --depth 1 --branch v0.20.2 https://github.com/vllm-project/vllm
cd vllm
VLLM_TARGET_DEVICE=empty pip install -e .
cd ..
# Install vLLM Ascend.
git clone --depth 1 --branch v0.20.2rc1 https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
git submodule update --init --recursive
pip install -e .
cd ..
如果您正在为 Atlas A3 构建自定义算子，请手动执行 git submodule update --init --recursive，或确保您的环境可以访问互联网。
备注
构建自定义算子需要 gcc/g++ 版本高于 8 且支持 C++17 或更高标准。如果您在使用 pip install -e . 时遇到 torch-npu 版本冲突，请使用 pip install --no-build-isolation -e . 以在系统环境下构建。如果在编译过程中遇到其他问题，通常是因为使用了非预期的编译器，您可以在编译前导出 CXX_COMPILER 和 C_COMPILER 环境变量来指定 g++ 和 gcc 的具体路径。
如果您在仅 CPU 的环境（无法使用 npu-smi）中构建，需要在执行 pip install -e . 之前设置 SOC_VERSION，以便构建过程针对正确的芯片。您可以参考 Dockerfile* 中的默认值，例如：
Atlas A2：
export SOC_VERSION=ascend910b1Atlas A3：
export SOC_VERSION=ascend910_9391Atlas 300I：
export SOC_VERSION=ascend310p1Atlas A5：
export SOC_VERSION=<以 "ascend950" 开头的值>
使用 Docker 安装#
vllm-ascend 提供了用于部署的 Docker 镜像。您可以直接从镜像仓库 ascend/vllm-ascend 拉取预构建镜像并运行。
支持的镜像如下：
镜像名称 |
硬件 |
操作系统 |
|---|---|---|
vllm-ascend:v0.20.2rc1 |
Atlas A2 |
Ubuntu |
vllm-ascend:v0.20.2rc1-openeuler |
Atlas A2 |
openEuler |
vllm-ascend:v0.20.2rc1-a3 |
Atlas A3 |
Ubuntu |
vllm-ascend:v0.20.2rc1-a3-openeuler |
Atlas A3 |
openEuler |
vllm-ascend:v0.20.2rc1-310p |
Atlas 300I |
Ubuntu |
vllm-ascend:v0.20.2rc1-310p-openeuler |
Atlas 300I |
openEuler |
点击此处查看“从 Dockerfile 构建”
或从源代码构建镜像：
git clone https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
docker build -t vllm-ascend-dev-image:latest -f ./Dockerfile .
# Update --device according to your device (Atlas A2: /dev/davinci[0-7] Atlas A3:/dev/davinci[0-15]).
# Update the vllm-ascend image according to your environment.
# Note you should download the weight to /root/.cache in advance.
export IMAGE=quay.io/ascend/vllm-ascend:v0.20.2rc1
docker run --rm \
--name vllm-ascend-env \
--shm-size=1g \
--net=host \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
--device /dev/davinci3 \
--device /dev/davinci4 \
--device /dev/davinci5 \
--device /dev/davinci6 \
--device /dev/davinci7 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-it $IMAGE bash
默认工作目录为 /workspace，vLLM 和 vLLM Ascend 代码位于 /vllm-workspace，并以开发模式 (pip install -e) 安装，以便开发者无需重新安装即可立即应用更改。
额外信息#
验证安装#
创建并运行一个简单的推理测试。example.py 内容示例如下：
from vllm import LLM, SamplingParams
prompts = [
"Hello, my name is",
"The president of the United States is",
"The capital of France is",
"The future of AI is",
]
# Create a sampling params object.
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
# Create an LLM.
llm = LLM(model="Qwen/Qwen3-0.6B")
# Generate texts from the prompts.
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
然后运行：
python example.py
如果遇到 Hugging Face 连接错误（例如：We couldn't connect to 'https://huggingface.co' to load the files, and couldn't find them in the cached files.），请运行以下命令以使用 ModelScope 作为替代方案：
export VLLM_USE_MODELSCOPE=True
pip install modelscope
python example.py
输出结果将类似于：
INFO 05-12 11:29:25 [__init__.py:44] Available plugins for group vllm.platform_plugins:
INFO 05-12 11:29:25 [__init__.py:46] - ascend -> vllm_ascend:register
INFO 05-12 11:29:25 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 05-12 11:29:25 [__init__.py:239] Platform plugin ascend is activated
INFO 05-12 11:29:29 [__init__.py:110] Registered model loader `<class 'vllm_ascend.model_loader.netloader.netloader.ModelNetLoaderElastic'>` with load format `netloader`
INFO 05-12 11:29:29 [__init__.py:110] Registered model loader `<class 'vllm_ascend.model_loader.rfork.rfork_loader.RForkModelLoader'>` with load format `rfork`
INFO 05-12 11:29:29 [utils.py:233] non-default args: {'disable_log_stats': True, 'model': 'Qwen/Qwen3-0.6B'}
INFO 05-12 11:29:29 [model.py:533] Resolved architecture: Qwen3ForCausalLM
INFO 05-12 11:29:29 [model.py:1582] Using max model len 40960
INFO 05-12 11:29:29 [scheduler.py:231] Chunked prefill is enabled with max_num_batched_tokens=8192.
INFO 05-12 11:29:29 [vllm.py:754] Asynchronous scheduling is enabled.
WARNING 05-12 11:29:29 [platform.py:765] Parameter '--disable-cascade-attn' is a GPU-specific feature. Resetting to False for Ascend.
WARNING 05-12 11:29:29 [platform.py:854] Ignored parameter 'disable_flashinfer_prefill'. This is a GPU-specific feature not supported on Ascend. Resetting to False.
INFO 05-12 11:29:29 [ascend_config.py:425] Dynamic EPLB is False
INFO 05-12 11:29:29 [ascend_config.py:426] The number of redundant experts is 0
INFO 05-12 11:29:29 [platform.py:370] PIECEWISE compilation enabled on NPU. use_inductor not supported - using only ACL Graph mode
INFO 05-12 11:29:29 [utils.py:549] Calculated maximum supported batch sizes for ACL graph: 48
WARNING 05-12 11:29:29 [utils.py:550] Currently, communication is performed using FFTS+ method, which reduces the number of available streams and, as a result, limits the range of runtime shapes that can be handled. To both improve communication performance and increase the number of supported shapes, set HCCL_OP_EXPANSION_MODE=AIV.
INFO 05-12 11:29:29 [utils.py:582] No adjustment needed for ACL graph batch sizes: Qwen3ForCausalLM model (layers: 36) with 35 sizes
INFO 05-12 11:29:29 [utils.py:1186] Block size is set to 128 if prefix cache or chunked prefill is enabled.
INFO 05-12 11:29:29 [platform.py:518] Set PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
INFO 05-12 11:29:29 [compilation.py:289] Enabled custom fusions: norm_quant, act_quant
(EngineCore pid=970) INFO 05-12 11:29:29 [core.py:103] Initializing a V1 LLM engine (v0.18.0) with config: model='Qwen/Qwen3-0.6B', speculative_config=None, tokenizer='Qwen/Qwen3-0.6B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=npu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3-0.6B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'vllm_ascend.compilation.compiler_interface.AscendCompiler', 'custom_ops': ['all'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update', 'vllm::mla_forward'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.PIECEWISE: 1>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(EngineCore pid=970) WARNING 05-12 11:29:30 [warnings.py:110] /usr/local/python3.11.10/lib/python3.11/site-packages/vllm_ascend/patch/worker/patch_weight_utils.py:80: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
(EngineCore pid=970) INFO 05-12 11:29:33 [parallel_state.py:1395] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://90.90.97.27:41723 backend=hccl
[W512 11:29:53.116059090 socket.cpp:209] [c10d] The hostname of the client socket cannot be retrieved. err=-3
[rank0]:[W512 11:30:33.152077370 ProcessGroupGloo.cpp:516] Warning: Unable to resolve hostname to a (local) address. Using the loopback address as fallback. Manually set the network interface to bind to with GLOO_SOCKET_IFNAME. (function operator())
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
(EngineCore pid=970) INFO 05-12 11:33:53 [parallel_state.py:1717] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
(EngineCore pid=970) INFO 05-12 11:34:36 [cpu_binding.py:329] [cpu_bind_mode] mode=global_slice rank=0 visible_npus=[0]
(EngineCore pid=970) INFO 05-12 11:34:36 [cpu_binding.py:376] The CPU allocation plan is as follows:
(EngineCore pid=970) INFO 05-12 11:34:36 [cpu_binding.py:381] NPU0: main=[2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21] acl=[22] release=[[23]]
(EngineCore pid=970) INFO 05-12 11:34:36 [cpu_binding.py:403] [migrate] NPU:0 -> NUMA [0]
(EngineCore pid=970) INFO 05-12 11:34:39 [cpu_binding.py:497] NPU0(PCI 0000:9d:00.0): sq_send_trigger_irq IRQ_ID=1113 -> CPU0, cq_update_irq IRQ_ID=1114 -> CPU1
(EngineCore pid=970) INFO 05-12 11:34:39 [model_runner_v1.py:2572] Starting to load model Qwen/Qwen3-0.6B...
(EngineCore pid=970) INFO 05-12 11:34:40 [compilation.py:942] Using OOT custom backend for compilation.
(EngineCore pid=970) INFO 05-12 11:34:40 [compilation.py:942] Using OOT custom backend for compilation.
Loading safetensors checkpoint shards: 0% Completed | 0/5 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 20% Completed | 1/5 [00:04<00:16, 4.20s/it]
Loading safetensors checkpoint shards: 40% Completed | 2/5 [00:08<00:11, 3.97s/it]
Loading safetensors checkpoint shards: 60% Completed | 3/5 [00:11<00:07, 3.83s/it]
Loading safetensors checkpoint shards: 80% Completed | 4/5 [00:14<00:03, 3.52s/it]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:16<00:00, 2.94s/it]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:16<00:00, 3.33s/it]
(EngineCore pid=970)
(EngineCore pid=970) INFO 05-12 11:34:57 [default_loader.py:384] Loading weights took 16.73 seconds
(EngineCore pid=970) INFO 05-12 11:34:57 [model_runner_v1.py:2599] Loading model weights took 15.2859 GB
(EngineCore pid=970) INFO 05-12 11:35:02 [backends.py:988] Using cache directory: /root/.cache/vllm/torch_compile_cache/594f71dc42/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=970) INFO 05-12 11:35:02 [backends.py:1048] Dynamo bytecode transform time: 3.98 s
(EngineCore pid=970) INFO 05-12 11:35:32 [backends.py:387] Compiling a graph for compile range (1, 8192) takes 12.79 s
(EngineCore pid=970) INFO 05-12 11:35:42 [monitor.py:48] torch.compile and initial profiling/warmup run together took 44.36 s in total
(EngineCore pid=970) INFO 05-12 11:35:44 [worker.py:357] Available KV cache memory: 39.06 GiB
(EngineCore pid=970) INFO 05-12 11:35:44 [kv_cache_utils.py:1316] GPU KV cache size: 284,416 tokens
(EngineCore pid=970) INFO 05-12 11:35:44 [kv_cache_utils.py:1321] Maximum concurrency for 40,960 tokens per request: 6.94x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:11<00:00, 3.03it/s]
(EngineCore pid=970) INFO 05-12 11:35:59 [gpu_model_runner.py:5746] Graph capturing finished in 12 secs, took 0.13 GiB
(EngineCore pid=970) INFO 05-12 11:35:59 [core.py:281] init engine (profile, create kv cache, warmup model) took 61.45 seconds
(EngineCore pid=970) INFO 05-12 11:36:00 [platform.py:370] PIECEWISE compilation enabled on NPU. use_inductor not supported - using only ACL Graph mode
(EngineCore pid=970) INFO 05-12 11:36:00 [utils.py:549] Calculated maximum supported batch sizes for ACL graph: 48
(EngineCore pid=970) WARNING 05-12 11:36:00 [utils.py:550] Currently, communication is performed using FFTS+ method, which reduces the number of available streams and, as a result, limits the range of runtime shapes that can be handled. To both improve communication performance and increase the number of supported shapes, set HCCL_OP_EXPANSION_MODE=AIV.
(EngineCore pid=970) INFO 05-12 11:36:00 [utils.py:582] No adjustment needed for ACL graph batch sizes: Qwen3ForCausalLM model (layers: 36) with 35 sizes
(EngineCore pid=970) INFO 05-12 11:36:00 [platform.py:518] Set PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
INFO 05-12 11:36:00 [llm.py:391] Supported tasks: ['generate']
Rendering prompts: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 92.51it/s]
Processed prompts: 0%| | 0/4 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s](EngineCore pid=970) INFO 05-12 11:36:00 [acl_graph.py:196] Replaying aclgraph
Processed prompts: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 8.75it/s, est. speed input: 48.17 toks/s, output: 140.12 toks/s]
Prompt: 'Hello, my name is', Generated text: ' Lucy and I am an 8 year old who loves to draw and write stories'
Prompt: 'The president of the United States is', Generated text: " a key leader in the federal government, and the president's role in the executive"
Prompt: 'The capital of France is', Generated text: ' a city. What is the capital of France? The capital of France is Paris'
Prompt: 'The future of AI is', Generated text: ' a topic that is being discussed in various contexts. In the business world, AI'
(EngineCore pid=970) INFO 05-12 11:36:00 [core.py:1201] Shutdown initiated (timeout=0)
(EngineCore pid=970) INFO 05-12 11:36:00 [core.py:1224] Shutdown complete
ERROR 05-12 11:36:01 [core_client.py:704] Engine core proc EngineCore died unexpectedly, shutting down client.
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute
多节点部署#
验证多节点通信#
首先检查物理层连通性，然后验证每个节点，最后验证节点间连通性。
物理层要求#
物理机必须位于同一局域网内，且网络互通。
所有 NPU 均通过光模块连接，且连接状态必须正常。
单节点验证#
在每个节点上依次执行以下命令。结果必须全部为 success 且状态为 UP：
# Check the remote switch ports
for i in {0..7}; do hccn_tool -i $i -lldp -g | grep Ifname; done
# Get the link status of the Ethernet ports (UP or DOWN)
for i in {0..7}; do hccn_tool -i $i -link -g ; done
# Check the network health status
for i in {0..7}; do hccn_tool -i $i -net_health -g ; done
# View the network detected IP configuration
for i in {0..7}; do hccn_tool -i $i -netdetect -g ; done
# View gateway configuration
for i in {0..7}; do hccn_tool -i $i -gateway -g ; done
# View NPU network configuration
cat /etc/hccn.conf
# Check the remote switch ports
for i in {0..15}; do hccn_tool -i $i -lldp -g | grep Ifname; done
# Get the link status of the Ethernet ports (UP or DOWN)
for i in {0..15}; do hccn_tool -i $i -link -g ; done
# Check the network health status
for i in {0..15}; do hccn_tool -i $i -net_health -g ; done
# View the network detected IP configuration
for i in {0..15}; do hccn_tool -i $i -netdetect -g ; done
# View gateway configuration
for i in {0..15}; do hccn_tool -i $i -gateway -g ; done
# View NPU network configuration
cat /etc/hccn.conf
互联验证#
1.获取 NPU IP 地址#
for i in {0..7}; do hccn_tool -i $i -ip -g | grep ipaddr; done
for i in {0..15}; do hccn_tool -i $i -ip -g | grep ipaddr; done
2.跨节点 PING 测试#
# Execute on the target node (replace with actual IP)
hccn_tool -i 0 -ping -g address x.x.x.x
在每个节点运行容器#
使用 vLLM-ascend 官方容器运行多节点环境效率更高。
在每个节点运行以下命令以启动容器（您应提前将权重下载到 /root/.cache）：
# Update the vllm-ascend image
# openEuler:
# export IMAGE=quay.io/ascend/vllm-ascend:v0.20.2rc1-openeuler
# Ubuntu:
# export IMAGE=quay.io/ascend/vllm-ascend:v0.20.2rc1
export IMAGE=quay.io/ascend/vllm-ascend:v0.20.2rc1
# Run the container using the defined variables
# Note if you are running bridge network with docker, Please expose available ports
# for multiple nodes communication in advance
docker run --rm \
--name vllm-ascend \
--net=host \
--shm-size=1g \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
--device /dev/davinci3 \
--device /dev/davinci4 \
--device /dev/davinci5 \
--device /dev/davinci6 \
--device /dev/davinci7 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-it $IMAGE bash
# Update the vllm-ascend image
# openEuler:
# export IMAGE=quay.io/ascend/vllm-ascend:v0.20.2rc1-a3-openeuler
# Ubuntu:
# export IMAGE=quay.io/ascend/vllm-ascend:v0.20.2rc1-a3
export IMAGE=quay.io/ascend/vllm-ascend:v0.20.2rc1-a3
# Run the container using the defined variables
# Note if you are running bridge network with docker, Please expose available ports
# for multiple nodes communication in advance
docker run --rm \
--name vllm-ascend \
--net=host \
--shm-size=1g \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
--device /dev/davinci3 \
--device /dev/davinci4 \
--device /dev/davinci5 \
--device /dev/davinci6 \
--device /dev/davinci7 \
--device /dev/davinci8 \
--device /dev/davinci9 \
--device /dev/davinci10 \
--device /dev/davinci11 \
--device /dev/davinci12 \
--device /dev/davinci13 \
--device /dev/davinci14 \
--device /dev/davinci15 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-it $IMAGE bash