# Installation This document describes how to install vllm-ascend manually. ## Requirements - OS: Linux - Python: >= 3.10, < 3.13 - Hardware with Ascend NPUs. It's usually the Atlas 800 A2 series. - Software: | Software | Supported version | Note | |---------------|----------------------------------|-------------------------------------------| | Ascend HDK | Refer to the documentation [CANN 9.0.0](https://www.hiascend.com/document/detail/zh/canncommercial/900/releasenote/releasenote_0000.html) | Required for CANN | | CANN | == 9.0.0 | Required for vllm-ascend and torch-npu | | torch-npu | == 2.10.0 | Required for vllm-ascend, No need to install manually, it will be auto installed in below steps | | torch | == 2.10.0 | Required for torch-npu and vllm | | NNAL | == 9.0.0 | Required for libatb.so, enables advanced tensor operations | There are two installation methods: - **Using pip**: first prepare the environment manually or via a CANN image, then install `vllm-ascend` using pip. - **Using docker**: use the `vllm-ascend` pre-built docker image directly. ## Configure Ascend CANN environment Before installation, you need to make sure firmware/driver, and CANN are installed correctly, refer to [Ascend Environment Setup Guide](https://ascend.github.io/docs/sources/ascend/quick_install.html) for more details. ### Configure hardware environment To verify that the Ascend NPU firmware and driver were correctly installed, run: ```bash npu-smi info ``` Refer to [Ascend Environment Setup Guide](https://ascend.github.io/docs/sources/ascend/quick_install.html) for more details. ### Configure software environment :::::{tab-set} :sync-group: install ::::{tab-item} Before using pip :selected: :sync: pip The easiest way to prepare your software environment is using CANN image directly: ```{note} The CANN prebuilt image includes NNAL (Ascend Neural Network Acceleration Library), which provides libatb.so for advanced tensor operations. No additional installation is required when using the prebuilt image. ``` ```{code-block} bash :substitutions: # Update DEVICE according to your device (/dev/davinci[0-7]) export DEVICE=/dev/davinci7 # Update the vllm-ascend image export IMAGE=quay.io/ascend/cann:|cann_image_tag| docker run --rm \ --name vllm-ascend-env \ --shm-size=1g \ --device $DEVICE \ --device /dev/davinci_manager \ --device /dev/devmm_svm \ --device /dev/hisi_hdc \ -v /usr/local/dcmi:/usr/local/dcmi \ -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \ -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \ -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /root/.cache:/root/.cache \ -it $IMAGE bash ``` :::{dropdown} Click here to see "Install CANN manually" :animate: fade-in-slide-down You can also install CANN manually: ```{warning} If you encounter "libatb.so not found" errors during runtime, please ensure NNAL is properly installed as shown in the manual installation steps below. ``` ```bash # Create a virtual environment. python -m venv vllm-ascend-env source vllm-ascend-env/bin/activate # Install required Python packages. python -m pip install --upgrade pip pip3 install attrs numpy decorator sympy cffi pyyaml pathlib2 psutil protobuf scipy requests absl-py wheel typing_extensions # Download and install the CANN package. wget --header="Referer: https://www.hiascend.com/" https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%209.0.0/Ascend-cann-toolkit_9.0.0_linux-"$(uname -i)".run chmod +x ./Ascend-cann-toolkit_9.0.0_linux-"$(uname -i)".run ./Ascend-cann-toolkit_9.0.0_linux-"$(uname -i)".run --full source /usr/local/Ascend/ascend-toolkit/set_env.sh wget --header="Referer: https://www.hiascend.com/" https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%209.0.0/Ascend-cann-910b-ops_9.0.0_linux-"$(uname -i)".run chmod +x ./Ascend-cann-910b-ops_9.0.0_linux-"$(uname -i)".run ./Ascend-cann-910b-ops_9.0.0_linux-"$(uname -i)".run --install wget --header="Referer: https://www.hiascend.com/" https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%209.0.0/Ascend-cann-nnal_9.0.0_linux-"$(uname -i)".run chmod +x ./Ascend-cann-nnal_9.0.0_linux-"$(uname -i)".run ./Ascend-cann-nnal_9.0.0_linux-"$(uname -i)".run --install source /usr/local/Ascend/nnal/atb/set_env.sh ``` ::: :::: ::::{tab-item} Before using docker :sync: docker No extra steps are needed if you are using the `vllm-ascend` prebuilt Docker image. :::: ::::: Once this is done, you can start to set up `vllm` and `vllm-ascend`. ## Set up using Python First, install system dependencies and configure the pip mirror: ```bash # Using apt-get with mirror sed -i 's|ports.ubuntu.com|mirrors.tuna.tsinghua.edu.cn|g' /etc/apt/sources.list apt-get update -y && apt-get install -y gcc g++ cmake libnuma-dev wget git curl jq # Or using yum # yum update -y && yum install -y gcc g++ cmake numactl-devel wget git curl jq # Config pip mirror,only versions 0.11.0 and earlier are supported, if using a version later than 0.11.0, do not execute this command pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple ``` **[Optional]** Then configure the extra-index of `pip` if you are working on an x86 machine or using torch-npu dev version: ```bash # For torch-npu dev version or x86 machine pip config set global.extra-index-url "https://download.pytorch.org/whl/cpu/" ``` Then you can install `vllm` and `vllm-ascend` from a **pre-built wheel** using one of the following methods: :::::{tab-set} :sync-group: install-method ::::{tab-item} Original installation :sync: original ```{code-block} bash :substitutions: # Install vllm-project/vllm. The newest supported version is |vllm_version|. pip install vllm==|pip_vllm_version| # Install vllm-project/vllm-ascend. pip install \ --extra-index-url https://mirrors.huaweicloud.com/repository/pypi/simple \ vllm-ascend==|pip_vllm_ascend_version| ``` :::: ::::{tab-item} uv-wheelnext installation :sync: uv-wheelnext The `uv-wheelnext` installation downloads only the delta on top of vllm, resulting in a smaller download size. First install `uv-wheelnext` to support incremental wheels: ```bash # install uv-wheelnext curl -LsSf https://astral.sh/uv/install.sh | sed 's/verify_checksum "$_file"/true/' | INSTALLER_DOWNLOAD_URL=https://wheelnext.astral.sh sh source $HOME/.local/bin/env ``` ```{code-block} bash :substitutions: # Install vllm-project/vllm. The newest supported version is |vllm_version|. pip install vllm==|pip_vllm_version| # Install vllm-project/vllm-ascend from wheelnext index. uv pip install --system \ --extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi/variant \ vllm-ascend==|pip_vllm_ascend_version| ``` :::: ::::: :::{dropdown} Click here to see "Build from source code" or build from **source code**: ```{note} To install `triton-ascend`, run: pip install triton-ascend==3.2.1 --extra-index-url https://mirrors.huaweicloud.com/ascend/repos/pypi If you are installing via `uv`, make sure to install `triton-ascend` **last**, after all other packages have been installed, to avoid dependency resolution conflicts. ``` ```{code-block} bash :substitutions: # Install vLLM. git clone --depth 1 --branch |vllm_version| https://github.com/vllm-project/vllm cd vllm VLLM_TARGET_DEVICE=empty pip install -e . cd .. # Install vLLM Ascend. git clone --depth 1 --branch |vllm_ascend_version| https://github.com/vllm-project/vllm-ascend.git cd vllm-ascend git submodule update --init --recursive pip install -e . cd .. ``` If you are building custom operators for Atlas A3, you should run `git submodule update --init --recursive` manually, or ensure your environment has internet access. ::: ```{note} To build custom operators, gcc/g++ higher than 8 and C++17 or higher are required. If you are using `pip install -e .` and encounter a torch-npu version conflict, please install with `pip install --no-build-isolation -e .` to build on system env. If you encounter other problems during compiling, it is probably because an unexpected compiler is being used, you may export `CXX_COMPILER` and `C_COMPILER` in the environment to specify your g++ and gcc locations before compiling. If you are building in a CPU-only environment where `npu-smi` is unavailable, you need to set `SOC_VERSION` before `pip install -e .` so the build can target the correct chip. You can refer to `Dockerfile*` defaults, for example: - Atlas A2: `export SOC_VERSION=ascend910b1` - Atlas A3: `export SOC_VERSION=ascend910_9391` - Atlas 300I: `export SOC_VERSION=ascend310p1` - Atlas A5: `export SOC_VERSION=` ``` ## Set up using Docker `vllm-ascend` offers Docker images for deployment. You can just pull the **prebuilt image** from the image repository [ascend/vllm-ascend](https://quay.io/repository/ascend/vllm-ascend?tab=tags) and run it with bash. Supported images as following. | image name | Hardware | OS | |-|-|-| | vllm-ascend:{{ vllm_ascend_version }} | Atlas A2 | Ubuntu | | vllm-ascend:{{ vllm_ascend_version }}-openeuler | Atlas A2 | openEuler | | vllm-ascend:{{ vllm_ascend_version }}-a3 | Atlas A3 | Ubuntu | | vllm-ascend:{{ vllm_ascend_version }}-a3-openeuler | Atlas A3 | openEuler | | vllm-ascend:{{ vllm_ascend_version }}-310p | Atlas 300I | Ubuntu | | vllm-ascend:{{ vllm_ascend_version }}-310p-openeuler | Atlas 300I | openEuler | :::{dropdown} Click here to see "Build from Dockerfile" or build IMAGE from **source code**: ```bash git clone https://github.com/vllm-project/vllm-ascend.git cd vllm-ascend docker build -t vllm-ascend-dev-image:latest -f ./Dockerfile . ``` ::: ```{code-block} bash :substitutions: # Update --device according to your device (Atlas A2: /dev/davinci[0-7] Atlas A3:/dev/davinci[0-15]). # Update the vllm-ascend image according to your environment. # Note you should download the weight to /root/.cache in advance. export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version| docker run --rm \ --name vllm-ascend-env \ --shm-size=1g \ --net=host \ --device /dev/davinci0 \ --device /dev/davinci1 \ --device /dev/davinci2 \ --device /dev/davinci3 \ --device /dev/davinci4 \ --device /dev/davinci5 \ --device /dev/davinci6 \ --device /dev/davinci7 \ --device /dev/davinci_manager \ --device /dev/devmm_svm \ --device /dev/hisi_hdc \ -v /usr/local/dcmi:/usr/local/dcmi \ -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \ -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \ -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \ -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /root/.cache:/root/.cache \ -it $IMAGE bash ``` The default workdir is `/workspace`, vLLM and vLLM Ascend code are placed in `/vllm-workspace` and installed in [development mode](https://setuptools.pypa.io/en/latest/userguide/development_mode.html) (`pip install -e`) to help developers immediately make changes without requiring a new installation. ## Extra information ### Verify installation Create and run a simple inference test. The `example.py` can be like: ```python from vllm import LLM, SamplingParams prompts = [ "Hello, my name is", "The president of the United States is", "The capital of France is", "The future of AI is", ] # Create a sampling params object. sampling_params = SamplingParams(temperature=0.8, top_p=0.95) # Create an LLM. llm = LLM(model="Qwen/Qwen3-0.6B") # Generate texts from the prompts. outputs = llm.generate(prompts, sampling_params) for output in outputs: prompt = output.prompt generated_text = output.outputs[0].text print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") ``` Then run: ```bash python example.py ``` If you encounter a connection error with Hugging Face (e.g., `We couldn't connect to 'https://huggingface.co' to load the files, and couldn't find them in the cached files.`), run the following commands to use ModelScope as an alternative: ```bash export VLLM_USE_MODELSCOPE=True pip install modelscope python example.py ``` The output is shown below, and it may change with version updates: ```bash INFO 05-27 11:40:38 [__init__.py:44] Available plugins for group vllm.platform_plugins: INFO 05-27 11:40:38 [__init__.py:46] - ascend -> vllm_ascend:register INFO 05-27 11:40:38 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load. INFO 05-27 11:40:38 [__init__.py:238] Platform plugin ascend is activated INFO 05-27 11:40:43 [nixl_utils.py:20] Setting UCX_RCACHE_MAX_UNRELEASED to '1024' to avoid a rare memory leak in UCX when using NIXL. INFO 05-27 11:40:43 [__init__.py:110] Registered model loader `` with load format `netloader` INFO 05-27 11:40:43 [__init__.py:110] Registered model loader `` with load format `rfork` INFO 05-27 11:40:43 [utils.py:233] non-default args: {'disable_log_stats': True, 'model': 'Qwen/Qwen3-0.6B'} INFO 05-27 11:40:44 [model.py:555] Resolved architecture: Qwen3ForCausalLM INFO 05-27 11:40:44 [model.py:1680] Using max model len 40960 INFO 05-27 11:40:44 [scheduler.py:239] Chunked prefill is enabled with max_num_batched_tokens=8192. INFO 05-27 11:40:44 [vllm.py:840] Asynchronous scheduling is enabled. INFO 05-27 11:40:44 [kernel.py:205] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native']) INFO 05-27 11:40:44 [ascend_config.py:593] Dynamic EPLB is False INFO 05-27 11:40:44 [ascend_config.py:594] The number of redundant experts is 0 INFO 05-27 11:40:44 [platform.py:396] PIECEWISE compilation enabled on NPU. use_inductor not supported - using only ACL Graph mode INFO 05-27 11:40:44 [utils.py:607] Calculated maximum supported batch sizes for ACL graph: 62 INFO 05-27 11:40:44 [utils.py:640] No adjustment needed for ACL graph batch sizes: Qwen3ForCausalLM model (layers: 28) with 35 sizes INFO 05-27 11:40:44 [utils.py:1251] Block size is set to 128 if prefix cache or chunked prefill is enabled. INFO 05-27 11:40:44 [platform.py:569] Set PYTORCH_NPU_ALLOC_CONF=expandable_segments:True INFO 05-27 11:40:44 [compilation.py:303] Enabled custom fusions: norm_quant, act_quant (EngineCore pid=75652) INFO 05-27 11:40:44 [core.py:109] Initializing a V1 LLM engine (vx.x.x) with config: model='Qwen/Qwen3-0.6B', speculative_config=None, tokenizer='Qwen/Qwen3-0.6B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=npu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen/Qwen3-0.6B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'vllm_ascend.compilation.compiler_interface.AscendCompiler', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update', 'vllm::mla_forward'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 256, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto') (EngineCore pid=75652) INFO 05-27 11:40:48 [parallel_state.py:1402] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://90.90.97.28:38959 backend=hccl (EngineCore pid=75652) INFO 05-27 11:40:48 [parallel_state.py:1715] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A (EngineCore pid=75652) INFO 05-27 11:40:48 [model_runner_v1.py:3146] Starting to load model Qwen/Qwen3-0.6B... (EngineCore pid=75652) INFO 05-27 11:40:49 [compilation.py:1049] Using OOT custom backend for compilation. (EngineCore pid=75652) INFO 05-27 11:40:49 [compilation.py:1049] Using OOT custom backend for compilation. (EngineCore pid=75652) INFO 05-27 11:40:49 [weight_utils.py:904] Filesystem type for checkpoints: EXT4. Checkpoint size: 1.40 GiB. Available RAM: 1944.66 GiB. (EngineCore pid=75652) INFO 05-27 11:40:49 [weight_utils.py:927] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. (EngineCore pid=75652) INFO 05-27 11:40:51 [default_loader.py:384] Loading weights took 1.99 seconds (EngineCore pid=75652) INFO 05-27 11:40:51 [model_runner_v1.py:3187] Loading model weights took 1.1397 GB (EngineCore pid=75652) INFO 05-27 11:40:52 [backends.py:1069] Using cache directory: /root/.cache/vllm/torch_compile_cache/5a9d9c976f/rank_0_0/backbone for vLLM's torch.compile (EngineCore pid=75652) INFO 05-27 11:40:52 [backends.py:1128] Dynamo bytecode transform time: 0.84 s (EngineCore pid=75652) INFO 05-27 11:41:01 [backends.py:391] Compiling a graph for compile range (1, 8192) takes 7.77 s (EngineCore pid=75652) INFO 05-27 11:41:01 [decorators.py:305] Directly load AOT compilation from path /root/.cache/vllm/torch_compile_cache/torch_aot_compile/74892a710dff08ef76700608ac46ccb60eebda2bdd3037d825f26b0db836de5a/rank_0_0/model (EngineCore pid=75652) INFO 05-27 11:41:01 [monitor.py:53] torch.compile took 9.69 s in total (EngineCore pid=75652) INFO 05-27 11:41:01 [monitor.py:81] Initial profiling/warmup run took 0.07 s (EngineCore pid=75652) INFO 05-27 11:41:02 [worker.py:394] Available KV cache memory: 54.98 GiB (EngineCore pid=75652) INFO 05-27 11:41:02 [kv_cache_utils.py:1708] GPU KV cache size: 514,688 tokens (EngineCore pid=75652) INFO 05-27 11:41:02 [kv_cache_utils.py:1709] Maximum concurrency for 40,960 tokens per request: 12.57x (EngineCore pid=75652) INFO 05-27 11:41:12 [gpu_model_runner.py:6133] Graph capturing finished in 7 secs, took 0.09 GiB (EngineCore pid=75652) INFO 05-27 11:41:12 [worker.py:546] Free memory on device (60.89/61.27 GiB) on startup. Desired GPU memory utilization is (0.92, 56.37 GiB). Actual usage: 1.14 GiB for weights, 0.22 GiB for peak activation, 0.03 GiB for non-torch memory, 0.09 GiB for NPU graph memory. Replace gpu_memory_utilization with `--kv-cache-memory=58786898124` (54.75 GiB) to fit into requested memory, or `--kv-cache-memory=63643382784` (59.27 GiB) to fully utilize NPU free memory. Current KV cache memory: 54.98 GiB. (EngineCore pid=75652) INFO 05-27 11:41:15 [cpu_binding.py:328] [cpu_bind_mode] mode=global_slice rank=0 visible_npus=[0] (EngineCore pid=75652) INFO 05-27 11:41:15 [cpu_binding.py:388] The CPU allocation plan is as follows: (EngineCore pid=75652) INFO 05-27 11:41:15 [cpu_binding.py:393] NPU0: main=[2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21] acl=[22] release=[[23]] (EngineCore pid=75652) INFO 05-27 11:41:15 [cpu_binding.py:415] [migrate] NPU:0 -> NUMA [0] (EngineCore pid=75652) INFO 05-27 11:41:20 [cpu_binding.py:510] NPU0(PCI 0000:9d:00.0): sq_send_trigger_irq IRQ_ID=1037 -> CPU0, cq_update_irq IRQ_ID=1038 -> CPU1 (EngineCore pid=75652) INFO 05-27 11:41:20 [core.py:299] init engine (profile, create kv cache, warmup model) took 29.13 s (compilation: 9.69 s) (EngineCore pid=75652) INFO 05-27 11:41:21 [kernel.py:205] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native']) (EngineCore pid=75652) INFO 05-27 11:41:21 [platform.py:396] PIECEWISE compilation enabled on NPU. use_inductor not supported - using only ACL Graph mode (EngineCore pid=75652) INFO 05-27 11:41:21 [utils.py:607] Calculated maximum supported batch sizes for ACL graph: 62 (EngineCore pid=75652) INFO 05-27 11:41:21 [utils.py:640] No adjustment needed for ACL graph batch sizes: Qwen3ForCausalLM model (layers: 28) with 35 sizes (EngineCore pid=75652) INFO 05-27 11:41:21 [platform.py:569] Set PYTORCH_NPU_ALLOC_CONF=expandable_segments:True (EngineCore pid=75652) INFO 05-27 11:41:21 [acl_graph.py:198] Replaying aclgraph Prompt: 'Hello, my name is', Generated text: ' Lucy and I am an 8 year old who loves to draw and write stories' Prompt: 'The president of the United States is', Generated text: " a key leader in the federal government, and the president's role in the executive" Prompt: 'The capital of France is', Generated text: ' a city. What is the capital of France? The capital of France is Paris' Prompt: 'The future of AI is', Generated text: ' a topic that is being discussed in various contexts. In the business world, AI' (EngineCore pid=970) INFO 05-12 11:36:00 [core.py:1201] Shutdown initiated (timeout=0) (EngineCore pid=970) INFO 05-12 11:36:00 [core.py:1224] Shutdown complete ERROR 05-12 11:36:01 [core_client.py:704] Engine core proc EngineCore died unexpectedly, shutting down client. sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute ``` ## Multi-node Deployment ### Verify Multi-Node Communication First, check physical layer connectivity, then verify each node, and finally verify the inter-node connectivity. #### Physical Layer Requirements - The physical machines must be located on the same WLAN, with network connectivity. - All NPUs are connected with optical modules, and the connection status must be normal. #### Each Node Verification Execute the following commands on each node in sequence. The results must all be `success` and the status must be `UP`: :::::{tab-set} :sync-group: multi-node ::::{tab-item} A2 series :sync: A2 ```bash # Check the remote switch ports for i in {0..7}; do hccn_tool -i $i -lldp -g | grep Ifname; done # Get the link status of the Ethernet ports (UP or DOWN) for i in {0..7}; do hccn_tool -i $i -link -g ; done # Check the network health status for i in {0..7}; do hccn_tool -i $i -net_health -g ; done # View the network detected IP configuration for i in {0..7}; do hccn_tool -i $i -netdetect -g ; done # View gateway configuration for i in {0..7}; do hccn_tool -i $i -gateway -g ; done # View NPU network configuration cat /etc/hccn.conf ``` :::: ::::{tab-item} A3 series :sync: A3 ```bash # Check the remote switch ports for i in {0..15}; do hccn_tool -i $i -lldp -g | grep Ifname; done # Get the link status of the Ethernet ports (UP or DOWN) for i in {0..15}; do hccn_tool -i $i -link -g ; done # Check the network health status for i in {0..15}; do hccn_tool -i $i -net_health -g ; done # View the network detected IP configuration for i in {0..15}; do hccn_tool -i $i -netdetect -g ; done # View gateway configuration for i in {0..15}; do hccn_tool -i $i -gateway -g ; done # View NPU network configuration cat /etc/hccn.conf ``` :::: ::::: #### Interconnect Verification ##### 1. Get NPU IP Addresses :::::{tab-set} :sync-group: multi-node ::::{tab-item} A2 series :sync: A2 ```bash for i in {0..7}; do hccn_tool -i $i -ip -g | grep ipaddr; done ``` :::: ::::{tab-item} A3 series :sync: A3 ```bash for i in {0..15}; do hccn_tool -i $i -ip -g | grep ipaddr; done ``` :::: ::::: ##### 2. Cross-Node PING Test ```bash # Execute on the target node (replace with actual IP) hccn_tool -i 0 -ping -g address x.x.x.x ``` ### Run Container In Each Node Using vLLM-ascend official container is more efficient to run multi-node environment. Run the following command to start the container in each node (You should download the weight to /root/.cache in advance): :::::{tab-set} :sync-group: multi-node ::::{tab-item} A2 series :sync: A2 ```{code-block} bash :substitutions: # Update the vllm-ascend image # openEuler: # export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-openeuler # Ubuntu: # export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version| export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version| # Run the container using the defined variables # Note if you are running bridge network with docker, Please expose available ports # for multiple nodes communication in advance docker run --rm \ --name vllm-ascend \ --net=host \ --shm-size=1g \ --device /dev/davinci0 \ --device /dev/davinci1 \ --device /dev/davinci2 \ --device /dev/davinci3 \ --device /dev/davinci4 \ --device /dev/davinci5 \ --device /dev/davinci6 \ --device /dev/davinci7 \ --device /dev/davinci_manager \ --device /dev/devmm_svm \ --device /dev/hisi_hdc \ -v /usr/local/dcmi:/usr/local/dcmi \ -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \ -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \ -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \ -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /root/.cache:/root/.cache \ -it $IMAGE bash ``` :::: ::::{tab-item} A3 series :sync: A3 ```{code-block} bash :substitutions: # Update the vllm-ascend image # openEuler: # export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-a3-openeuler # Ubuntu: # export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-a3 export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-a3 # Run the container using the defined variables # Note if you are running bridge network with docker, Please expose available ports # for multiple nodes communication in advance docker run --rm \ --name vllm-ascend \ --net=host \ --shm-size=1g \ --device /dev/davinci0 \ --device /dev/davinci1 \ --device /dev/davinci2 \ --device /dev/davinci3 \ --device /dev/davinci4 \ --device /dev/davinci5 \ --device /dev/davinci6 \ --device /dev/davinci7 \ --device /dev/davinci8 \ --device /dev/davinci9 \ --device /dev/davinci10 \ --device /dev/davinci11 \ --device /dev/davinci12 \ --device /dev/davinci13 \ --device /dev/davinci14 \ --device /dev/davinci15 \ --device /dev/davinci_manager \ --device /dev/devmm_svm \ --device /dev/hisi_hdc \ -v /usr/local/dcmi:/usr/local/dcmi \ -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \ -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \ -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \ -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /root/.cache:/root/.cache \ -it $IMAGE bash ``` :::: :::::