# Atlas 300I DUO / Atlas 200I Pro

This tutorial covers vLLM deployment on Ascend 310P inference hardware, including
Atlas 300I DUO and Atlas 200I Pro acceleration modules.

```{note}
Atlas 300I DUO does not support `triton` or `triton-ascend`.
```

## Run vLLM on Atlas 300I DUO

### Install Notes

If installing from source, `vllm` and `vllm-ascend` may automatically pull in `triton` and `triton-ascend` dependencies, which may cause unexpected issues on Atlas 300I DUO. Please uninstall them before running on Atlas 300I DUO:

```bash
pip uninstall -y triton-ascend triton
```

### Graph Mode Notes

```{warning}
The current release supports `FULL_DECODE_ONLY` graph mode on Atlas 300I DUO devices, but the following limitations apply due to hardware event-id resource constraints:

- When multiple Tensor Parallel (TP) ranks are enabled, the number of capturable graphs is limited and depends on the model depth. For example, Qwen3-32B can capture and replay 2 graphs.
- There is no such limitation when TP=1.
- We have reached out to the relevant experts for a solution. A software-based fix is considered feasible, but full support will take additional time. Thank you for your understanding.

```

### Deployment

Choose the startup command according to your hardware form factor.

#### Atlas 300I DUO

Run docker container:

```{code-block} bash
   :substitutions:

# Use the vllm-ascend image
export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-310p

docker run --rm \
--name vllm-ascend \
--shm-size=10g \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
--device /dev/davinci3 \
--device /dev/davinci4 \
--device /dev/davinci5 \
--device /dev/davinci6 \
--device /dev/davinci7 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-p 8080:8080 \
-it $IMAGE bash
```

#### Atlas 200I Pro Acceleration Module

When starting a container on Atlas 200I Pro, mount additional driver libraries and
configuration files required by `npu-smi`. Without these mounts, `npu-smi` commands
may fail inside the container. Use the command below for your container OS.

```{note}
Atlas 200I Pro also uses Ascend 310P. Adjust `--device=/dev/davinci0` according to
the NPU ID you want to use. You can query available devices with
`ll /dev/ | grep davinci`.
```

:::::{tab-set}

::::{tab-item} Ubuntu 24.04
:selected:

```{code-block} bash
   :substitutions:

export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-310p

docker run --rm \
--privileged \
--name vllm-ascend \
--shm-size=10g \
--device=/dev/davinci0:/dev/davinci0 \
--device=/dev/davinci_manager \
--device=/dev/ascend_manager \
--device=/dev/user_config \
-v /etc/sys_version.conf:/etc/sys_version.conf \
-v /etc/ld.so.conf.d/mind_so.conf:/etc/ld.so.conf.d/mind_so.conf \
-v /etc/hdcBasic.cfg:/etc/hdcBasic.cfg \
-v /var/dmp_daemon:/var/dmp_daemon \
-v /usr/lib64/libmmpa.so:/usr/lib64/libmmpa.so \
-v /usr/lib64/libcrypto.so.1.1:/usr/lib64/libcrypto.so.1.1 \
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
-v /usr/lib64/libstackcore.so:/usr/lib64/libstackcore.so \
-v /usr/lib/aarch64-linux-gnu/libyaml-0.so.2:/usr/lib64/libyaml-0.so.2 \
-v /etc/slog.conf:/etc/slog.conf \
-v /var/slogd:/var/slogd \
-v /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64 \
-v /usr/lib64/libtensorflow.so:/usr/lib64/libtensorflow.so \
-v /root/.cache:/root/.cache \
-p 8080:8080 \
-it $IMAGE bash
```

::::

::::{tab-item} openEuler 24.03

```{code-block} bash
   :substitutions:

export IMAGE=quay.io/ascend/vllm-ascend:|vllm_ascend_version|-310p-openeuler

docker run --rm \
--privileged \
--name vllm-ascend \
--shm-size=10g \
--device=/dev/davinci0:/dev/davinci0 \
--device=/dev/davinci_manager \
--device=/dev/ascend_manager \
--device=/dev/user_config \
-v /etc/sys_version.conf:/etc/sys_version.conf \
-v /etc/ld.so.conf.d/mind_so.conf:/etc/ld.so.conf.d/mind_so.conf \
-v /etc/hdcBasic.cfg:/etc/hdcBasic.cfg \
-v /var/dmp_daemon:/var/dmp_daemon \
-v /usr/lib64/libsemanage.so.2:/usr/lib64/libsemanage.so.2 \
-v /usr/lib64/libmmpa.so:/usr/lib64/libmmpa.so \
-v /usr/lib64/libcrypto.so.1.1:/usr/lib64/libcrypto.so.1.1 \
-v /usr/lib64/libyaml-0.so.2.0.9:/usr/lib64/libyaml-0.so.2 \
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
-v /usr/lib64/libstackcore.so:/usr/lib64/libstackcore.so \
-v /etc/slog.conf:/etc/slog.conf \
-v /var/slogd:/var/slogd \
-v /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64 \
-v /usr/lib64/libtensorflow.so:/usr/lib64/libtensorflow.so \
-v /root/.cache:/root/.cache \
-p 8080:8080 \
-it $IMAGE bash
```

::::
:::::

#### Host Page Cache Cleanup (Recommended)

````{tip}
On Atlas 200I Pro, if host memory is tight or you previously ran inference
workloads, consider running the following commands on the **host** (requires
`root`) before `vllm serve` to release reclaimable kernel page caches. This
may help free host memory and reduce OOM risk during model loading. The
command does not delete user files.

Whether to run this cleanup depends on your environment. Check available
host memory (for example, with `free -h`) and decide based on your actual
situation. Skip it if memory is already sufficient.

```bash
sync
echo 3 > /proc/sys/vm/drop_caches
```
````

Set up environment variables:

```bash
export VLLM_USE_MODELSCOPE=True
```

### Online Inference on NPU

```{warning}
For Atlas 300I DUO (310P), do not rely on `max-model-len` auto detection
(that is, do not omit the `--max-model-len` argument), because it may cause OOM.

Reason, based on the current 310P attention path:

- `AscendAttentionMetadataBuilder310` passes `model_config.max_model_len`
  to `AttentionMaskBuilder310`.
- `AttentionMaskBuilder310` builds a full causal mask with shape
  `[max_model_len, max_model_len]` in float16, then converts it to FRACTAL_NZ.
- In the 310P `attention_v1` prefill/chunked-prefill path
  (`_npu_flash_attention` / `_npu_paged_attention_splitfuse`),
  this explicit mask tensor is consumed directly, and there is currently
  no compressed-mask path.

If auto detection resolves to a large context length, the mask allocation
(`O(max_model_len^2)`) may exceed NPU memory and trigger OOM.
Always set an explicit and conservative value, for example `--max-model-len 16384`.
```

Run the following commands to start the vLLM server on NPU for the Qwen3 Dense series.

#### Prepare Model Weights

Use the W8A8SC quantized weights from the Eco-Tech official ModelScope repository.

```{list-table}
:header-rows: 1

* - Model
  - ModelScope Link
* - Qwen3-8B-W8A8SC-310
  - [Eco-Tech/Qwen3-8B-w8a8sc-310-vllm](https://www.modelscope.cn/models/Eco-Tech/Qwen3-8B-w8a8sc-310-vllm)
* - Qwen3-14B-W8A8SC-310
  - [Eco-Tech/Qwen3-14B-w8a8sc-310-vllm](https://www.modelscope.cn/models/Eco-Tech/Qwen3-14B-w8a8sc-310-vllm)
* - Qwen3-32B-W8A8SC-310
  - [Eco-Tech/Qwen3-32B-w8a8sc-310-vllm](https://www.modelscope.cn/models/Eco-Tech/Qwen3-32B-w8a8sc-310-vllm)
```

:::::{tab-set}

::::{tab-item} Qwen3-8B-W8A8SC
:selected:

```{code-block} bash
   :substitutions:

vllm serve Eco-Tech/Qwen3-8B-w8a8sc-310-vllm/TP1/Qwen3-8B-w8a8sc-310-vllm-tp1 \
    --host 127.0.0.1 \
    --port 8080 \
    --tensor-parallel-size 1 \
    --gpu_memory_utilization 0.90 \
    --max_num_seqs 32 \
    --served_model_name qwen \
    --dtype float16 \
    --additional-config '{"ascend_compilation_config": {"fuse_norm_quant": false}}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [1,2,4,8,16,32]}' \
    --quantization ascend \
    --max_model_len 16384 \
    --no-enable-prefix-caching \
    --load_format sharded_state
```

::::

::::{tab-item} Qwen3-14B-W8A8SC

```{code-block} bash
   :substitutions:

vllm serve Eco-Tech/Qwen3-14B-w8a8sc-310-vllm/TP1/Qwen3-14B-w8a8sc-310-vllm-tp1 \
    --host 127.0.0.1 \
    --port 8080 \
    --tensor-parallel-size 1 \
    --gpu_memory_utilization 0.90 \
    --max_num_seqs 16 \
    --served_model_name qwen \
    --dtype float16 \
    --additional-config '{"ascend_compilation_config": {"fuse_norm_quant": false}}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [1,2,4,8,16]}' \
    --quantization ascend \
    --max_model_len 16384 \
    --no-enable-prefix-caching \
    --load_format sharded_state
```

::::

::::{tab-item} Qwen3-32B-W8A8SC

```{code-block} bash
   :substitutions:

export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3

vllm serve Eco-Tech/Qwen3-32B-w8a8sc-310-vllm/TP4/Qwen3-32B-w8a8sc-310-vllm-tp4 \
    --host 127.0.0.1 \
    --port 8080 \
    --tensor-parallel-size 4 \
    --gpu_memory_utilization 0.90 \
    --max_num_seqs 32 \
    --served_model_name qwen \
    --dtype float16 \
    --additional-config '{"ascend_compilation_config": {"fuse_norm_quant": false}}' \
    --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes": [16,32]}' \
    --quantization ascend \
    --max_model_len 20480 \
    --no-enable-prefix-caching \
    --load_format sharded_state
```

::::
:::::

Once the server is started, you can query the model with input prompts:

```bash
curl http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "The future of AI is",
    "max_completion_tokens": 64,
    "temperature": 0.0
  }'
```

If the script runs successfully, you can see the generated result.

### Offline Inference

Run the following script, `example.py`, to execute offline inference on NPU.

:::::{tab-set}

::::{tab-item} Qwen3-8B-W8A8SC
:selected:

```{code-block} python
   :substitutions:

import gc
import torch

from vllm import LLM, SamplingParams
from vllm.distributed.parallel_state import (
    destroy_distributed_environment,
    destroy_model_parallel,
)


def clean_up():
    destroy_model_parallel()
    destroy_distributed_environment()
    gc.collect()
    torch.npu.empty_cache()


prompts = [
    "Hello, my name is",
    "The future of AI is",
]

sampling_params = SamplingParams(
    max_completion_tokens=100,
    temperature=0.0,
)

llm = LLM(
    model="Eco-Tech/Qwen3-8B-w8a8sc-310-vllm/TP1/Qwen3-8B-w8a8sc-310-vllm-tp1",
    tensor_parallel_size=1,
    max_model_len=16384,
    dtype="float16",
    quantization="ascend",
    load_format="sharded_state",
    additional_config={
        "ascend_compilation_config": {
            "fuse_norm_quant": False,
        }
    },
    compilation_config={
        "cudagraph_mode": "FULL_DECODE_ONLY",
        "cudagraph_capture_sizes": [1, 2, 4, 8, 16, 32],
    },
    enable_prefix_caching=False,
)

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

del llm
clean_up()
```

::::

::::{tab-item} Qwen3-14B-W8A8SC

```{code-block} python
   :substitutions:

import gc
import torch

from vllm import LLM, SamplingParams
from vllm.distributed.parallel_state import (
    destroy_distributed_environment,
    destroy_model_parallel,
)


def clean_up():
    destroy_model_parallel()
    destroy_distributed_environment()
    gc.collect()
    torch.npu.empty_cache()


prompts = [
    "Hello, my name is",
    "The future of AI is",
]

sampling_params = SamplingParams(
    max_completion_tokens=100,
    temperature=0.0,
)

llm = LLM(
    model="Eco-Tech/Qwen3-14B-w8a8sc-310-vllm/TP1/Qwen3-14B-w8a8sc-310-vllm-tp1",
    tensor_parallel_size=1,
    max_model_len=16384,
    dtype="float16",
    quantization="ascend",
    load_format="sharded_state",
    additional_config={
        "ascend_compilation_config": {
            "fuse_norm_quant": False,
        }
    },
    compilation_config={
        "cudagraph_mode": "FULL_DECODE_ONLY",
        "cudagraph_capture_sizes": [1, 2, 4, 8, 16],
    },
    enable_prefix_caching=False,
)

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

del llm
clean_up()
```

::::

::::{tab-item} Qwen3-32B-W8A8SC

```{code-block} python
   :substitutions:

import gc
import os
import torch

from vllm import LLM, SamplingParams
from vllm.distributed.parallel_state import (
    destroy_distributed_environment,
    destroy_model_parallel,
)


def clean_up():
    destroy_model_parallel()
    destroy_distributed_environment()
    gc.collect()
    torch.npu.empty_cache()


os.environ["ASCEND_RT_VISIBLE_DEVICES"] = "0,1,2,3"

prompts = [
    "Hello, my name is",
    "The future of AI is",
]

sampling_params = SamplingParams(
    max_completion_tokens=100,
    temperature=0.0,
)

llm = LLM(
    model="Eco-Tech/Qwen3-32B-w8a8sc-310-vllm/TP4/Qwen3-32B-w8a8sc-310-vllm-tp4",
    tensor_parallel_size=4,
    max_model_len=20480,
    dtype="float16",
    quantization="ascend",
    load_format="sharded_state",
    additional_config={
        "ascend_compilation_config": {
            "fuse_norm_quant": False,
        }
    },
    compilation_config={
        "cudagraph_mode": "FULL_DECODE_ONLY",
        "cudagraph_capture_sizes": [16, 32],
    },
    enable_prefix_caching=False,
)

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

del llm
clean_up()
```

::::
:::::

Run script:

```bash
python example.py
```

If the script runs successfully, you can see the generated result.

### Closing Notes

For early access to Qwen3-MoE, Qwen3-VL, and preview support for Qwen3.5 and Qwen3.6 with performance acceleration, follow #7394 for updated deployment guidance.