# Qwen3.6-35B-A3B ## Introduction Qwen3.6-35B-A3B is a sparse MoE model in the Qwen3.6 family. It has 35B total parameters with about 3B activated per token, and uses the same hybrid attention architecture (GDN + full attention) as Qwen3.5. Deployment on vLLM Ascend follows the same pattern as other Qwen3.5-style MoE models. This document covers supported features, environment setup, single-node deployment, and accuracy/performance evaluation. The `Qwen3.6-35B-A3B` model is first supported in `vllm-ascend:v0.18.0rc1`. ## Supported Features Refer to [supported features](../../user_guide/support_matrix/supported_models.md) to get the model's supported feature matrix. Refer to [feature guide](../../user_guide/feature_guide/index.md) to get the feature's configuration. ## Environment Preparation ### Model Weight - `Qwen3.6-35B-A3B`(BF16 version): requires 1 Atlas 800 A3 (64G × 16) node or 1 Atlas 800 A2 (64G × 8) node. [Download model weight](https://modelscope.cn/models/Qwen/Qwen3.6-35B-A3B) - `Qwen3.6-35B-A3B-w8a8`(Quantized version): requires 1 Atlas 800 A3 (64G × 16) node or 1 Atlas 800 A2 (64G × 8) node. [Download model weight](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-35B-A3B-w8a8) It is recommended to download the model weight to `/root/.cache/`. ### Installation :::::{tab-set} ::::{tab-item} Use docker image For example, using images `quay.io/ascend/vllm-ascend:v0.18.0rc1`(for Atlas 800 A2) and `quay.io/ascend/vllm-ascend:v0.18.0rc1-a3`(for Atlas 800 A3). Select an image based on your machine type and start the docker image on your node, refer to [using docker](../../installation.md#set-up-using-docker). ```{code-block} bash :substitutions: # Update --device according to your device (Atlas A2: /dev/davinci[0-7] Atlas A3:/dev/davinci[0-15]). # Update the vllm-ascend image according to your environment. # Note you should download the weight to /root/.cache in advance. # Update the vllm-ascend image export IMAGE=m.daocloud.io/quay.io/ascend/vllm-ascend:|vllm_ascend_version| export NAME=vllm-ascend # Run the container using the defined variables docker run --rm \ --name $NAME \ --net=host \ --shm-size=1g \ --device /dev/davinci0 \ --device /dev/davinci1 \ --device /dev/davinci2 \ --device /dev/davinci3 \ --device /dev/davinci4 \ --device /dev/davinci5 \ --device /dev/davinci6 \ --device /dev/davinci7 \ --device /dev/davinci_manager \ --device /dev/devmm_svm \ --device /dev/hisi_hdc \ -v /usr/local/dcmi:/usr/local/dcmi \ -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \ -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \ -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \ -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -it $IMAGE bash ``` :::: ::::{tab-item} Build from source You can build all from source. - Install `vllm-ascend`, refer to [set up using python](../../installation.md#set-up-using-python). :::: ::::: ## Deployment ### Single-node Deployment `Qwen3.6-35B-A3B-w8a8` can be deployed on 1 Atlas 800 A3 (64G × 16) or 1 Atlas 800 A2 (64G × 8). Start with `--quantization ascend` and `--enable-expert-parallel`. Run the following script to execute online inference with up to 262144 context length on 1 Atlas 800 A3 (64G × 16). ```shell #!/bin/sh # Load model from ModelScope to speed up download export VLLM_USE_MODELSCOPE=True # To reduce memory fragmentation and avoid out of memory export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export HCCL_OP_EXPANSION_MODE="AIV" export HCCL_BUFFSIZE=1024 export OMP_NUM_THREADS=1 export TASK_QUEUE_ENABLE=1 echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor sysctl -w vm.swappiness=0 sysctl -w kernel.numa_balancing=0 sysctl kernel.sched_migration_cost_ns=50000 export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2:$LD_PRELOAD export VLLM_ASCEND_ENABLE_FLASHCOMM1=1 vllm serve Eco-Tech/Qwen3.6-35B-A3B-w8a8 \ --host 0.0.0.0 \ --port 8000 \ --data-parallel-size 1 \ --tensor-parallel-size 2 \ --enable-expert-parallel \ --seed 1024 \ --quantization ascend \ --served-model-name qwen3.6 \ --max-num-seqs 128 \ --max-model-len 262144 \ --max-num-batched-tokens 16384 \ --trust-remote-code \ --gpu-memory-utilization 0.90 \ --enable-prefix-caching \ --speculative_config '{"method": "qwen3_5_mtp", "num_speculative_tokens": 3, "enforce_eager": true}' \ --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \ --additional-config '{"enable_cpu_binding":true, "multistream_overlap_shared_expert": true}' \ --async-scheduling ``` **Notice:** The parameters are explained as follows: - `--data-parallel-size` 1 and `--tensor-parallel-size` 2 are common settings for data parallelism (DP) and tensor parallelism (TP) sizes. - `--enable-expert-parallel` indicates that EP is enabled. Note that vLLM does not support a mixed approach of ETP and EP; that is, MoE can either use pure EP or pure TP. - `--max-model-len` represents the context length, which is the maximum value of the input plus output for a single request. - `--max-num-seqs` indicates the maximum number of requests that each DP group is allowed to process. If the number of requests sent to the service exceeds this limit, the excess requests will remain in a waiting state and will not be scheduled. Note that the time spent in the waiting state is also counted in metrics such as TTFT and TPOT. Therefore, when testing performance, it is generally recommended that `--max-num-seqs` * `--data-parallel-size` >= the actual total concurrency. - `--max-num-batched-tokens` represents the maximum number of tokens that the model can process in a single step. Currently, vLLM v1 scheduling enables ChunkPrefill/SplitFuse by default, which means: - (1) If the input length of a request is greater than `--max-num-batched-tokens`, it will be divided into multiple rounds of computation according to `--max-num-batched-tokens`; - (2) Decode requests are prioritized for scheduling, and prefill requests are scheduled only if there is available capacity. - Generally, if `--max-num-batched-tokens` is set to a larger value, the overall latency will be lower, but the pressure on GPU memory (activation value usage) will be greater. - `--gpu-memory-utilization` represents the proportion of HBM that vLLM will use for actual inference. Its essential function is to calculate the available kv_cache size. During the warm-up phase (referred to as profile run in vLLM), vLLM records the peak GPU memory usage during an inference process with an input size of `--max-num-batched-tokens`. The available kv_cache size is then calculated as: `--gpu-memory-utilization` * HBM size - peak GPU memory usage. Therefore, the larger the value of `--gpu-memory-utilization`, the more kv_cache can be used. However, since the GPU memory usage during the warm-up phase may differ from that during actual inference (e.g., due to uneven EP load), setting `--gpu-memory-utilization` too high may lead to OOM (Out of Memory) issues during actual inference. The default value is `0.9`. - `--enable-prefix-caching` can be enabled for Qwen3.6 models. To disable it, use ``--no-enable-prefix-caching``. Note that for models with large context lengths, prefix caching may require significant memory. - `--quantization` "ascend" indicates that quantization is used. To disable quantization, remove this option. - `--speculative_config` uses `qwen3_5_mtp` because Qwen3.6 shares the same MTP head design as Qwen3.5. - `--additional-config` `"multistream_overlap_shared_expert": true` is recommended for MoE models to overlap shared-expert computation. - `--compilation-config` contains configurations related to the aclgraph graph mode. The most significant configurations are "cudagraph_mode" and "cudagraph_capture_sizes", which have the following meanings: "cudagraph_mode": represents the specific graph mode. Currently, "PIECEWISE" and "FULL_DECODE_ONLY" are supported. The graph mode is mainly used to reduce the cost of operator dispatch. Currently, "FULL_DECODE_ONLY" is recommended. - "cudagraph_capture_sizes": represents different levels of graph modes. The default value is [1, 2, 4, 8, 16, 24, 32, 40,..., `--max-num-seqs`]. In the graph mode, the input for graphs at different levels is fixed, and inputs between levels are automatically padded to the next level. Currently, the default setting is recommended. Only in some scenarios is it necessary to set this separately to achieve optimal performance. ## Functional Verification Once your server is started, you can query the model with input prompts: ```shell curl http://localhost:8000/v1/completions \ -H "Content-Type: application/json" \ -d '{ "model": "qwen3.6", "prompt": "The future of AI is", "max_completion_tokens": 50, "temperature": 0 }' ``` ## Accuracy Evaluation ### Using AISBench 1. Refer to [Using AISBench](../../developer_guide/evaluation/using_ais_bench.md) for details. 2. After execution, you can get the accuracy result of `Qwen3.6-35B-A3B-w8a8` for reference. ## Performance ### Using AISBench Refer to [Using AISBench for performance evaluation](../../developer_guide/evaluation/using_ais_bench.md#execute-performance-evaluation) for details. ### Using vLLM Benchmark Run performance evaluation of `Qwen3.6-35B-A3B-w8a8` as an example. Refer to [vllm benchmark](https://docs.vllm.ai/en/latest/contributing/benchmarks.html) for more details. There are three `vllm bench` subcommands: - `latency`: Benchmark the latency of a single batch of requests. - `serve`: Benchmark the online serving throughput. - `throughput`: Benchmark offline inference throughput. Take the `serve` as an example. Run the code as follows. ```shell export VLLM_USE_MODELSCOPE=True vllm bench serve --model Eco-Tech/Qwen3.6-35B-A3B-w8a8 --dataset-name random --random-input 200 --num-prompts 200 --request-rate 1 --save-result --result-dir ./ ``` After about several minutes, you can get the performance evaluation result.