图模式指南

图模式指南#

备注

此功能目前为实验性功能。在未来的版本中，配置、覆盖率和性能改进等方面的行为可能会有变化。

This guide provides instructions for using Ascend Graph Mode with vLLM Ascend. Please note that graph mode is only available on V1 Engine. And only Qwen, DeepSeek series models are well tested from 0.9.0rc1. We will make it stable and generalized in the next release.

快速入门#

From v0.9.1rc1 with V1 Engine, vLLM Ascend will run models in graph mode by default to keep the same behavior with vLLM. If you hit any issues, please feel free to open an issue on GitHub and fallback to the eager mode temporarily by set enforce_eager=True when initializing the model.

vLLM Ascend 支持两种图模式：

  • ACLGraph：这是 vLLM Ascend 支持的默认图模式。在 v0.9.1rc1 版本中，只有 Qwen 系列模型得到了充分测试。

  • TorchAirGraph：这是GE图模式。在v0.9.1rc1版本中，仅支持DeepSeek系列模型。

使用 ACLGraph#

ACLGraph 默认启用。以 Qwen 系列模型为例，只需设置为使用 V1 引擎即可。

Offline example:

import os

from vllm import LLM

model = LLM(model="Qwen/Qwen2-7B-Instruct")
outputs = model.generate("Hello, how are you?")

Online example:

vllm serve Qwen/Qwen2-7B-Instruct

使用 TorchAirGraph#

If you want to run DeepSeek series models with the graph mode, you should use TorchAirGraph. In this case, additional configuration is required.

Offline example:

import os
from vllm import LLM

# TorchAirGraph is only work without chunked-prefill now
model = LLM(model="deepseek-ai/DeepSeek-R1-0528", additional_config={"torchair_graph_config": {"enabled": True},"ascend_scheduler_config": {"enabled": True,}})
outputs = model.generate("Hello, how are you?")

Online example:

vllm serve Qwen/Qwen2-7B-Instruct --additional-config='{"torchair_graph_config": {"enabled": true},"ascend_scheduler_config": {"enabled": true,}}'

You can find more details about additional configuration here.

Fallback to the Eager Mode#

If both ACLGraph and TorchAirGraph fail to run, you should fallback to the eager mode.

Offline example:

import os
from vllm import LLM

model = LLM(model="someother_model_weight", enforce_eager=True)
outputs = model.generate("Hello, how are you?")

Online example:

vllm serve Qwen/Qwen2-7B-Instruct --enforce-eager
目录