# Graph Mode Guide

```{note}
This feature is currently experimental. In future versions, there may be behavioral changes around configuration, coverage, performance improvement.
```

This guide provides instructions for using Ascend Graph Mode with vLLM Ascend. Please note that graph mode is only available on V1 Engine.

## Getting Started

In v0.11.0, vLLM Ascend will run models in graph mode by default to keep the same behavior with vLLM. If you hit any issues, please feel free to open an issue on GitHub and fallback to the eager mode temporarily by set `enforce_eager=True` when initializing the model.

There are two kinds for graph mode supported by vLLM Ascend:
- **ACLGraph**: This is the default graph mode supported by vLLM Ascend.
- **TorchAirGraph**: This is the GE graph mode. In v0.11.0, only DeepSeek series models are supported.

:::{note}
ACLGraph improves the performance of all the models by default, TorchAirGraph will be dropped in next version, and v0.11.0 will be the last stable version that support TorchAirGraph.
:::

## Using ACLGraph
ACLGraph is enabled by default. Take Qwen series models as an example, just set to use V1 Engine is enough.

Offline example:

```python
import os

from vllm import LLM

model = LLM(model="Qwen/Qwen2-7B-Instruct")
outputs = model.generate("Hello, how are you?")
```

Online example:

```shell
vllm serve Qwen/Qwen2-7B-Instruct
```

## Using TorchAirGraph

If you want to run DeepSeek series models with the graph mode, you should use [TorchAirGraph](https://www.hiascend.com/document/detail/zh/Pytorch/700/modthirdparty/torchairuseguide/torchair_0002.html). In this case, additional configuration is required.

Offline example:

```python
import os
from vllm import LLM

# TorchAirGraph is only work without chunked-prefill now
model = LLM(model="deepseek-ai/DeepSeek-R1-0528", additional_config={"torchair_graph_config": {"enabled": True},"ascend_scheduler_config": {"enabled": True,}})
outputs = model.generate("Hello, how are you?")
```

Online example:

```shell
vllm serve Qwen/Qwen2-7B-Instruct --additional-config='{"torchair_graph_config": {"enabled": true},"ascend_scheduler_config": {"enabled": true,}}'
```

You can find more details about additional configuration [here](../configuration/additional_config.md).

## Fallback to the Eager Mode

If both `ACLGraph` and `TorchAirGraph` fail to run, you should fallback to the eager mode.

Offline example:

```python
import os
from vllm import LLM

model = LLM(model="someother_model_weight", enforce_eager=True)
outputs = model.generate("Hello, how are you?")
```

Online example:

```shell
vllm serve Qwen/Qwen2-7B-Instruct --enforce-eager
```