跳转至

Nightly CI 测试

This document explains how to trigger nightly hardware CI tests against your own PR code on Ascend NPU hardware (A2/A3), without waiting for the scheduled nightly run.

背景

By default, nightly CI tests run on a fixed schedule using pre-built nightly images. Contributors can self-service trigger these tests directly against their PR changes by combining a GitHub label with a comment command.

如何触发

1.发布评论

Post one of the following comments in the PR to specify which tests to run. The comment itself triggers the workflow — no label is required.

评论 效果
/nightly 运行**所有** nightly 测试
/nightly all 运行**所有** nightly 测试（与上同）
/nightly test1 test2 ... 仅运行**指定名称**的测试
/nightly <tests> --aop_enabled Run named tests with AOP bisect / classify enabled

Note

Only repository Contributors (Triage role) and Maintainers (Write role) can trigger the /nightly command. If you do not have this permission, ask a maintainer to post the comment for you. You can find the list of maintainers and contributors in the project's Governance page or by checking the CODEOWNERS file.

2.等待结果

GitHub Actions will trigger the Nightly-A2 or Nightly-A3 workflow. Only tests matching the filter will be dispatched, which saves hardware resources.

PR 触发与定时运行的区别

定时/手动触发 PR 触发
触发方式 Cron（每日）或 workflow_dispatch /nightly 评论
测试的代码 预构建的 nightly 镜像 您的 PR 的 HEAD 提交（从源码全新安装）
测试范围 所有测试 可通过 /nightly <names> 配置
vLLM + vllm-ascend 来自镜像 从源码检出并安装
测试矩阵 来自主分支的矩阵 YAML 来自 PR 分支的矩阵 YAML

当检测到 PR 运行时（is_pr_test: true），工作流还会执行以下额外步骤：

  1. 卸载容器中任何已有的 vllm 包。
  2. 从源码检出特定的 vllm 版本以及您 PR 中的 vllm-ascend 提交。
  3. 从源码安装所有依赖。
  4. 安装 aisbench 基准测试套件。

测试矩阵数据源

The set of nightly test cases (their names, runners, test paths, model configs) is declared in a single data file:

.github/workflows/configs/nightly_config.yaml

The file is organized as a2: and a3: top-level keys (one per SoC). Under each SoC, tests are grouped by execution shape (single-node, multi-node, double-node, multi-card, accuracy) and each group holds a test_config (or nightly / pr_only for accuracy) list whose entries carry a name plus the fields consumed by the downstream reusable workflows (os, tests, config_file_path, size, etc.).

Both the Nightly-A2 and Nightly-A3 workflows dynamically read this file at run time — there is no hardcoded test matrix in the workflow YAMLs. The /nightly <name> slash command resolves names by walking the same file from the PR branch, so newly added entries can be exercised on a PR before they land on main.

添加新的 Nightly 测试用例

要添加新的测试用例（无需修改工作流 YAML）：

  1. Append an entry under the appropriate section in .github/workflows/configs/nightly_config.yaml. Each entry needs at least:
  2. name：在 /nightly <name> 筛选器中使用的唯一标识符
  3. os（用于单节点/多卡 pytest+yaml 测试）或自动推断的 runner
  4. tests:（pytest 目录）或 config_file_path:（YAML 驱动的模型配置）之一
  5. size（仅多节点/双节点）
  6. Add the actual test files (pytest modules under tests/e2e/nightly/... or YAML model configs in tests/e2e/nightly/.../configs/).
  7. Open a PR. Once CI is green, you can validate the new entry against real NPU hardware without merging the PR — see 示例 below.

可用的测试名称

The test names you can pass to /nightly correspond to the name fields under the matching section in .github/workflows/configs/nightly_config.yaml. The tables below mirror the current contents of that file.

A2 工作流（.github/workflows/schedule_nightly_test_a2.yaml）

单节点测试（a2.single_node.test_config）：

测试名称 描述
test_custom_op_multi_card 自定义算子测试（多卡）
qwen3-vl-32b-instruct-w8a8 Qwen3-VL-32B-Instruct W8A8
qwen3-32b-int8 Qwen3-32B INT8 量化
Qwen3.5-27B-w8a8-A2 Qwen3.5-27B W8A8
Qwen3.5-397B-A17B-w4a8-mtp Qwen3.5-397B-A17B W4A8 + MTP

多节点测试（a2.multi_node.test_config）：

测试名称 描述
multi-node-qwen3-235b-dp Qwen3-235B-A22B，2 节点 DP
multi-node-GLM-5.1-w8a8-A2 GLM-5.1 W8A8，2 节点
multi-node-Kimi-K2.5-W4A8-A2 Kimi-K2.5 W4A8，2 节点

精度测试（a2.accuracy.nightly 和 a2.accuracy.pr_only）：

测试名称 描述 测试范围
accuracy-group-1 Qwen3-VL-8B、Qwen3-8B、Qwen2-Audio-7B 等 nightly
accuracy-group-2 ERNIE-4.5、Molmo-7B、Llama-3.2-3B 等 nightly
accuracy-group-3 Qwen3-30B-A3B、Qwen3-VL-30B-A3B 等 nightly
accuracy-group-4 Qwen3-Next-80B-A3B、Qwen3-Omni-30B-A3B 等 nightly
pr-accuracy-group-1 gemma-3-4b-it、internlm3-8b-instruct 等 pr_only
pr-accuracy-group-2 Qwen2.5-Math-RM-72B、Hunyuan-A13B-Instruct pr_only

The pr-accuracy-group-* entries only run on /nightly (PR 触发) runs; /nightly all on the schedule skips them.

A3 工作流（.github/workflows/schedule_nightly_test_a3.yaml）

多节点测试（a3.multi_node.test_config，4 节点）：

测试名称 描述
multi-node-deepseek-v3.2-W8A8-EP DeepSeek-V3.2-W8A8 带 EP，4 节点

双节点测试（a3.double_node.test_config，2 节点，在多节点测试之后运行）：

测试名称 描述
multi-node-deepseek-r1-w8a8-longseq DeepSeek-R1-W8A8 长序列，2 节点
multi-node-qwen3-dp Qwen3-235B-A22B，2 节点 DP
multi-node-qwenw8a8-2node-eplb Qwen3-235B-W8A8 带 EPLB，2 节点
multi-node-dpsk3.2-2node DeepSeek-V3.2-W8A8，2 节点
multi-node-qwenw8a8-2node-longseq Qwen3-235B-W8A8 长序列，2 节点
multi-node-qwen-disagg-pd Qwen3-235B 分离式 PD，2 节点
multi-node-qwen-vl-disagg-pd Qwen3-VL-235B 分离式 PD，2 节点
multi-node-deepseek-v3.1 DeepSeek-V3.1-BF16，2 节点
multi-node-deepseek-v3.2-W8A8-EP DeepSeek-V3.2-W8A8 带 EP，4 节点
multi-node-glm-5.2 GLM-5.1-W8A8，2 节点

单节点测试（a3.single_node.test_config）：

测试名称 描述
mtpx-deepseek-r1-0528-w8a8 MTP-X + DeepSeek-R1-0528-W8A8
deepseek-r1-0528-w8a8 DeepSeek-R1-0528-W8A8
kimi-k2-thinking Kimi-K2-Thinking
qwen3-vl-235b-a22b-instruct-w8a8 Qwen3-VL-235B-A22B-Instruct-W8A8
deepseek-r1-0528-w8a8-prefix-cache DeepSeek-R1-0528-W8A8 前缀缓存
deepseek-v3-2-w8a8 DeepSeek-V3.2-W8A8
glm-4.7-w8a8 GLM-4.7 W8A8
kimi-k2.5 Kimi-K2.5
qwen3-235b-a22b-w8a8 Qwen3-235B-A22B-W8A8
Qwen3.5-397B-A17B-w8a8-mtp Qwen3.5-397B-A17B W8A8 + MTP
MiniMax-M2.5-w8a8-QuaRot-A3 MiniMax-M2.5 W8A8 + QuaRot
Qwen3.5-27B-w8a8-A3 Qwen3.5-27B W8A8
Qwen3.5-122B-A10B-W8A8-A3 Qwen3.5-122B-A10B W8A8
DeepSeek-V4-Flash-W8A8-A3 DeepSeek-V4-Flash W8A8

多卡测试（a3.multi_card.test_config）：

测试名称 描述
qwen3-30b-acc Qwen3-30B 精度测试
qwen3-30b-a3b-w8a8 Qwen3-30B-A3B-W8A8
qwen3-32b-int8 Qwen3-32B-Int8
qwen3-32b-int8-prefix-cache Qwen3-32B-Int8 前缀缓存
Qwen3-30B-A3B-W4A8-llm-compressor Qwen3-30B-A3B W4A8 通过 llm-compressor
Qwen3-30B-QuaRot Qwen3-30B QuaRot + eagle3
Qwen3-32B-QuaRot Qwen3-32B QuaRot + eagle3 模型测试

Warning

The A3 resource pool has a maximum concurrency of 5×16 NPUs. Multi-node tests run with max-parallel: 2 to avoid resource exhaustion. Running /nightly all on A3 will queue a large number of jobs — prefer targeting specific test names when possible.

示例

针对您的 PR 运行所有可用的 nightly 测试：

/nightly

仅运行自定义算子多卡测试：

/nightly test_custom_op_multi_card

同时运行两个指定的测试（每个 SoC 一个）：

/nightly test_custom_op_multi_card mtpx-deepseek-r1-0528-w8a8

运行单个精度组（包含其所有模型）：

/nightly accuracy-group-1

运行单个精度模型（仅运行组中的该模型）：

/nightly accuracy-group-1/Qwen3-8B

Re-trigger after fixing an issue: just push a new commit. The synchronize event re-runs the workflow and picks up the existing /nightly 评论 automatically — no need to post a new comment.

AOP Hooks (Bisect)

Add --aop_enabled to any /nightly command to enable the AOP pipeline:

/nightly all --aop_enabled

When enabled, the workflow will:

  1. Capture the test result (pass / fail).
  2. Classify the failure as environmental (network, infra) or code-related.
  3. Bisect genuine code failures to pinpoint the offending commit.

This is useful for automated root-cause analysis of nightly regressions.

添加新测试用例——操作示例

要将 my-new-test 添加到 A2 单节点部分：

  1. Edit .github/workflows/configs/nightly_config.yaml, append under a2.single_node.test_config:
  - name: my-new-test
    os: linux-aarch64-a2b3-4
    tests: tests/e2e/nightly/single_node/ops/multicard_ops_a2/test_my_new.py

  1. Commit the new pytest file (test_my_new.py) in the same PR.

  2. Trigger from the PR:

/nightly my-new-test

The workflow will:

  • pr_nightly_command.yml reads your PR's nightly_config.yaml and resolves my-new-test → dispatch A2 only.
  • Nightly-A2 is dispatched at main, but generate-a2-matrix checks out your PR commit and reads the new entry from the matrix.
  • single-node-tests runs one matrix job for my-new-test, with should_run=true. The reusable workflow checks out your PR code (via vllm_ascend_ref) and runs your pytest.

Troubleshooting

The workflow didn't start after I posted the comment.

  • Check that the comment starts exactly with /nightly with no leading spaces or extra characters before the slash.
  • Confirm you have at least Triage permission on the repository; unauthorized users' comments are ignored.
  • To re-trigger after fixing an issue, simply push a new commit — the workflow will reuse the existing /nightly comment automatically.

Only some tests ran, not the ones I expected.

  • Test names are case-sensitive and must match the name field in .github/workflows/configs/nightly_config.yaml exactly (see the tables above).
  • For a PR-triggered run, the matrix is loaded from your PR's nightly_config.yaml, not main. If a name isn't in your PR's file, it won't be recognized and the dispatch will be skipped.
  • Check the parse-trigger job output in GitHub Actions for the resolved test_filter value.

The workflow ran with the scheduled image, not my PR code.

  • Confirm the workflow was triggered by repository_dispatch (slash command), not bare workflow_dispatch. The pr_nightly_command.yml workflow is what actually dispatches schedule_nightly_test_a2.yaml / _a3.yaml with vllm_ascend_ref pointing at your PR SHA.

A new test I added isn't being recognized.

  • Confirm the entry is well-formed YAML under .github/workflows/configs/nightly_config.yaml. The name field is required and must be unique within the SoC's section.
  • The matrix is loaded from your PR branch, so make sure the file is committed to the same branch the /nightly comment was posted on.

How to obtain more detailed logs to pinpoint problems for multi-node tests

  • For most issues, the stdout pop-up logs from GitHub actions are sufficient (this log always represents the logs from the first node).
  • If the logs from a first node are no longer sufficient to provide effective logging information, see the summary of your jobs to download log archive for the corresponding test, which includes the framework-side logs and plog information for each node, structured as follows:
.
├── node0
│   ├── root
│   │   └── ascend
│   │       └── log
│   └── var
│       └── log
│           └── vllm-deepseek-v3-0f233d-0_logs.txt
└── node1
    ├── root
    │   └── ascend
    │       └── log
    └── var
        └── log
            └── vllm-deepseek-v3-0f233d-0-1_logs.txt