Nightly CI Test¶

This document explains how to trigger nightly hardware CI tests against your own PR code on Ascend NPU hardware (A2/A3), without waiting for the scheduled nightly run.

Background¶

By default, nightly CI tests run on a fixed schedule using pre-built nightly images. Contributors can self-service trigger these tests directly against their PR changes by combining a GitHub label with a comment command.

How to Trigger¶

1. Post a comment¶

Post one of the following comments in the PR to specify which tests to run. The comment itself triggers the workflow — no label is required.

Comment	Effect
`/nightly`	Run all nightly tests
`/nightly all`	Run all nightly tests (same as above)
`/nightly test1 test2 ...`	Run only the named tests

Note

Only repository Contributors (Triage role) and Maintainers (Write role) can trigger the /nightly command. If you do not have this permission, ask a maintainer to post the comment for you. You can find the list of maintainers and contributors in the project's Governance page or by checking the CODEOWNERS file.

2. Wait for results¶

GitHub Actions will trigger the Nightly-A2 or Nightly-A3 workflow. Only tests matching the filter will be dispatched, which saves hardware resources.

Differences Between PR and Scheduled Runs¶

	Scheduled / Manual Dispatch	PR-triggered
Trigger	Cron (daily) or `workflow_dispatch`	`/nightly` comment
Code tested	Pre-built nightly image	Your PR's HEAD commit (source installed fresh)
Test scope	All tests	Configurable via `/nightly <names>`
vLLM + vllm-ascend	From image	Checked out and installed from source
Test matrix	From main branch's matrix YAML	From PR branch's matrix YAML

When a PR run is detected (is_pr_test: true), the workflow additionally:

Uninstalls any existing vllm packages in the container.
Checks out the specific vllm version and your PR's vllm-ascend commit from source.
Installs all dependencies from source.
Installs the aisbench benchmark suite.

Test Matrix Data Source¶

The set of nightly test cases (their names, runners, test paths, model configs) is declared in a single data file:

.github/workflows/configs/nightly_config.yaml

The file is organized as a2: and a3: top-level keys (one per SoC). Under each SoC, tests are grouped by execution shape (single-node, multi-node, double-node, multi-card, accuracy) and each group holds a test_config (or nightly / pr_only for accuracy) list whose entries carry a name plus the fields consumed by the downstream reusable workflows (os, tests, config_file_path, size, etc.).

Both the Nightly-A2 and Nightly-A3 workflows dynamically read this file at run time — there is no hardcoded test matrix in the workflow YAMLs. The /nightly <name> slash command resolves names by walking the same file from the PR branch, so newly added entries can be exercised on a PR before they land on main.

Adding a New Nightly Test Case¶

To add a new test case (no need to touch the workflow YAMLs):

Append an entry under the appropriate section in .github/workflows/configs/nightly_config.yaml. Each entry needs at least:
name: unique identifier used in /nightly <name> filters
os (for single-node / multi-card pytest+yaml tests) or runner is inferred
one of tests: (pytest directory) or config_file_path: (YAML-driven model config)
size (multi-node / double-node only)
Add the actual test files (pytest modules under tests/e2e/nightly/... or YAML model configs in tests/e2e/nightly/.../configs/).
Open a PR. Once CI is green, you can validate the new entry against real NPU hardware without merging the PR — see Examples below.

Available Test Names¶

The test names you can pass to /nightly correspond to the name fields under the matching section in .github/workflows/configs/nightly_config.yaml. The tables below mirror the current contents of that file.

A2 workflow (`.github/workflows/schedule_nightly_test_a2.yaml`)¶

Single-node tests (a2.single_node.test_config):

Test name	Description
`test_custom_op_multi_card`	Custom operator tests (multi card)
`qwen3-vl-32b-instruct-w8a8`	Qwen3-VL-32B-Instruct W8A8
`qwen3-32b-int8`	Qwen3-32B INT8 quantization
`Qwen3.5-27B-w8a8-A2`	Qwen3.5-27B W8A8
`Qwen3.5-397B-A17B-w4a8-mtp`	Qwen3.5-397B-A17B W4A8 + MTP

Multi-node tests (a2.multi_node.test_config):

Test name	Description
`multi-node-qwen3-235b-dp`	Qwen3-235B-A22B, 2-node DP
`multi-node-GLM-5.1-w8a8-A2`	GLM-5.1 W8A8, 2 nodes
`multi-node-Kimi-K2.5-W4A8-A2`	Kimi-K2.5 W4A8, 2 nodes

Accuracy tests (a2.accuracy.nightly and a2.accuracy.pr_only):

Test name	Description	Scope
`accuracy-group-1`	Qwen3-VL-8B, Qwen3-8B, Qwen2-Audio-7B, etc.	nightly
`accuracy-group-2`	ERNIE-4.5, Molmo-7B, Llama-3.2-3B, etc.	nightly
`accuracy-group-3`	Qwen3-30B-A3B, Qwen3-VL-30B-A3B, etc.	nightly
`accuracy-group-4`	Qwen3-Next-80B-A3B, Qwen3-Omni-30B-A3B, etc.	nightly
`pr-accuracy-group-1`	gemma-3-4b-it, internlm3-8b-instruct, etc.	pr_only
`pr-accuracy-group-2`	Qwen2.5-Math-RM-72B, Hunyuan-A13B-Instruct	pr_only

The pr-accuracy-group-* entries only run on /nightly (PR-triggered) runs; /nightly all on the schedule skips them.

A3 workflow (`.github/workflows/schedule_nightly_test_a3.yaml`)¶

Multi-node tests (a3.multi_node.test_config, 4-node):

Test name	Description
`multi-node-deepseek-v3.2-W8A8-EP`	DeepSeek-V3.2-W8A8 with EP, 4-node

Double-node tests (a3.double_node.test_config, 2-node, run after multi-node):

Test name	Description
`multi-node-deepseek-r1-w8a8-longseq`	DeepSeek-R1-W8A8 long sequence, 2-node
`multi-node-qwen3-dp`	Qwen3-235B-A22B, 2-node DP
`multi-node-qwenw8a8-2node-eplb`	Qwen3-235B-W8A8 with EPLB, 2-node
`multi-node-dpsk3.2-2node`	DeepSeek-V3.2-W8A8, 2-node
`multi-node-qwenw8a8-2node-longseq`	Qwen3-235B-W8A8 long sequence, 2-node
`multi-node-qwen-disagg-pd`	Qwen3-235B disaggregated PD, 2-node
`multi-node-qwen-vl-disagg-pd`	Qwen3-VL-235B disaggregated PD, 2-node
`multi-node-deepseek-v3.1`	DeepSeek-V3.1-BF16, 2-node
`multi-node-deepseek-v3.2-W8A8-EP`	DeepSeek-V3.2-W8A8 with EP, 4-node
`multi-node-glm-5.2`	GLM-5.1-W8A8, 2-node

Single-node tests (a3.single_node.test_config):

Test name	Description
`mtpx-deepseek-r1-0528-w8a8`	MTP-X + DeepSeek-R1-0528-W8A8
`deepseek-r1-0528-w8a8`	DeepSeek-R1-0528-W8A8
`kimi-k2-thinking`	Kimi-K2-Thinking
`qwen3-vl-235b-a22b-instruct-w8a8`	Qwen3-VL-235B-A22B-Instruct-W8A8
`deepseek-r1-0528-w8a8-prefix-cache`	DeepSeek-R1-0528-W8A8 prefix cache
`deepseek-v3-2-w8a8`	DeepSeek-V3.2-W8A8
`glm-4.7-w8a8`	GLM-4.7 W8A8
`kimi-k2.5`	Kimi-K2.5
`qwen3-235b-a22b-w8a8`	Qwen3-235B-A22B-W8A8
`Qwen3.5-397B-A17B-w8a8-mtp`	Qwen3.5-397B-A17B W8A8 + MTP
`MiniMax-M2.5-w8a8-QuaRot-A3`	MiniMax-M2.5 W8A8 + QuaRot
`Qwen3.5-27B-w8a8-A3`	Qwen3.5-27B W8A8
`Qwen3.5-122B-A10B-W8A8-A3`	Qwen3.5-122B-A10B W8A8
`DeepSeek-V4-Flash-W8A8-A3`	DeepSeek-V4-Flash W8A8

Multi-card tests (a3.multi_card.test_config):

Test name	Description
`qwen3-30b-acc`	Qwen3-30B accuracy test
`qwen3-30b-a3b-w8a8`	Qwen3-30B-A3B-W8A8
`qwen3-32b-int8`	Qwen3-32B-Int8
`qwen3-32b-int8-prefix-cache`	Qwen3-32B-Int8 prefix cache
`Qwen3-30B-A3B-W4A8-llm-compressor`	Qwen3-30B-A3B W4A8 via llm-compressor
`Qwen3-30B-QuaRot`	Qwen3-30B QuaRot + eagle3
`Qwen3-32B-QuaRot`	Qwen3-32B QuaRot + eagle3

Warning

The A3 resource pool has a maximum concurrency of 5×16 NPUs. Multi-node tests run with max-parallel: 2 to avoid resource exhaustion. Running /nightly all on A3 will queue a large number of jobs — prefer targeting specific test names when possible.

Examples¶

Run all available nightly tests against your PR:

/nightly

Run only the custom operator multi-card test:

/nightly test_custom_op_multi_card

Run two specific tests at once (one per SoC):

/nightly test_custom_op_multi_card mtpx-deepseek-r1-0528-w8a8

Run a single accuracy group (with all of its models):

/nightly accuracy-group-1

Run a single accuracy model (only that model from a group):

/nightly accuracy-group-1/Qwen3-8B

Re-trigger after fixing an issue: just push a new commit. The synchronize event re-runs the workflow and picks up the existing /nightly comment automatically — no need to post a new comment.

Adding a New Test Case — Worked Example¶

To add my-new-test to the A2 single-node section:

Edit .github/workflows/configs/nightly_config.yaml, append under a2.single_node.test_config:

  - name: my-new-test
    os: linux-aarch64-a2b3-4
    tests: tests/e2e/nightly/single_node/ops/multicard_ops_a2/test_my_new.py

Commit the new pytest file (test_my_new.py) in the same PR.
Trigger from the PR:

/nightly my-new-test

The workflow will:

pr_nightly_command.yml reads your PR's nightly_config.yaml and resolves my-new-test → dispatch A2 only.
Nightly-A2 is dispatched at main, but generate-a2-matrix checks out your PR commit and reads the new entry from the matrix.
single-node-tests runs one matrix job for my-new-test, with should_run=true. The reusable workflow checks out your PR code (via vllm_ascend_ref) and runs your pytest.

Troubleshooting¶

The workflow didn't start after I posted the comment.

Check that the comment starts exactly with /nightly with no leading spaces or extra characters before the slash.
Confirm you have at least Triage permission on the repository; unauthorized users' comments are ignored.
To re-trigger after fixing an issue, simply push a new commit — the workflow will reuse the existing /nightly comment automatically.

Only some tests ran, not the ones I expected.

Test names are case-sensitive and must match the name field in .github/workflows/configs/nightly_config.yaml exactly (see the tables above).
For a PR-triggered run, the matrix is loaded from your PR's nightly_config.yaml, not main. If a name isn't in your PR's file, it won't be recognized and the dispatch will be skipped.
Check the parse-trigger job output in GitHub Actions for the resolved test_filter value.

The workflow ran with the scheduled image, not my PR code.

Confirm the workflow was triggered by repository_dispatch (slash command), not bare workflow_dispatch. The pr_nightly_command.yml workflow is what actually dispatches schedule_nightly_test_a2.yaml / _a3.yaml with vllm_ascend_ref pointing at your PR SHA.

A new test I added isn't being recognized.

Confirm the entry is well-formed YAML under .github/workflows/configs/nightly_config.yaml. The name field is required and must be unique within the SoC's section.
The matrix is loaded from your PR branch, so make sure the file is committed to the same branch the /nightly comment was posted on.

How to obtain more detailed logs to pinpoint problems for multi-node tests

For most issues, the stdout pop-up logs from GitHub actions are sufficient (this log always represents the logs from the first node).
If the logs from a first node are no longer sufficient to provide effective logging information, see the summary of your jobs to download log archive for the corresponding test, which includes the framework-side logs and plog information for each node, structured as follows:

.
├── node0
│   ├── root
│   │   └── ascend
│   │       └── log
│   └── var
│       └── log
│           └── vllm-deepseek-v3-0f233d-0_logs.txt
└── node1
    ├── root
    │   └── ascend
    │       └── log
    └── var
        └── log
            └── vllm-deepseek-v3-0f233d-0-1_logs.txt