Nightly CI Test

Nightly CI Test#

This document explains how to trigger nightly hardware CI tests against your own PR code on Ascend NPU hardware (A2/A3), without waiting for the scheduled nightly run.

Background#

By default, nightly CI tests run on a fixed schedule using pre-built nightly images. Contributors can self-service trigger these tests directly against their PR changes by combining a GitHub label with a comment command.

How to Trigger#

1. Post a comment#

Post one of the following comments in the PR to specify which tests to run. The comment itself triggers the workflow — no label is required.

Comment

Effect

/nightly

Run all nightly tests

/nightly all

Run all nightly tests (same as above)

/nightly test1 test2 ...

Run only the named tests

Note

Only repository Contributors (Triage role) and Maintainers (Write role) can trigger the /nightly command. If you do not have this permission, ask a maintainer to post the comment for you. You can find the list of maintainers and contributors in the project’s Governance page or by checking the CODEOWNERS file.

2. Wait for results#

GitHub Actions will trigger the Nightly-A2 or Nightly-A3 workflow. Only tests matching the filter will be dispatched, which saves hardware resources.

Differences Between PR and Scheduled Runs#

Scheduled / Manual Dispatch

PR-triggered

Trigger

Cron (daily) or workflow_dispatch

/nightly comment

Code tested

Pre-built nightly image

Your PR’s HEAD commit (source installed fresh)

Test scope

All tests

Configurable via /nightly <names>

vLLM + vllm-ascend

From image

Checked out and installed from source

Test matrix

From main branch’s matrix YAML

From PR branch’s matrix YAML

When a PR run is detected (is_pr_test: true), the workflow additionally:

  1. Uninstalls any existing vllm packages in the container.

  2. Checks out the specific vllm version and your PR’s vllm-ascend commit from source.

  3. Installs all dependencies from source.

  4. Installs the aisbench benchmark suite.

Test Matrix Data Source#

The set of nightly test cases (their names, runners, test paths, model configs) is declared in a single data file:

.github/workflows/configs/nightly_config.yaml

The file is organized as a2: and a3: top-level keys (one per SoC). Under each SoC, tests are grouped by execution shape (single-node, multi-node, double-node, multi-card, accuracy) and each group holds a test_config (or nightly / pr_only for accuracy) list whose entries carry a name plus the fields consumed by the downstream reusable workflows (os, tests, config_file_path, size, etc.).

Both the Nightly-A2 and Nightly-A3 workflows dynamically read this file at run time — there is no hardcoded test matrix in the workflow YAMLs. The /nightly <name> slash command resolves names by walking the same file from the PR branch, so newly added entries can be exercised on a PR before they land on main.

Adding a New Nightly Test Case#

To add a new test case (no need to touch the workflow YAMLs):

  1. Append an entry under the appropriate section in .github/workflows/configs/nightly_config.yaml. Each entry needs at least:

    • name: unique identifier used in /nightly <name> filters

    • os (for single-node / multi-card pytest+yaml tests) or runner is inferred

    • one of tests: (pytest directory) or config_file_path: (YAML-driven model config)

    • size (multi-node / double-node only)

  2. Add the actual test files (pytest modules under tests/e2e/nightly/... or YAML model configs in tests/e2e/nightly/.../configs/).

  3. Open a PR. Once CI is green, you can validate the new entry against real NPU hardware without merging the PR — see Examples below.

Available Test Names#

The test names you can pass to /nightly correspond to the name fields under the matching section in .github/workflows/configs/nightly_config.yaml. The tables below mirror the current contents of that file.

A2 workflow (.github/workflows/schedule_nightly_test_a2.yaml)#

Single-node tests (a2.single_node.test_config):

Test name

Description

test_custom_op_multi_card

Custom operator tests (multi card)

qwen3-vl-32b-instruct-w8a8

Qwen3-VL-32B-Instruct W8A8

qwen3-32b-int8

Qwen3-32B INT8 quantization

Qwen3.5-27B-w8a8-A2

Qwen3.5-27B W8A8

Qwen3.5-397B-A17B-w4a8-mtp

Qwen3.5-397B-A17B W4A8 + MTP

Multi-node tests (a2.multi_node.test_config):

Test name

Description

multi-node-qwen3-235b-dp

Qwen3-235B-A22B, 2-node DP

multi-node-GLM-5.1-w8a8-A2

GLM-5.1 W8A8, 2 nodes

multi-node-Kimi-K2.5-W4A8-A2

Kimi-K2.5 W4A8, 2 nodes

Accuracy tests (a2.accuracy.nightly and a2.accuracy.pr_only):

Test name

Description

Scope

accuracy-group-1

Qwen3-VL-8B, Qwen3-8B, Qwen2-Audio-7B, etc.

nightly

accuracy-group-2

ERNIE-4.5, Molmo-7B, Llama-3.2-3B, etc.

nightly

accuracy-group-3

Qwen3-30B-A3B, Qwen3-VL-30B-A3B, etc.

nightly

accuracy-group-4

Qwen3-Next-80B-A3B, Qwen3-Omni-30B-A3B, etc.

nightly

pr-accuracy-group-1

gemma-3-4b-it, internlm3-8b-instruct, etc.

pr_only

pr-accuracy-group-2

Qwen2.5-Math-RM-72B, Hunyuan-A13B-Instruct

pr_only

The pr-accuracy-group-* entries only run on /nightly (PR-triggered) runs; /nightly all on the schedule skips them.

A3 workflow (.github/workflows/schedule_nightly_test_a3.yaml)#

Multi-node tests (a3.multi_node.test_config, 4-node):

Test name

Description

multi-node-deepseek-v3.2-W8A8-EP

DeepSeek-V3.2-W8A8 with EP, 4-node

Double-node tests (a3.double_node.test_config, 2-node, run after multi-node):

Test name

Description

multi-node-deepseek-r1-w8a8-longseq

DeepSeek-R1-W8A8 long sequence, 2-node

multi-node-qwen3-dp

Qwen3-235B-A22B, 2-node DP

multi-node-qwenw8a8-2node-eplb

Qwen3-235B-W8A8 with EPLB, 2-node

multi-node-dpsk3.2-2node

DeepSeek-V3.2-W8A8, 2-node

multi-node-qwenw8a8-2node-longseq

Qwen3-235B-W8A8 long sequence, 2-node

multi-node-qwen-disagg-pd

Qwen3-235B disaggregated PD, 2-node

multi-node-qwen-vl-disagg-pd

Qwen3-VL-235B disaggregated PD, 2-node

multi-node-deepseek-v3.1

DeepSeek-V3.1-BF16, 2-node

multi-node-deepseek-v3.2-W8A8-EP

DeepSeek-V3.2-W8A8 with EP, 4-node

multi-node-glm-5.2

GLM-5.1-W8A8, 2-node

Single-node tests (a3.single_node.test_config):

Test name

Description

mtpx-deepseek-r1-0528-w8a8

MTP-X + DeepSeek-R1-0528-W8A8

deepseek-r1-0528-w8a8

DeepSeek-R1-0528-W8A8

kimi-k2-thinking

Kimi-K2-Thinking

qwen3-vl-235b-a22b-instruct-w8a8

Qwen3-VL-235B-A22B-Instruct-W8A8

deepseek-r1-0528-w8a8-prefix-cache

DeepSeek-R1-0528-W8A8 prefix cache

deepseek-v3-2-w8a8

DeepSeek-V3.2-W8A8

glm-4.7-w8a8

GLM-4.7 W8A8

kimi-k2.5

Kimi-K2.5

qwen3-235b-a22b-w8a8

Qwen3-235B-A22B-W8A8

Qwen3.5-397B-A17B-w8a8-mtp

Qwen3.5-397B-A17B W8A8 + MTP

MiniMax-M2.5-w8a8-QuaRot-A3

MiniMax-M2.5 W8A8 + QuaRot

Qwen3.5-27B-w8a8-A3

Qwen3.5-27B W8A8

Qwen3.5-122B-A10B-W8A8-A3

Qwen3.5-122B-A10B W8A8

DeepSeek-V4-Flash-W8A8-A3

DeepSeek-V4-Flash W8A8

Multi-card tests (a3.multi_card.test_config):

Test name

Description

qwen3-30b-acc

Qwen3-30B accuracy test

qwen3-30b-a3b-w8a8

Qwen3-30B-A3B-W8A8

qwen3-32b-int8

Qwen3-32B-Int8

qwen3-32b-int8-prefix-cache

Qwen3-32B-Int8 prefix cache

Qwen3-30B-A3B-W4A8-llm-compressor

Qwen3-30B-A3B W4A8 via llm-compressor

Qwen3-30B-QuaRot

Qwen3-30B QuaRot + eagle3

Qwen3-32B-QuaRot

Qwen3-32B QuaRot + eagle3

Warning

The A3 resource pool has a maximum concurrency of 5×16 NPUs. Multi-node tests run with max-parallel: 2 to avoid resource exhaustion. Running /nightly all on A3 will queue a large number of jobs — prefer targeting specific test names when possible.

Examples#

Run all available nightly tests against your PR:

/nightly

Run only the custom operator multi-card test:

/nightly test_custom_op_multi_card

Run two specific tests at once (one per SoC):

/nightly test_custom_op_multi_card mtpx-deepseek-r1-0528-w8a8

Run a single accuracy group (with all of its models):

/nightly accuracy-group-1

Run a single accuracy model (only that model from a group):

/nightly accuracy-group-1/Qwen3-8B

Re-trigger after fixing an issue: just push a new commit. The synchronize event re-runs the workflow and picks up the existing /nightly comment automatically — no need to post a new comment.

Adding a New Test Case — Worked Example#

To add my-new-test to the A2 single-node section:

  1. Edit .github/workflows/configs/nightly_config.yaml, append under a2.single_node.test_config:

      - name: my-new-test
    os: linux-aarch64-a2b3-4
    tests: tests/e2e/nightly/single_node/ops/multicard_ops_a2/test_my_new.py

  2. Commit the new pytest file (test_my_new.py) in the same PR.

  3. Trigger from the PR:

    /nightly my-new-test

The workflow will:

  • pr_nightly_command.yml reads your PR’s nightly_config.yaml and resolves my-new-test → dispatch A2 only.

  • Nightly-A2 is dispatched at main, but generate-a2-matrix checks out your PR commit and reads the new entry from the matrix.

  • single-node-tests runs one matrix job for my-new-test, with should_run=true. The reusable workflow checks out your PR code (via vllm_ascend_ref) and runs your pytest.

Troubleshooting#

The workflow didn’t start after I posted the comment.

  • Check that the comment starts exactly with /nightly with no leading spaces or extra characters before the slash.

  • Confirm you have at least Triage permission on the repository; unauthorized users’ comments are ignored.

  • To re-trigger after fixing an issue, simply push a new commit — the workflow will reuse the existing /nightly comment automatically.

Only some tests ran, not the ones I expected.

  • Test names are case-sensitive and must match the name field in .github/workflows/configs/nightly_config.yaml exactly (see the tables above).

  • For a PR-triggered run, the matrix is loaded from your PR’s nightly_config.yaml, not main. If a name isn’t in your PR’s file, it won’t be recognized and the dispatch will be skipped.

  • Check the parse-trigger job output in GitHub Actions for the resolved test_filter value.

The workflow ran with the scheduled image, not my PR code.

  • Confirm the workflow was triggered by repository_dispatch (slash command), not bare workflow_dispatch. The pr_nightly_command.yml workflow is what actually dispatches schedule_nightly_test_a2.yaml / _a3.yaml with vllm_ascend_ref pointing at your PR SHA.

A new test I added isn’t being recognized.

  • Confirm the entry is well-formed YAML under .github/workflows/configs/nightly_config.yaml. The name field is required and must be unique within the SoC’s section.

  • The matrix is loaded from your PR branch, so make sure the file is committed to the same branch the /nightly comment was posted on.

How to obtain more detailed logs to pinpoint problems for multi-node tests

  • For most issues, the stdout pop-up logs from GitHub actions are sufficient (this log always represents the logs from the first node).

  • If the logs from a first node are no longer sufficient to provide effective logging information, see the summary of your jobs to download log archive for the corresponding test, which includes the framework-side logs and plog information for each node, structured as follows:

    .
├── node0
│   ├── root
│   │   └── ascend
│   │       └── log
│   └── var
│       └── log
│           └── vllm-deepseek-v3-0f233d-0_logs.txt
└── node1
    ├── root
    │   └── ascend
    │       └── log
    └── var
        └── log
            └── vllm-deepseek-v3-0f233d-0-1_logs.txt