CI Settings¶

This document describes where Buildkite YAML lives in the repo, how each platform organizes CI, and how to add a new job. It does not document agent queues, GPU types, or container plugin details—those belong in infra / preset files (for example .buildkite/common/ci_mirror_hardwares.yml).

For CI levels (L1–L5) and triggers, see Test System Overview. For test authoring, see Test Writing Guide. For running tests locally or replaying CI jobs, see Test Execution Guide.

Directory layout¶

Canonical layout (prefer these paths for new changes):

.buildkite/
├── common/                          # Shared across platforms
│   ├── scripts/
│   │   ├── skip_ci.py               # skip-ci decision (docs / skip-mark / CI YAML paths)
│   │   ├── upload_pipeline.py       # Bootstrap + test-pipeline uploader (CUDA/NPU)
│   │   └── resolve_skip_ci.sh       # Shell helpers for AMD/Intel bootstrap
│   └── ci_mirror_hardwares.yml      # CUDA uploader presets (referenced by name only)
├── cuda/                            # Primary NVIDIA CUDA CI
│   ├── pipeline.yml                 # Bootstrap entry (hook upload)
│   ├── bootstrap-upload-steps.yml   # Bootstrap child steps (upload_pipeline --upload)
│   ├── test-ready.yml               # L2
│   ├── test-merge.yml               # L3
│   ├── test-nightly.yml             # L4
│   ├── test-weekly.yml              # L5
│   └── rebase-pipeline.yml
├── npu/
│   ├── pipeline-npu.yml             # Bootstrap entry (hook upload)
│   ├── bootstrap-upload-steps.yml   # Bootstrap child steps (upload_pipeline --upload)
│   ├── pipeline-npu-a3.yml          # A3 variant (when used)
│   ├── test-npu-ready.yml           # L2
│   ├── test-npu-nightly.yml         # L4
│   └── scripts/
├── amd/
│   ├── test-amd-ready.yml           # L2 job definitions (template input)
│   ├── test-amd-merge.yml           # L3 job definitions
│   ├── test-template-amd-omni.j2    # Renders final pipeline.yaml
│   └── scripts/
│       ├── bootstrap-amd-omni.sh    # Entry: skip-ci → Jinja → upload
│       └── run-amd-test.sh          # Wraps pytest inside ROCm docker
├── intel/
│   ├── pipeline-intel.yml           # Static Intel XPU pipeline
│   └── scripts/
│       ├── bootstrap-intel-omni.sh
│       └── run-xpu-test.sh
└── release/
    ├── release-pipeline.yml
    └── scripts/

Placement rules

There are still legacy copies at .buildkite/*.yaml (without the cuda/ prefix). Treat .buildkite/cuda/* as source of truth.

Platform comparison¶

Platform configuration style¶

python3 .buildkite/common/scripts/upload_pipeline.py .buildkite/cuda/test-ready.yml

python3 .buildkite/common/scripts/upload_pipeline.py .buildkite/npu/test-npu-ready.yml

Cross-cutting conventions¶

Diff-aware CI¶

PR diffs drive two independent skip layers. Both read changed files from git, but at different pipeline stages and with different granularity:

Changed files (both layers):

Label triggers (ready, merge-test) are unchanged—diff-aware logic only reduces what runs after a pipeline is already scheduled.

Bootstrap skip¶

skip_ci.py classifies the git diff into buckets (docs / qualifying skip-mark / whitelisted L2·L3·L4/L5 YAML / other), then picks one decision path. CUDA/NPU apply that decision by injecting Buildkite if on bootstrap-upload-steps.yml step keys; AMD/Intel call gate_bootstrap_ci <platform> <l2|l3>.

Decision overview¶

skip_all exceptions (CUDA/NPU bootstrap only): PR labels (nightly-test, merge-test, npu-test, weekly-test, …) do not revive jobs. On main, scheduled NIGHTLY=1 still builds the image and uploads L4 plus L2/L3 (with --e2e); WEEKLY=1 still builds the image and uploads L5 (CUDA).

Yaml-gated nightly/weekly: L4/L5 upload steps keep their normal label / NIGHTLY / WEEKLY conditions (for example PR nightly-test still uploads nightly). Only L2/L3 upload steps are matrix-gated.

Whitelisted CI YAML → platform¶

Register new files in L2_YAML_FILES, L3_YAML_FILES, or L45_YAML_FILES in skip_ci.py.

Yaml-gated category branches¶

L2? / L3? / L45? = whether any file in that whitelist changed. Untouched platforms/levels end up off.

Yaml-gated examples (platform-level)¶

Docs/skip-mark mixed with any row below follows the same matrix.

How platforms consume the decision¶

Unit coverage: tests/buildkite/test_skip_ci.py.

Step filtering¶

CUDA L2 (.buildkite/cuda/test-ready.yml) and L3 (.buildkite/cuda/test-merge.yml) only. Bootstrap upload entry: upload_pipeline.py --upload .buildkite/cuda/bootstrap-upload-steps.yml.

Uploader-only keys — removed before Buildkite sees the YAML; never used at runtime on agents:

Policy

• Always uploaded (no key): groups outside E2E Test—Simple Test, Diffusion unit tests, Engine/Model Executor, Distributed, Custom Pipeline, Entrypoints (L2), LoRA / Entrypoints (L3).
• Diff-gated: every E2E Test leaf job. List the smallest prefix set per step—pytest file(s), model code under vllm_omni/model_executor/models/ or vllm_omni/diffusion/models/, plus stage_input_processors/ and vllm_omni/deploy/*.yaml when applicable. A group may define the key instead; the whole group drops if no prefix matches.

YAML examples

      - label: "Diffusion · Qwen Image Test"
        source_file_dependencies:
          - tests/e2e/online_serving/test_qwen_image.py
          - vllm_omni/diffusion/models/qwen_image/
        commands:
          - pytest -s -v tests/e2e/online_serving/test_qwen_image.py -m 'core_model' ...
        mirror_hardwares: h100_1

      - label: "TTS · Qwen3-TTS CustomVoice Test"
        source_file_dependencies:
          - tests/e2e/online_serving/test_qwen3_tts_customvoice.py
          - vllm_omni/model_executor/models/qwen3_tts/
          - vllm_omni/model_executor/stage_input_processors/qwen3_tts.py
          - vllm_omni/deploy/qwen3_tts.yaml
        commands:
          - pytest -s -v tests/e2e/online_serving/test_qwen3_tts_customvoice.py ...
        mirror_hardwares: l4_4

Local dry-run

python3 .buildkite/common/scripts/upload_pipeline.py .buildkite/cuda/test-ready.yml
python3 .buildkite/common/scripts/upload_pipeline.py .buildkite/cuda/test-merge.yml | grep source_file_dependencies
# (no output expected)

On PR builds, upload_pipeline.py logs skip '…' (no changes under …) for omitted steps.

Labels and grouping¶

• Use groups for dashboard readability; keep E2E Test as the group name CUDA diff filtering expects for --e2e nightly runs on main.
• Prefix labels by model domain: Omni ·, TTS ·, Diffusion ·, Simple ·, etc., matching existing steps.

Validation checklist¶

Related documentation¶

• Test System Overview — L1–L5 scope, triggers, test pyramid
• Test Writing Guide — markers, directories, L1–L5 examples
• Test Execution Guide — running CI-aligned jobs locally
• Implementation: .buildkite/common/scripts/upload_pipeline.py, .buildkite/common/scripts/skip_ci.py