CI Settings¶

This document describes where Buildkite YAML lives in the repo, how each platform organizes CI, and how to add a new job. It does not document agent queues, GPU types, or container plugin details—those belong in infra / preset files (for example .buildkite/common/ci_mirror_hardwares.yml).

For CI levels (L1–L5) and triggers, see Test System Overview. For test authoring, see Test Writing Guide. For running tests locally or replaying CI jobs, see Test Execution Guide.

Directory layout¶

Canonical layout (prefer these paths for new changes):

.buildkite/
├── common/                          # Shared across platforms
│   ├── scripts/
│   │   ├── skip_ci.py               # skip-ci decision (docs / skip-mark / CI YAML paths)
│   │   ├── upload_pipeline.py       # Bootstrap + test-pipeline uploader (CUDA/NPU)
│   │   └── resolve_skip_ci.sh       # Shell helpers for AMD/Intel bootstrap
│   └── ci_mirror_hardwares.yml      # CUDA uploader presets (referenced by name only)
├── cuda/                            # Primary NVIDIA CUDA CI
│   ├── pipeline.yml                 # Bootstrap entry (hook upload)
│   ├── bootstrap-upload-steps.yml   # Bootstrap child steps (upload_pipeline --upload)
│   ├── test-ready.yml               # L2
│   ├── test-merge.yml               # L3
│   ├── test-nightly.yml             # L4
│   ├── test-weekly.yml              # L5
│   └── rebase-pipeline.yml
├── npu/
│   ├── pipeline-npu.yml             # Bootstrap entry (hook upload)
│   ├── bootstrap-upload-steps.yml   # Bootstrap child steps (upload_pipeline --upload)
│   ├── pipeline-npu-a3.yml          # A3 variant (when used)
│   ├── test-npu-ready.yml           # L2
│   ├── test-npu-nightly.yml         # L4
│   └── scripts/
├── amd/
│   ├── test-amd-ready.yml           # L2 job definitions (template input)
│   ├── test-amd-merge.yml           # L3 job definitions
│   ├── test-template-amd-omni.j2    # Renders final pipeline.yaml
│   └── scripts/
│       ├── bootstrap-amd-omni.sh    # Entry: skip-ci → Jinja → upload
│       └── run-amd-test.sh          # Wraps pytest inside ROCm docker
├── intel/
│   ├── pipeline-intel.yml           # Static Intel XPU pipeline
│   └── scripts/
│       ├── bootstrap-intel-omni.sh
│       └── run-xpu-test.sh
└── release/
    ├── release-pipeline.yml
    └── scripts/

Placement rules¶

There are still legacy copies at .buildkite/*.yaml (without the cuda/ prefix). Treat .buildkite/cuda/* as source of truth.

Platform comparison¶

Platform configuration style¶

python3 .buildkite/common/scripts/upload_pipeline.py .buildkite/cuda/test-ready.yml

python3 .buildkite/common/scripts/upload_pipeline.py .buildkite/npu/test-npu-ready.yml

Cross-cutting conventions¶

Diff-aware CI¶

PR diffs drive two independent skip layers. Both read changed files from git, but at different pipeline stages and with different granularity:

Changed files (both layers):

Label triggers (ready, merge-test) are unchanged—diff-aware logic only reduces what runs after a pipeline is already scheduled.

Bootstrap skip¶

skip_ci.py classifies the git diff into buckets (docs / qualifying skip-mark / whitelisted L2·L3·L4/L5 YAML / other), then picks one decision path. CUDA/NPU apply that decision by injecting Buildkite if on bootstrap-upload-steps.yml step keys; AMD/Intel call gate_bootstrap_ci <platform> <l2|l3>.

Decision overview¶

skip_all exceptions (CUDA/NPU bootstrap only): PR labels (nightly-test, merge-test, weekly-test, ready, …) do not revive jobs under skip_all. On main, scheduled NIGHTLY=1 still builds the image and uploads L4; WEEKLY=1 or NON_CRITICAL=1 still builds the image and uploads L5 (CUDA). WEEKLY=1 on main also uploads L2/L3 with --e2e.

Yaml-gated nightly/weekly: L4/L5 upload steps keep their normal label / NIGHTLY / WEEKLY / NON_CRITICAL conditions (for example PR nightly-test still uploads nightly). Only L2/L3 upload steps are matrix-gated.

Whitelisted CI YAML → platform¶

Register new files in L2_YAML_FILES, L3_YAML_FILES, or L45_YAML_FILES in skip_ci.py.

Yaml-gated category branches¶

L2? / L3? / L45? = whether any file in that whitelist changed. Untouched platforms/levels end up off.

Yaml-gated examples (platform-level)¶

Docs/skip-mark mixed with any row below follows the same matrix.

How platforms consume the decision¶

Unit coverage: tests/buildkite/test_skip_ci.py.

Step filtering¶

CUDA L2 (.buildkite/cuda/test-ready.yml) and L3 (.buildkite/cuda/test-merge.yml) only. Bootstrap upload entry: upload_pipeline.py --upload .buildkite/cuda/bootstrap-upload-steps.yml.

Uploader-only keys — removed before Buildkite sees the YAML; never used at runtime on agents:

Policy¶

• Always uploaded (no key): groups outside E2E Test—Simple Test, Diffusion unit tests, Engine/Model Executor, Distributed, Custom Pipeline, Entrypoints (L2), LoRA / Entrypoints (L3).
• Diff-gated: every E2E Test leaf job. List the smallest prefix set per step—pytest file(s), model code under vllm_omni/model_executor/models/ or vllm_omni/diffusion/models/, plus stage_input_processors/ and vllm_omni/deploy/*.yaml when applicable. A group may define the key instead; the whole group drops if no prefix matches.

YAML examples¶

      - label: "Diffusion · Qwen Image Test"
        source_file_dependencies:
          - tests/e2e/online_serving/test_qwen_image.py
          - vllm_omni/diffusion/models/qwen_image/
        commands:
          - pytest -s -v tests/e2e/online_serving/test_qwen_image.py -m 'core_model' ...
        mirror_hardwares: h100_1

      - label: "TTS · Qwen3-TTS CustomVoice Test"
        source_file_dependencies:
          - tests/e2e/online_serving/test_qwen3_tts_customvoice.py
          - vllm_omni/model_executor/models/qwen3_tts/
          - vllm_omni/model_executor/stage_input_processors/qwen3_tts.py
          - vllm_omni/deploy/qwen3_tts.yaml
        commands:
          - pytest -s -v tests/e2e/online_serving/test_qwen3_tts_customvoice.py ...
        mirror_hardwares: l4_4

Local dry-run¶

python3 .buildkite/common/scripts/upload_pipeline.py .buildkite/cuda/test-ready.yml
python3 .buildkite/common/scripts/upload_pipeline.py .buildkite/cuda/test-merge.yml | grep source_file_dependencies
# (no output expected)

On PR builds, upload_pipeline.py logs skip '…' (no changes under …) for omitted steps.

Labels and grouping¶

• Use groups for dashboard readability; keep E2E Test as the group name CUDA diff filtering expects for --e2e weekly runs on main (WEEKLY=1).
• Prefix labels by model domain: Omni ·, TTS ·, Diffusion ·, Simple ·, etc., matching existing steps.

Per-model coverage¶

Pilot (v1): one Merge-tier (L3) job in .buildkite/cuda/test-merge.yml uploads per-model, per-entry-mode coverage as Buildkite artifacts — "TTS · Qwen3-TTS Base Test". Coverage + artifact upload run only when bootstrap uploaded that job with --e2e (WEEKLY=1 and BUILDKITE_BRANCH=main, via run_cov_split.sh); otherwise the same job runs plain pytest without --cov. It runs on a single GPU so the pilot is cheap to reproduce.

L1 package coverage (CPU): ready/merge Simple Test jobs run without --cov. Scheduled weekly on main with WEEKLY=1 runs Simple · CPU Coverage Test in .buildkite/cuda/test-weekly.yml (pytest -m 'core_model and cpu' + Cobertura XML artifact).

Naming convention: coverage-<model_id>-<mode>-<step_id>.xml.gz, where <model_id> is the model's directory name under vllm_omni/diffusion/models/<model_id>/ or vllm_omni/model_executor/models/<model_id>/ (e.g. z_image, qwen3_omni), <mode> is online or offline, and <step_id> is BUILDKITE_STEP_ID (local outside Buildkite). gunzip before feeding a report to a Cobertura consumer.

Reports are gzipped because --cov=vllm_omni sets coverage's source, which makes it walk the package and emit one <line> element per statement for every file — including the ones a single model's tests never import. That inventory, not the coverage, is the size: a Bagel report is 7.1 MB, of which 96.5% is <line> elements and 77% belongs to the 840 of 1169 files that run never imported. A run covering zero lines produces the same 7 MB. gzip takes it to roughly 410 KB and changes nothing about the data.

<step_id> is what keeps two steps that cover the same model apart. Artifacts are scoped to the build, and a weekly build can carry ready/merge (with --e2e) and weekly tiers at once, so several same-model steps land in one namespace — qwen3_tts already has both a Base and a CustomVoice job, and minicpmo_4_5 a base and a duplex one. Two uploads on one path do not fail loudly: an exact-name download can report the path as ambiguous, and a glob download fetches every match concurrently and renames each onto the same destination, so whichever finishes last wins. The step id remains the logical step identifier across retries, while Buildkite's default artifact lookup selects only the latest attempt — BUILDKITE_JOB_ID would instead give every attempt its own filename, leaving a failed attempt's partial report to be picked up alongside the real one. A step using parallelism/matrix would need BUILDKITE_PARALLEL_JOB added, since its jobs share a step id; neither pilot is one.

Opting in a new model: replace the job's combined pytest command with run_cov_split.sh, which runs each mode, names the reports, uploads them, and fails the step if either half or the upload failed:

commands:
  - |
    .buildkite/common/scripts/run_cov_split.sh \
      --model-id <model_id> \
      --offline <offline test path> \
      --online <online test path> \
      -- <the job's existing pytest flags, e.g. -m '...' --run-level '...'>

Everything after -- is forwarded verbatim to both runs with quoting preserved, so an expression like -m 'advanced_model and cuda' survives intact and a job can pass anything else it needs. --offline/--online are repeatable and only one is required, so a job with tests in a single mode runs just that half. Runtime is bounded solely by the step's timeout_in_minutes — raise it, since the split loads the model once per mode.

Before splitting, confirm each half actually collects at least one test under the job's forwarded filters — pytest exits with code 5 ("no tests collected") if a half is empty, which the script reports as a failure, red-ing a job that used to pass as one combined invocation.

Also check the tests' num_cards against the job's mirror_hardwares preset. _cuda_marks turns num_cards > 1 into skipif(device_count() < num_cards), so a test asking for more GPUs than the preset provides is silently skipped and its coverage artifact comes back empty. Splitting such a job produces a file that looks fine and measures nothing — "Diffusion · Z Image Test" is the current example (online tests want 4 cards, the job runs on l4_1), which is why it is not a coverage pilot.

Online-mode coverage depends on the [tool.coverage.run] patch/sigterm settings in pyproject.toml: online tests launch the server with subprocess.Popen and stop it with SIGTERM, and without those settings the XML reflects only the pytest parent process, not the server's code paths.

The upload depends on buildkite-agent being callable inside the container. The kubernetes presets (h100_*, *_npu_*) provide it; the docker ones (l4_*) only do because they set mount-buildkite-agent: true. A new docker preset that runs a coverage job needs the same.

List both run_cov_split.sh and pyproject.toml in every opted-in job's source_file_dependencies — both change what the job measures, so without them a change there is filtered out of normal PR builds and only surfaces in a later nightly. tests/buildkite/test_upload_pipeline.py covers the filter behavior with a synthetic job (it does not pin real merge labels). Editing only the surrounding CI YAML still does not schedule them, so a PR that touches just the wiring needs a full E2E run (or the commands run on a GPU host) to produce artifacts. When checking a new model's artifacts, compare lines-covered between the online and offline XML rather than just confirming both files exist.

Validation checklist¶

Related documentation¶

• Test System Overview — L1–L5 scope, triggers, test pyramid
• Test Writing Guide — markers, directories, L1–L5 examples
• Test Execution Guide — running CI-aligned jobs locally
• Implementation: .buildkite/common/scripts/upload_pipeline.py, .buildkite/common/scripts/skip_ci.py