Markers for Tests¶

By adding markers before test functions, tests can later be executed uniformly by simply declaring the corresponding marker type.

Current Markers¶

Defined in pyproject.toml:

* Means those markers are auto-added by @hardware_test (parametrization decorator) or hardware_marks (only returning the list of marks for flexibility).

Example usage for markers¶

from tests.helpers.mark import hardware_test

@pytest.mark.core_model
@pytest.mark.omni
@hardware_test(
   res={"cuda": "L4", "rocm": "MI325", "npu": "A2"},
   num_cards=2,
)
@pytest.mark.parametrize("omni_server", test_params, indirect=True)
def test_video_to_audio()
    ...

Decorator: @hardware_test¶

This decorator is intended to make hardware-aware, cross-platform test authoring easier and more robust for CI/CD environments. The hardware_test decorator in vllm-omni/tests/helpers/mark.py performs the following actions:

1. Applies platform and resource markers
   Adds the appropriate pytest markers for each specified hardware platform (e.g., cuda, rocm, xpu, npu) and resource type (e.g., L4, H100, MI325, B60, A2, A3).

   @pytest.mark.cuda
   @pytest.mark.L4
   

2. Handles multi-card (distributed) scenarios
   For tests requiring multiple cards, it automatically adds distributed markers such as distributed_cuda, distributed_rocm, or distributed_npu.

   @pytest.mark.distributed_cuda(num_cards=num_cards)
   

3. Supports flexible card requirements
   Accepts num_cards as either a single integer for all platforms or as a dictionary with per-platform values. If not specified, defaults to 1 card per platform.

4. Integrates resource validation
   On CUDA, adds a skip marker (skipif_cuda) if the system does not have the required number of devices. Support for skipif_rocm and skipif_npu will be implemented later.

5. Works with pytest filtering
   Allows tests to be filtered and selected at runtime using standard pytest marker expressions (e.g., -m "distributed_cuda and L4").

Example usage for decorator¶

• Single call for multiple platforms:

  @hardware_test(
      res={"cuda": "L4", "rocm": "MI325", "xpu": "B60", "npu": "A2"},
      num_cards={"cuda": 2, "rocm": 2, "xpu": 2, "npu": 2},
  )
  

  or

  @hardware_test(
      res={"cuda": "L4", "rocm": "MI325", "xpu": "B60", "npu": "A2"},
      num_cards=2,
  )
  

• res must be a dict; supported resources: CUDA (L4/H100), ROCm (MI325), NPU (A2/A3)
• num_cards can be int (all platforms) or dict (per platform); defaults to 1 when missing
• Distributed markers (distributed_cuda, distributed_rocm, distributed_npu) are auto-added for multi-card cases
• Filtering examples:
  • CUDA only: pytest -m "distributed_cuda and L4"
  • ROCm only: pytest -m "distributed_rocm and MI325"
  • NPU only: pytest -m "distributed_npu"

Function: hardware_marks¶

hardware_marks returns a list of pytest mark objects with the same signature as @hardware_test. Use it when you need more flexibility, such as attaching hardware marks to individual pytest.param entries rather than an entire test function.

from tests.helpers.mark import hardware_marks

MULTI_CARD_MARKS = hardware_marks(
    res={"cuda": "H100", "rocm": "MI325", "npu": "A2"}, num_cards=2
)

@pytest.mark.parametrize("omni_server", [
    pytest.param(OmniServerParams(...), id="case_001", marks=MULTI_CARD_MARKS),
], indirect=True)
def test_feature(omni_server):
    ...

Add Support for a New Platform¶

If you want to add support for a new platform (e.g., "tpu" for a new accelerator), follow these steps:

1. Extend the marker list in your pytest config so that platform/resource markers are defined:

   # In pyproject.toml or pytest.ini
   [tool.pytest.ini_options]
   markers = [
       # ... existing markers ...
       "tpu: Tests that require TPU device",
       "TPU_V3: Tests that require TPU v3 hardware",
       "distributed_tpu: Tests that require multiple TPU devices",
   ]
   

2. Implement a marker construction function for your platform in vllm-omni/tests/helpers/mark.py:

   # In vllm-omni/tests/helpers/mark.py
   
   def tpu_marks(*, res: str, num_cards: int):
       test_platform = pytest.mark.tpu
       if res == "TPU_V3":
           test_resource = pytest.mark.TPU_V3
       else:
           raise ValueError(
               f"Invalid TPU resource type: {res}. Supported: TPU_V3")
   
       if num_cards == 1:
           return [test_platform, test_resource]
       else:
           test_distributed = pytest.mark.distributed_tpu(num_cards=num_cards)
           # Optionally: add skipif_tpu when implemented
           return [test_platform, test_resource, test_distributed]
   

3. Update hardware_marks to recognize your new platform: In the relevant place (see the hardware_marks implementation), add:

   if platform == "tpu":
       marks = tpu_marks(res=resource, num_cards=cards)
   

   (hardware_test calls hardware_marks internally, so both will pick up the change.)
4. (Recommended) Add a test using your new markers:

   @hardware_test(
       res={"tpu": "TPU_V3"},
       num_cards=2,
   )
   def test_my_tpu_feature():
       ...
   

Summary:
- Add pytest markers for your new platform/resources
- Implement a marker function (xxx_marks)
- Plug into hardware_marks
- You're done: tests using @hardware_test or hardware_marks with your platform now automatically get the correct markers, distribution, and isolation!

See code in vllm-omni/tests/helpers/mark.py for existing examples (cuda_marks, rocm_marks, npu_marks).