Skip to content

vllm_omni.worker.runner_assisted_metadata ¶

RunnerAssistedAttentionMetadataProvider ¶

Bases: Protocol

Optional model hook for runner-built full attention metadata.

get_runner_assisted_full_attention_metadata_request ¶

get_runner_assisted_full_attention_metadata_request(
    *,
    req_ids: Sequence[str],
    num_reqs: int,
    num_scheduled_tokens: Sequence[int],
    num_computed_tokens: Sequence[int],
    max_num_scheduled_tokens: int,
) -> RunnerAssistedFullAttentionMetadataRequest | None

set_runner_assisted_full_attention_metadata_context ¶

set_runner_assisted_full_attention_metadata_context(
    *, enabled: bool, num_reqs: int = 0
) -> None

RunnerAssistedFullAttentionMetadataRequest ¶

Bases: NamedTuple

for_cudagraph_capture `instance-attribute` ¶

for_cudagraph_capture: bool

num_reqs_padded `instance-attribute` ¶

num_reqs_padded: int