Skip to content

vllm.models.inkling2.nvidia.ops.attention_sm90

Inkling2 relative-bias paged attention for Hopper (Triton).

Stands in for tml_fa4 (prefill) and the tcgen05 decode/verify kernel of ops/attention.py, reading the same sheared relative logits of attention_prologue: column c of token t holds key c + cdiv(past + tile_end, 128) * 128 - (E + 256), where tile_end ends the token's 128-row query tile (see qr_clip_rel_tokens).

A program owns BLOCK_M rows (tokens x GQA group) of one request and KV head, and one KV split: split s of S takes key blocks s, s + S, ... of the rows' visible range, so a split count that depends only on the batch holds for every replay of a CUDA graph.

Functions:

  • rel_attention –

    Launch _rel_attention_kernel over the requests of cu_seqlens_q.

rel_attention(q, key_cache, value_cache, out, lse, rel_logits, gate=None, sinks=None, *, block_table, cache_seqlens, cu_seqlens_q, max_query_len, softmax_scale, window, num_splits, mode)

Launch _rel_attention_kernel over the requests of cu_seqlens_q.

rel_logits (sheared, (T + 128, H, extent + 256)) is None for the DSpark draft: plain block attention with all queries of a request at its last position. window (keys up to a query's position, inclusive) is 0 for full layers. mode selects the output: MODE_OUT_LSE writes the fp32 out (T, H, D) and natural-log lse (H, T); MODE_GATED the bf16 (T, H * D) out (gated by gate and sinks unless drafting; a single split); MODE_PARTIAL the fp32 out (S, T, H, D) and log2 lse (S, T, H) partials.

Source code in vllm/models/inkling2/nvidia/ops/attention_sm90.py
def rel_attention(
    q: torch.Tensor,
    key_cache: torch.Tensor,
    value_cache: torch.Tensor,
    out: torch.Tensor,
    lse: torch.Tensor | None,
    rel_logits: torch.Tensor | None,
    gate: torch.Tensor | None = None,
    sinks: torch.Tensor | None = None,
    *,
    block_table: torch.Tensor,
    cache_seqlens: torch.Tensor,
    cu_seqlens_q: torch.Tensor,
    max_query_len: int,
    softmax_scale: float,
    window: int,
    num_splits: int,
    mode: int,
) -> None:
    """Launch `_rel_attention_kernel` over the requests of `cu_seqlens_q`.

    `rel_logits` (sheared, `(T + 128, H, extent + 256)`) is None for the
    DSpark draft: plain block attention with all queries of a request at
    its last position. `window` (keys up to a query's position, inclusive) is
    0 for full layers. `mode` selects the output: `MODE_OUT_LSE` writes the
    fp32 `out` `(T, H, D)` and natural-log `lse` `(H, T)`; `MODE_GATED`
    the bf16 `(T, H * D)` `out` (gated by `gate` and `sinks` unless
    drafting; a single split); `MODE_PARTIAL` the fp32 `out`
    `(S, T, H, D)` and log2 `lse` `(S, T, H)` partials.
    """
    num_tokens, num_heads, head_dim = q.shape
    page, num_kv_heads = key_cache.shape[1:3]
    group = num_heads // num_kv_heads
    assert key_cache.stride() == value_cache.stride() and key_cache.stride(3) == 1
    assert q.stride(2) == 1
    assert mode != MODE_GATED or num_splits == 1
    draft = rel_logits is None
    extent = 0
    if rel_logits is not None:
        extent = rel_logits.shape[2] - 256
        assert rel_logits.stride(1) == rel_logits.shape[2]
        assert rel_logits.stride(2) == 1
    if mode == MODE_GATED and not draft:
        assert gate is not None and sinks is not None
        assert gate.stride(1) == sinks.stride(1) == 1
    # Unused pointers keep the launches typed alike.
    rel = q if rel_logits is None else rel_logits
    gate = q if gate is None else gate
    sinks = q if sinks is None else sinks
    if lse is None:
        lse = out
    block_m = _block_m(max_query_len, group)
    tokens_per_tile = block_m // group
    num_reqs = cache_seqlens.shape[0]
    grid = (
        triton.cdiv(max_query_len, tokens_per_tile),
        num_kv_heads * num_splits,
        num_reqs,
    )
    if mode == MODE_OUT_LSE:
        out_strides = (out.stride(0), out.stride(1), lse.stride(0), lse.stride(1))
    elif mode == MODE_GATED:
        assert out.is_contiguous()
        out_strides = (out.stride(0), 0, 0, 0)
    else:
        assert out.is_contiguous() and lse.is_contiguous()
        out_strides = (0, 0, 0, 0)
    _rel_attention_kernel[grid](
        q,
        key_cache,
        value_cache,
        rel,
        block_table,
        cache_seqlens,
        cu_seqlens_q,
        out,
        lse,
        gate,
        sinks,
        softmax_scale,
        num_tokens,
        num_splits,
        q.stride(0),
        q.stride(1),
        key_cache.stride(0),
        key_cache.stride(1),
        key_cache.stride(2),
        rel.stride(0),
        block_table.stride(0),
        *out_strides,
        gate.stride(0),
        sinks.stride(0),
        H=num_heads,
        G=group,
        D=head_dim,
        E=extent,
        PAGE=page,
        WINDOW=window,
        DRAFT=draft,
        MODE=mode,
        BLOCK_M=block_m,
        BLOCK_N=64 if block_m >= 128 else 128 if block_m <= 32 else 64,
        num_warps=8 if block_m >= 128 else 4,
        num_stages=2,
    )