Skip to content

vllm.models.hy_v4.nvidia.indexer_fusion

Indexer RoPE and FP8 preparation with HY4's trailing rotary dimensions.

Functions:

fused_indexer_k_norm_rope(k, norm, positions, cos_sin_cache, is_neox_style)

Apply the indexer key LayerNorm and RoPE in one pass.

Parameters:

  • k

    (Tensor) –

    (tokens, 128) BF16 indexer keys with the 64 rotary dims last.

  • norm

    (LayerNorm) –

    A LayerNorm module with weight, bias and eps.

  • positions

    (Tensor) –

    (tokens,) row of each token in cos_sin_cache.

  • cos_sin_cache

    (Tensor) –

    (rows, 64) cache holding 32 cosines then 32 sines.

  • is_neox_style

    (bool) –

    Whether the RoPE pairs are NeoX-style halves.

Returns:

  • Tensor –

    (tokens, 128) normalized and rotated keys in the dtype of k.

Source code in vllm/models/hy_v4/nvidia/indexer_fusion.py
def fused_indexer_k_norm_rope(
    k: torch.Tensor,
    norm: LayerNorm,
    positions: torch.Tensor,
    cos_sin_cache: torch.Tensor,
    is_neox_style: bool,
) -> torch.Tensor:
    """Apply the indexer key LayerNorm and RoPE in one pass.

    Args:
        k: `(tokens, 128)` BF16 indexer keys with the 64 rotary dims last.
        norm: A `LayerNorm` module with `weight`, `bias` and `eps`.
        positions: `(tokens,)` row of each token in `cos_sin_cache`.
        cos_sin_cache: `(rows, 64)` cache holding 32 cosines then 32 sines.
        is_neox_style: Whether the RoPE pairs are NeoX-style halves.

    Returns:
        `(tokens, 128)` normalized and rotated keys in the dtype of `k`.

    """
    out = torch.empty((k.shape[0], 128), device=k.device, dtype=k.dtype)
    if k.shape[0]:
        _k_norm_rope[(triton.cdiv(k.shape[0], 4),)](
            k,
            norm.weight,
            norm.bias,
            positions,
            cos_sin_cache,
            out,
            k.shape[0],
            k.stride(0),
            cos_sin_cache.stride(0),
            norm.eps,
            is_neox_style,
            enable_fp_fusion=False,
        )
    return out

fused_indexer_q_rope_quant(positions, q, cos_sin_cache, weights, softmax_scale, head_scale, is_neox)

Prepare the indexer query and head weights for the FP8 MQA logits kernel.

Parameters:

  • positions

    (Tensor) –

    (tokens,) positions for the RoPE cache lookup.

  • q

    (Tensor) –

    (tokens, heads, 128) BF16 indexer queries with the 64 rotary dims last.

  • cos_sin_cache

    (Tensor) –

    (max_pos, 64) cache holding 32 cosines then 32 sines.

  • weights

    (Tensor) –

    (tokens, heads) per-head indexer weights.

  • softmax_scale

    (float) –

    Attention scale folded into the output weights.

  • head_scale

    (float) –

    Head-count normalization folded into the output weights.

  • is_neox

    (bool) –

    Rotary pair layout; NeoX pairs dim i with i + 32, otherwise adjacent dims are paired.

Returns:

  • Tensor –

    (q_fp8, weights_out): the rotated query in FP8 with one UE8M0 scale per

  • Tensor –

    token and head, and fp32 weights with that scale, softmax_scale and

  • tuple[Tensor, Tensor] –

    head_scale multiplied in.

Source code in vllm/models/hy_v4/nvidia/indexer_fusion.py
def fused_indexer_q_rope_quant(
    positions: torch.Tensor,
    q: torch.Tensor,
    cos_sin_cache: torch.Tensor,
    weights: torch.Tensor,
    softmax_scale: float,
    head_scale: float,
    is_neox: bool,
) -> tuple[torch.Tensor, torch.Tensor]:
    """Prepare the indexer query and head weights for the FP8 MQA logits kernel.

    Args:
        positions: `(tokens,)` positions for the RoPE cache lookup.
        q: `(tokens, heads, 128)` BF16 indexer queries with the 64 rotary dims
            last.
        cos_sin_cache: `(max_pos, 64)` cache holding 32 cosines then 32 sines.
        weights: `(tokens, heads)` per-head indexer weights.
        softmax_scale: Attention scale folded into the output weights.
        head_scale: Head-count normalization folded into the output weights.
        is_neox: Rotary pair layout; NeoX pairs dim `i` with `i + 32`,
            otherwise adjacent dims are paired.

    Returns:
        `(q_fp8, weights_out)`: the rotated query in FP8 with one UE8M0 scale per
        token and head, and fp32 weights with that scale, `softmax_scale` and
        `head_scale` multiplied in.

    """
    assert current_platform.is_cuda()
    assert q.dtype == torch.bfloat16
    assert q.shape[-1] == 128
    assert cos_sin_cache.shape[-1] == 64
    assert weights.shape == q.shape[:2]

    q_fp8 = torch.empty_like(q, dtype=current_platform.fp8_dtype())
    weights_out = torch.empty_like(weights, dtype=torch.float32)
    fp8_min, fp8_max = get_fp8_min_max()
    _fused_indexer_q_rope_quant_kernel[(q.shape[0], q.shape[1])](
        positions,
        q,
        q.stride(0),
        q.stride(1),
        cos_sin_cache,
        cos_sin_cache.stride(0),
        q_fp8,
        q_fp8.stride(0),
        q_fp8.stride(1),
        weights,
        weights.stride(0),
        weights.stride(1),
        weights_out,
        weights_out.stride(0),
        weights_out.stride(1),
        softmax_scale,
        head_scale,
        fp8_min=fp8_min,
        fp8_max=fp8_max,
        is_neox=is_neox,
        num_warps=1,
        enable_fp_fusion=False,
    )
    return q_fp8, weights_out