vllm.models.hy_v4.nvidia.indexer_fusion
¶
Indexer RoPE and FP8 preparation with HY4's trailing rotary dimensions.
Functions:
-
fused_indexer_k_norm_rope–Apply the indexer key LayerNorm and RoPE in one pass.
-
fused_indexer_q_rope_quant–Prepare the indexer query and head weights for the FP8 MQA logits kernel.
fused_indexer_k_norm_rope(k, norm, positions, cos_sin_cache, is_neox_style)
¶
Apply the indexer key LayerNorm and RoPE in one pass.
Parameters:
-
(k¶Tensor) –(tokens, 128)BF16 indexer keys with the 64 rotary dims last. -
(norm¶LayerNorm) –A
LayerNormmodule withweight,biasandeps. -
(positions¶Tensor) –(tokens,)row of each token incos_sin_cache. -
(cos_sin_cache¶Tensor) –(rows, 64)cache holding 32 cosines then 32 sines. -
(is_neox_style¶bool) –Whether the RoPE pairs are NeoX-style halves.
Returns:
-
Tensor–(tokens, 128)normalized and rotated keys in the dtype ofk.
Source code in vllm/models/hy_v4/nvidia/indexer_fusion.py
fused_indexer_q_rope_quant(positions, q, cos_sin_cache, weights, softmax_scale, head_scale, is_neox)
¶
Prepare the indexer query and head weights for the FP8 MQA logits kernel.
Parameters:
-
(positions¶Tensor) –(tokens,)positions for the RoPE cache lookup. -
(q¶Tensor) –(tokens, heads, 128)BF16 indexer queries with the 64 rotary dims last. -
(cos_sin_cache¶Tensor) –(max_pos, 64)cache holding 32 cosines then 32 sines. -
(weights¶Tensor) –(tokens, heads)per-head indexer weights. -
(softmax_scale¶float) –Attention scale folded into the output weights.
-
(head_scale¶float) –Head-count normalization folded into the output weights.
-
(is_neox¶bool) –Rotary pair layout; NeoX pairs dim
iwithi + 32, otherwise adjacent dims are paired.
Returns:
-
Tensor–(q_fp8, weights_out): the rotated query in FP8 with one UE8M0 scale per -
Tensor–token and head, and fp32 weights with that scale,
softmax_scaleand -
tuple[Tensor, Tensor]–head_scalemultiplied in.