vllm.v1.attention.ops.flydsl_kernels.gqa_verify
¶
Speculative-decode verify (full or sliding-window attention) on gfx950 in FlyDSL.
Up to 8 verify tokens x GQA 8 (64 query columns) per KV head (for fewer tokens the
unused columns duplicate the last token and their tiles are trimmed), one wave per KV
head, one workgroup per (request, split); BF16 Q, FP8 e4m3 paged KV
[pages, 16 or 32, kv_heads, 64], split-KV FP32 partials. QK and PV run on BF16
MFMAs (K and V upcast exactly from FP8).
Transposed form per wave: S^T = K Q^T (keys x query columns) and O^T = V^T P^T on 16x16x32 BF16 MFMAs. * K is the A operand straight from HBM: lane (g, i) loads 16 contiguous bytes (dims 16g..16g+15) of key i of a 16-key half chunk; dims are permuted consistently in Q (k-step ks holds dims 16g + 8ks + 0..7). * Softmax runs along keys = MFMA rows: in-thread over the 8 rows of a lane, across the four 16-lane groups with two permlane swaps, and only when some score clears the running max by 2^8 (lazy rescale, exact: P <= 256). * P^T's accumulator registers are the PV B operand once the 32 keys of a chunk are relabelled (k-position 8g + i <-> key i < 4 ? 4g + i : 12 + 4g + i); V is written to wave-private LDS in that order with contiguous ds_write_b128 and read back as the V^T A operand with ds_read_b64_tr_b8 (16-byte chunks XOR-swizzled by row so the transposed reads are bank-conflict free). * Page ids are loaded one chunk group ahead of use, as vector loads in order with K and V. * K / V go through buffer loads (32-bit byte offsets), or 64-bit global loads when a layer's pool reaches 2^31 bytes. * NS register load slots refilled right after use keep ~NS chunks in flight per wave.
Functions:
-
balance_map–Balanced-grid map, computed on device:
wgmap[G, 4]int32 = (request, first -
gqa_verify_balanced–Full-attention verify on the length-balanced grid, merged into
out. -
gqa_verify_window–Sliding-window verify with equal splits per request, merged into
out. -
pick_balanced–(groups, wg_per_cu, waves_per_eu) of the balanced grid for a batch, or None
-
pick_window_splits–Splits per request of the sliding-window verify (a power of two). A window
_allreduce_groups(x_f32, use_max)
¶
Reduce an f32 across lanes l, l^16, l^32, l^48 (the four 16-lane groups).
Source code in vllm/v1/attention/ops/flydsl_kernels/gqa_verify.py
_cvt8(lo_word, hi_word)
¶
8 FP8 e4m3 bytes (two i32) -> v8bf16 in byte order (exact upcast, scale 1).
Source code in vllm/v1/attention/ops/flydsl_kernels/gqa_verify.py
_global_load16(addr)
¶
16 bytes (vector<4 x i32>) at the 64-bit global byte address addr.
Source code in vllm/v1/attention/ops/flydsl_kernels/gqa_verify.py
_partials(q, k, v, seq_lens, block_table, scale, k_scale, splits, *, waves_per_eu, wg_per_cu, query_len, wgmap=None, window=0)
¶
FP32 split-KV partials plus max / denom (log2 domain).
Sliding window (wgmap None): equal splits, partials [tokens, heads, splits(,
64)]. Full attention (wgmap from balance_map): splits is the workgroup
count G and the partials are [G, query_len, heads(, 64)], indexed by workgroup.
Source code in vllm/v1/attention/ops/flydsl_kernels/gqa_verify.py
_sload(view, index)
¶
Wave-uniform i32 element of a buffer view (scalar load into an SGPR).
_swap(op_name, x)
¶
Partner values across lane groups.
permlane{32,16}_swap of x with itself -> (vdst, vsrc) pair, i32.
Source code in vllm/v1/attention/ops/flydsl_kernels/gqa_verify.py
balance_map(seq_lens, groups, out=None)
¶
Balanced-grid map, computed on device: wgmap [G, 4] int32 = (request, first
chunk, end chunk, -) and wgstart [n + 1] int32. out optionally gives
buffers of at least those sizes to fill.
Source code in vllm/v1/attention/ops/flydsl_kernels/gqa_verify.py
gqa_verify_balanced(q, k, v, out, seq_lens, block_table, scale, k_scale, v_scale, *, query_len, grid, maps)
¶
Full-attention verify on the length-balanced grid, merged into out.
grid is pick_balanced's (groups, wg_per_cu, waves_per_eu); maps the
balance_map of seq_lens for those groups.
Source code in vllm/v1/attention/ops/flydsl_kernels/gqa_verify.py
gqa_verify_window(q, k, v, out, seq_lens, block_table, scale, k_scale, v_scale, *, query_len, window, num_cus)
¶
Sliding-window verify with equal splits per request, merged into out.
Source code in vllm/v1/attention/ops/flydsl_kernels/gqa_verify.py
pick_balanced(num_seqs, kv_heads, num_cus)
¶
(groups, wg_per_cu, waves_per_eu) of the balanced grid for a batch, or None when the batch has too many requests for it (each needs its own workgroup).
Fixed per batch size: CUDA-graph safe. A workgroup runs one wave per KV head. One workgroup per CU suits 4 or more KV heads below 24 requests; two per CU (waves_per_eu 2: twice the loads in flight) from 24 requests up, and from 3 requests up with 2 or fewer KV heads (no waves_per_eu hint: the LDS pin already allows only one wave per SIMD).
Source code in vllm/v1/attention/ops/flydsl_kernels/gqa_verify.py
pick_window_splits(num_seqs, kv_heads, num_cus)
¶
Splits per request of the sliding-window verify (a power of two). A window spans few chunks: one or two requests split it finely, larger batches use fewer, longer splits once the grid covers the CUs.