vllm.models.muse_spark.amd.mono.kernels.draft
¶
DSpark draft: the 5 draft layers and the final norm of a draft step as one persistent launch.
One rank of a TP8 group, 256 CTAs x 512 threads, S = 8 / 16 rows: 1 / 2 requests of 7 query rows (vLLM's rows 7 r + j; rows 7 NR .. S - 1 are padding: computed, never stored to vLLM's tensors). A draft layer is K3's attention block (k3.py) followed by a dense SwiGLU MLP:
IN x = embeds -> the fp32 stream (smid), the input of the first boundary
BND (FFN boundary of layer l - 1; the embeddings for l = 0) the owner of column block b
(rank b // 32) sums the W down_proj partials in rank order (bf16, the all-reduce output),
post-FFN norm (weight first, eps 1e-8), gated residual, the pre-QKV norm of layer l
(eps 1e-5) of the bf16-rounded stream; the reduced rows and the norm partials all-gathered
QKV layer l's qkv + o-gate GEMV (K in eighths, as K3); the owners finish whole heads: qk-norm,
RoPE, q, the bf16 K / V into the layer's LBHNC draft KV cache (slot_mapping), the o-gate
ATT after every CTA's QKV flag: the causal sliding-window attention over the draft KV
(attn_frag.py, 7 query rows a request)
O o-norm x sigmoid(gate), o_proj (this rank's K = 1024), partial to the block owner rank
AB the attention boundary (as BND: post-attention norm weightless eps 1e-8, gated residual,
pre-FFN norm) -> ffn_in
GU gate_up (2 x 2048 rows a rank, K in eighths over 8-CTA groups; the partials go to the
token owner CTA), bf16 gate / up, SwiGLU -> h (bf16, 2048 a rank)
DN down_proj rows 32 b .. (K = 2048), partial to the block owner rank
after layer 4: BND with the final norm -> the draft hidden states (bf16, rows < 7 NR)
Every norm takes the bf16-rounded stream like the stock layer; its rstd comes from one token reduction round of the unrounded stream (k3.tok_rstd2's two-norm identity). The norm multipliers are fp32 (weight + gain center). Peer regions are double buffered by tag parity; the tags of a launch are epoch * 64 + 1 .. 6 (6 phases, so consecutive phases alternate parity across launches), and CTA 0 bumps the epoch at the end of the launch.
Functions:
-
build_vllm–The draft build vLLM launches for S rows (8 / 16: 1 / 2 requests of 7 rows), TP = 8.
-
peer_layout–Every rank's peer buffer (IPC-shared), [parity] first.
-
scratch_layout–This rank's scratch regions.
-
sfresh–An opaque s_mov of a wave-uniform i32 (dev.fresh for SGPRs).
build_vllm(S=8, W=8)
¶
The draft build vLLM launches for S rows (8 / 16: 1 / 2 requests of 7 rows), TP = 8.
peer_layout(S, world=W)
¶
Every rank's peer buffer (IPC-shared), [parity] first.
Source code in vllm/models/muse_spark/amd/mono/kernels/draft.py
scratch_layout(S, world=W)
¶
This rank's scratch regions.
Source code in vllm/models/muse_spark/amd/mono/kernels/draft.py
sfresh(v)
¶
An opaque s_mov of a wave-uniform i32 (dev.fresh for SGPRs).