vllm.models.kimi_k3.nvidia.specialized_decode.drafter.gqa
¶
The specialized decode's GQA DSpark drafter (Qwen3DSparkModel drafts of
Kimi K3.1, e.g. Inferact/Kimi-K3.1-Pro-DSpark: GQA sliding-window layers).
What changes against the stock Qwen3DSparkForCausalLM (all exact up to the
summation order of a GEMM):
- fc (the context projection, replicated in the stock model) is
column-parallel with its output gathered on the drafter's mailbox.
- every drafter linear, the fused context K/V projection and the lm_head shard
run on linears.DraftGemm.
- the k sequential Markov steps run fused (markov.py).
- the row-parallel o_proj / down_proj partials are reduced in the RMSNorm
that follows (fused_allreduce_rms_norm, as the compiled stock model fuses
them), and at small batches the embedding's all-reduce is fused with the first
RMSNorm (comm.DrafterComm.ar_norm).
The draft attention stays the stock sliding-window backend.
Classes:
-
SpecializedQwen3DSparkForCausalLM–The stock GQA DSpark drafter with the changes listed above, built in place
-
SpecializedQwen3DSparkModel–Qwen3DSparkModelwhose row-parallel outputs are reduced by the RMSNorm
SpecializedQwen3DSparkForCausalLM
¶
Bases: Qwen3DSparkForCausalLM
The stock GQA DSpark drafter with the changes listed above, built in place
of Qwen3DSparkForCausalLM (model.register). Same weights and names.
Methods:
-
sample_sequential–The speculator asks the model for its own Markov sampling first.
Source code in vllm/models/kimi_k3/nvidia/specialized_decode/drafter/gqa.py
sample_sequential(speculator, num_reqs, head_hidden)
¶
The speculator asks the model for its own Markov sampling first.
Source code in vllm/models/kimi_k3/nvidia/specialized_decode/drafter/gqa.py
SpecializedQwen3DSparkModel
¶
Bases: Qwen3DSparkModel
Qwen3DSparkModel whose row-parallel outputs are reduced by the RMSNorm
after them (forward).
Methods:
-
forward–DFlashQwen3Model.forwardwith explicit reductions (see the class).
Source code in vllm/models/kimi_k3/nvidia/specialized_decode/drafter/gqa.py
38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 | |
forward(input_ids, positions, input_embeds=None)
¶
DFlashQwen3Model.forward with explicit reductions (see the class).
Source code in vllm/models/kimi_k3/nvidia/specialized_decode/drafter/gqa.py
_after_loading(model)
¶
Route the drafter's linears and its fused context K/V projection through DraftGemm and build its collectives (every TP rank, in the same order).
Source code in vllm/models/kimi_k3/nvidia/specialized_decode/drafter/gqa.py
_gathered_fc(fc, prefix)
¶
fc column-parallel with a gathered output when TP > 1 divides it,
replacing the replicated one before the weights load; None keeps it.