vllm.models.mistral.mistral_large4.amd.specialized_decode.runtime.decode
¶
Step-mode specialized decode: lifecycle, per-step plan and fail-stop.
plan() runs once per forward. A single-token decode step (eager, or inside
a FULL graph capture) runs every MoE layer as one kernel launch (run);
every other step runs vLLM's layers. Every predicate reads host state that
every TP rank holds alike: the kernels all-reduce in-kernel, so ranks that
disagreed would wait on each other until the fail-stop bound.
Classes:
-
SpecializedDecode–Built after weight loading; never replaces stock modules or parameters.
-
Step–One planned step:
rowstokens,qconsecutive ones per request.
Functions:
-
step_refusal–Why this forward runs vLLM's layers, or None for the kernels: a decode
SpecializedDecode
¶
Built after weight loading; never replaces stock modules or parameters.
Methods:
-
after_step–After every forward (kernels or stock, eager or in a FULL graph),
Source code in vllm/models/mistral/mistral_large4/amd/specialized_decode/runtime/decode.py
76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 | |
_bind(context)
¶
The first planned step: check the caches, build the launchers.
Source code in vllm/models/mistral/mistral_large4/amd/specialized_decode/runtime/decode.py
after_step()
¶
After every forward (kernels or stock, eager or in a FULL graph), before logits: a device-side assert that the fail-stop word is clear. A set word traps the GPU queue, so none of the step's later work runs and the worker aborts: no token of a failed step reaches a client.
Source code in vllm/models/mistral/mistral_large4/amd/specialized_decode/runtime/decode.py
Step
dataclass
¶
step_refusal(context, layer_name, positions, widths, q=1)
¶
Why this forward runs vLLM's layers, or None for the kernels: a decode
step of whole requests of q rows each (a speculative verify block, its
causality from each row's position; padded rows have slot -1).