Skip to content

vllm.models.minimax_m3.amd.glue.kernel

MiniMax-M3.1 decode glue (TP4, gfx950, 17 .. 192 rows): one persistent FlyDSL launch from the unreduced o_proj partial to the MoE inputs, ahead of AITER's A16W4 gemm1 / gemm2.

Stages of the default build (policy; stage walk, BLOCKS x THREADS, S rows, G = S / 4 rows a TP owner):

  • rs push (CTAs 0 .. 127, 48 columns a task): this rank's partial, rows of owner g -> rank g's RS inbox [this rank]. Transport "lamport" (default): plain bf16 16-byte stores (lane = (row, vector)) into the owner's sentinel inbox, the 3 peers only; lpmix 4: a lane's j-th store goes to peer (j + row // 4) % 3, so each destination gets 384-byte (3-line) chunks. "tagged": tagged triple pairs (3 bf16 + 16-bit tag in one 64-bit store); mixdst: each store instruction's four 16-lane groups go to four different ranks.
  • rs reduce (CTAs 0 .. 127): this rank's G rows, the 4 ranks' partials summed in rank order (f32) -> bf16 h (the own partial read straight from part in lamport builds), pushed to every rank's AG slot [this rank]. lpag (lamport, rows >= LP_AG_MIN_ROWS): plain bf16 16-byte stores into the AG sentinel inboxes, this rank's own first (ownfirst); else tagged triples. Mode 2 (the residual sharded by owner rows): bf16(h + this rank's residual rows).
  • own rows (orouter: CTAs 128 .. 128 + G - 1, which issued no remote stores): this rank's G owner rows (reduced here, landing first): h + residual -> residual out, Gemma RMSNorm -> normed (gemm1 input), MXFP8 copy; xfirst: normed stored, drained and XRDY published before residual out and the MXFP8 copy.
  • router (CTAs 128 .. 255: 8 tiles of 16 experts x 16 K parts) on the G owner rows only: the FP32 weight (loaded at kernel entry) as bf16 hi + mid + lo (exact; rbf16) x normed on 16x16x16 bf16 MFMAs -> partial logits.
  • route (CTAs 128 .. 128 + G - 1, a CTA an owner row): parts summed in order, sigmoid,
  • bias (prefetched) for selection, top-4 (ties to the lower expert), weights = sigmoid / sum x 2.0 (the routed scale) -> tagged (id, weight) pairs into every rank's peer heap (the only cross-rank routing exchange).
  • peer rows (CTAs 0 .. 127, and 128 + G .. past 128 rows): the 3G gathered rows: residual out, normed, MXFP8.
  • sort (CTA 255): every rank's top-k pairs (one poll batch) -> top-k ids / weights for all rows + AITER moe_sorting's outputs (block 16); fastsort: one-wave ds_bpermute scan, (expert, slot) padding threads.
  • re-arm (lamport): every CTA, after a barrier at its end, writes the sentinel back over the inbox slots it consumed (seam CTAs: their task's RS segments; row CTAs: their row's AG segments).
  • finish (afinish): the CTA whose device-counter add completes the launch's BLOCKS moves the epoch.

Every option is a GlueBuild field (None = policy(rows), the measured per-width choice, MI355X TP4; options measured and left off: rowvec, agdefer, rnorm, prepoll outside 48 .. 64 rows; lamport: lpmix 1, lpprobe, lpack).

mode: 0 = load only (compile), 1 = full residual (every row on every rank), 2 = residual sharded by owner rows.

Lamport transport (specialized_decode/mono/lamport.py): a payload is its own flag. The inboxes (peer buffer past the tagged halves, two parities by epoch & 1) hold the sentinel 0xFFFF in every half-word until a push lands; a reader re-loads until none of the half-words it uses is the sentinel, and the CTA re-arms them after a barrier at its end. The host fills the inboxes with 0xFF at setup and at every reset (arm_peer). Numerics are the tagged build's: the same rank-order sums, the same row-stage lane layout (outputs bitwise equal to the tagged kernel's).

Sync, tags, the device-side epoch and fail-stop are the upstream M3 monokernel's (specialized_decode/mono/sync.py, deepseek_v41 mono/common): every wait (tagged pair, triple or Lamport slot) is bounded by WAIT_TICKS; a timeout sets the error record [1, rank, epoch, site] on every rank and every waiter that sees it gives up.

Functions:

  • ag_push_all –

    Pair i of every rank's AG region := bits (mixdst: lane group q's j-th store to

  • ag_push_peers –

    Pair i of the three peers' AG regions := bits (mixdst: rotated by lane group).

  • arm_peer –

    Host side, no launch in flight: every Lamport inbox half-word := the sentinel

  • bf16_bits –

    The 16 bits of bf16(v) (round to nearest even), in an Int64.

  • finish –

    Every CTA marks the launch done; CTA 0 waits for the marks, then moves the epoch

  • lamport_span –

    (byte offset, bytes) of the Lamport inboxes in the peer buffer: 0xFF at setup and

  • lp_count_acked –

    lpack: this CTA's pushes acked (vmcnt 0 in every wave), then one add to the epoch

  • lp_peer –

    Rank p's peer buffer base (p traced).

  • lp_rearm_end –

    Consumers re-arm at their CTA's end: after a CTA barrier every lane of the CTA

  • lp_rearm_row –

    AG inbox row t (owner t // G, row t % G) on this rank: its 128 task segments, 768

  • lp_rearm_rs –

    The RS slots seam task task consumed on this rank: 3 peer segments x G rows

  • lp_row_ag –

    lamport: row t's 4 triples a lane from this rank's AG inbox (owner t // G) ->

  • lp_row_of –

    (has a row, the row t) of the row stage CTA bid ran (run_glue's placement).

  • lp_seg –

    Byte address of segment (this launch's parity, rank who, seam task, row r) of

  • lp_wait_acked –

    lpack: until every seam CTA of this rank counted its acked RS pushes (thread 0

  • max_pad –

    sorted_ids length, as aiter.fused_moe.moe_sorting allocates it.

  • mfma_bf16 –

    16x16x16 bf16 MFMA on 64-bit packed operands (4 bf16 each, element 0 =

  • peer_row –

    Peer row q (0 .. 3G - 1) in owner order rank + 1, rank + 2, rank + 3.

  • policy –

    The options a width builds with when its GlueBuild leaves them None (measured,

  • prepoll –

    One lane a wave spins on one pair before the CTA's full poll (a hint only: the

  • resolve –

    key with every None option set by policy.

  • router_weights –

    The task's FP32 weights (a lane's float4 x 3 chunks), issued at kernel entry.

  • rs_done_wait –

    Every seam CTA of this launch has reduced: one add each, then wait for the

  • sort_scan_wave –

    Exclusive scan of the padded counts by wave 0 (two experts a lane, ds_bpermute

  • split3 –

    A float4 -> three 64-bit words of 4 bf16 (hi, mid, lo) with w = hi + mid + lo

  • stage_route –

    Row t: top-4 of sigmoid + bias (ties to the lower expert); weights = selected

  • stage_router –

    Experts 16 tile .. over K part part for every row (16-row tiles): f32 MFMAs,

  • stage_router_rn –

    stage_router on the owner rows with its own normalization (rnorm): rstd from

  • stage_row –

    Row t: its owner's gathered h + residual (f32) -> residual out (bf16); Gemma

  • stage_row_vec –

    stage_row with vector stores: the f32 sums go through LDS; each thread then

  • stage_rs_push –

    Triples 16 task ..: every row of this rank's partial to its owner's RS slot [this

  • stage_rs_push_lp –

    This rank's partial, owner g's rows, the task's 48 columns -> rank g's RS inbox

  • stage_rs_reduce –

    Triples 16 task .. of this rank's G rows: the 4 ranks' triples in rank order

  • stage_rs_reduce_batched –

    stage_rs_reduce with every pass's poll in one batch (rows past G poll row G - 1

  • stage_rs_reduce_lp3 –

    lamport_rs: this rank's G rows of the task's columns in the tagged reducer's

  • stage_rs_reduce_lpv –

    lamport: this rank's G rows of the task's columns, lane = (row, 16-byte vector):

  • stage_sort –

    Every row's top-k -> AITER moe_sorting's outputs (block 16): expert segments in

  • stage_zero –

    moe_buf [S, HIDDEN] bf16 := 0, slice part of ZERO_CTAS (gemm2 adds into it

  • stamp –

    s_memrealtime -> tl[CTA][point] (timeline builds); name a TL name or a raw

  • tl_participants –

    The CTAs that do the work a stamp closes (the harness reduces min / median / max

  • triple_bits –

    Row t's bf16 at columns 3 triple .. of a [*, HIDDEN] bf16 buffer, in bits

  • triple_value –

    Value k (0 .. 2) of a triple pair's (low word, high word).

_by_rank(c, own, got)

[own] + got[d - 1] (peer at position d after this rank) -> the same words per absolute source rank.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def _by_rank(c, own, got):
    """[own] + got[d - 1] (peer at position d after this rank) -> the same words per
    absolute source rank."""
    rank = c["rank"]
    out = []
    for s in range(TP):
        ws = list(own)
        for d in range(1, TP):
            m = ((rank + d) % TP) == s
            ws = [sel(m, got[d - 1][e], ws[e]) for e in range(len(ws))]
        out.append(ws)
    return out

_err(c)

This launch's error word reader (the glue's peer buffer starts with the error record).

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def _err(c):
    """This launch's error word reader (the glue's peer buffer starts with the error
    record)."""
    return lambda: error_word(c["sym"])

_ld16(base, dw)

Four dwords at dword dw of a (cached) buffer, as Int32 words.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def _ld16(base, dw):
    """Four dwords at dword `dw` of a (cached) buffer, as Int32 words."""
    return lp.words(lp.load16(base, dw))

_pad(col)

LDS index of a row column with one pad word every 32 (a thread's 32-column MXFP8 group: distinct banks).

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def _pad(col):
    """LDS index of a row column with one pad word every 32 (a thread's 32-column MXFP8
    group: distinct banks)."""
    return col + (col >> 5)

ag_push_all(c, i, bits)

Pair i of every rank's AG region := bits (mixdst: lane group q's j-th store to rank (rank + 1 + q + j) % 4; ownfirst: this rank's own slot first, then the three peers rotated by lane group).

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def ag_push_all(c, i, bits):
    """Pair i of every rank's AG region := bits (mixdst: lane group q's j-th store to
    rank (rank + 1 + q + j) % 4; ownfirst: this rank's own slot first, then the three
    peers rotated by lane group)."""
    if const_expr(c["ownfirst"]):
        push(c, c["own"], i, bits)
        ag_push_peers(c, i, bits)
    elif const_expr(c["mixdst"]):
        q = c["lane"] // 16
        for j in range_constexpr(TP):
            psel = (c["rank"] + 1 + q + j) % TP
            base = c["peer"][0].value
            for k in range_constexpr(1, TP):
                base = sel(psel == k, c["peer"][k].value, base)
            push(c, Addr("peer", Space.PEER, base), i, bits)
    else:
        for p in range_constexpr(TP):
            push(c, c["peer"][p], i, bits)

ag_push_peers(c, i, bits)

Pair i of the three peers' AG regions := bits (mixdst: rotated by lane group).

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def ag_push_peers(c, i, bits):
    """Pair i of the three peers' AG regions := bits (mixdst: rotated by lane group)."""
    q = c["lane"] // 16 if c["mixdst"] else 0
    for j in range_constexpr(TP - 1):
        psel = (c["rank"] + 1 + (q + j) % (TP - 1)) % TP
        base = c["peer"][0].value
        for k in range_constexpr(1, TP):
            base = sel(psel == k, c["peer"][k].value, base)
        push(c, Addr("peer", Space.PEER, base), i, bits)

arm_peer(peer_bytes_tensor)

Host side, no launch in flight: every Lamport inbox half-word := the sentinel (the tagged regions untouched).

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def arm_peer(peer_bytes_tensor) -> None:
    """Host side, no launch in flight: every Lamport inbox half-word := the sentinel
    (the tagged regions untouched)."""
    lp.arm(peer_bytes_tensor, *lamport_span())

bf16_bits(v)

The 16 bits of bf16(v) (round to nearest even), in an Int64.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def bf16_bits(v):
    """The 16 bits of bf16(v) (round to nearest even), in an Int64."""
    return fx.Int64(fx.Uint32(bf16_round(v).bitcast(fx.Int32)) >> 16)

finish(c, epoch, tag)

Every CTA marks the launch done; CTA 0 waits for the marks, then moves the epoch on. afinish: every CTA adds one to a device counter; the CTA whose add completes a multiple of BLOCKS (the launch's last) moves the epoch.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def finish(c, epoch, tag):
    """Every CTA marks the launch done; CTA 0 waits for the marks, then moves the epoch
    on. afinish: every CTA adds one to a device counter; the CTA whose add completes a
    multiple of BLOCKS (the launch's last) moves the epoch."""
    tid, red = c["tid"], c["red"]
    if const_expr(c.get("afinish", False)):
        cta_sync(drain=True)
        if tid == 0:
            ptr = fx.inttoptr(
                fx.PointerType.get(T.i32, fx.AddressSpace.Global, 4),
                epoch + fx.Int64(4 * DONE_WORD),
            )
            old = fx.llvm.atomic_add(ptr, fx.Int32(1), syncscope="agent")
            if ((old + 1) & (BLOCKS - 1)) == 0:
                store32(epoch, 0, tag, "agent")
        return
    cta_sync(drain=True)
    if tid == 0:
        sync_release("agent")
        store32(epoch, EPOCH_MARKS + fx.block_idx.x, tag, "agent")
    if fx.block_idx.x == 0:
        if tid == 0:
            fx.ptr_store(fx.Int32(0).bitcast(fx.Float32), red)
        gpu.barrier()
        if tid < BLOCKS:
            t0 = memrealtime()
            mark = load32(epoch, EPOCH_MARKS + tid, "agent")
            late = fx.Int64(0)
            while (mark != tag) & (late < c["mb"].ticks):
                rocdl.s_sleep(1)
                mark = load32(epoch, EPOCH_MARKS + tid, "agent")
                late = memrealtime() - t0
            if mark != tag:
                fx.ptr_store(fx.Int32(1).bitcast(fx.Float32), red)
            sync_acquire("agent")
        cta_sync()
        if tid == 0:
            if fx.ptr_load(red).bitcast(fx.Int32) == 0:
                store32(epoch, 0, tag, "agent")
            else:
                c["mb"].fail(SITE_EPOCH)

lamport_span()

(byte offset, bytes) of the Lamport inboxes in the peer buffer: 0xFF at setup and at every reset.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def lamport_span() -> tuple[int, int]:
    """(byte offset, bytes) of the Lamport inboxes in the peer buffer: 0xFF at setup and
    at every reset."""
    return LP_RS, LP_END - LP_RS

lp_count_acked(c)

lpack: this CTA's pushes acked (vmcnt 0 in every wave), then one add to the epoch block's RS_DONE_WORD; thread 0 keeps the launch's target (the next multiple of SEAM_TASKS) in LDS for lp_wait_acked.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def lp_count_acked(c):
    """lpack: this CTA's pushes acked (vmcnt 0 in every wave), then one add to the epoch
    block's RS_DONE_WORD; thread 0 keeps the launch's target (the next multiple of
    SEAM_TASKS) in LDS for lp_wait_acked."""
    cta_sync(drain=True)
    if c["tid"] == 0:
        ptr = fx.inttoptr(
            fx.PointerType.get(T.i32, fx.AddressSpace.Global, 4),
            c["epoch"] + fx.Int64(4 * RS_DONE_WORD),
        )
        old = fx.llvm.atomic_add(ptr, fx.Int32(1), syncscope="agent")
        fx.ptr_store(
            ((old // SEAM_TASKS + 1) * SEAM_TASKS).bitcast(fx.Float32),
            c["red"] + LP_ACK_LDS,
        )

lp_peer(c, p)

Rank p's peer buffer base (p traced).

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def lp_peer(c, p):
    """Rank p's peer buffer base (p traced)."""
    b = c["raw"][0]
    for q in range_constexpr(1, TP):
        b = sel(p == q, c["raw"][q], b)
    return b

lp_rearm_end(c, bid)

Consumers re-arm at their CTA's end: after a CTA barrier every lane of the CTA has consumed its slots.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def lp_rearm_end(c, bid):
    """Consumers re-arm at their CTA's end: after a CTA barrier every lane of the CTA
    has consumed its slots."""
    gpu.barrier()
    if bid < SEAM_TASKS:
        lp_rearm_rs(c, bid)
    if const_expr(c["lp_ag"]):
        has, t = lp_row_of(c, bid)
        if has:
            lp_rearm_row(c, t)

lp_rearm_row(c, t)

AG inbox row t (owner t // G, row t % G) on this rank: its 128 task segments, 768 sentinel stores.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def lp_rearm_row(c, t):
    """AG inbox row t (owner t // G, row t % G) on this rank: its 128 task segments, 768
    sentinel stores."""
    g_rows, tid = c["G"], c["tid"]
    owner = t // g_rows
    r = t % g_rows
    sent = [fx.Int32(-1)] * SENT_WORDS
    nv = SEAM_TASKS * LP_VEC
    for i in range_constexpr(-(-nv // THREADS)):
        v = tid + THREADS * i
        if v < nv:
            lp_st16(
                lp_seg(c, c["sym"], LP_AG, owner, v // LP_VEC, r)
                + fx.Int64(16) * fx.Int64(v % LP_VEC),
                sent,
            )

lp_rearm_rs(c, task)

The RS slots seam task task consumed on this rank: 3 peer segments x G rows (contiguous), sentinel stores.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def lp_rearm_rs(c, task):
    """The RS slots seam task `task` consumed on this rank: 3 peer segments x G rows
    (contiguous), sentinel stores."""
    g_rows, rank, tid = c["G"], c["rank"], c["tid"]
    n = LP_VEC * g_rows
    sent = [fx.Int32(-1)] * SENT_WORDS
    for d in range_constexpr(1, TP):
        base = lp_seg(c, c["sym"], LP_RS, (rank + d) % TP, task, 0)
        for i in range_constexpr(-(-n // THREADS)):
            v = tid + THREADS * i
            if v < n:
                lp_st16(base + fx.Int64(16) * fx.Int64(v), sent)

lp_row_ag(c, t, trs)

lamport: row t's 4 triples a lane from this rank's AG inbox (owner t // G) -> [[(lo, hi)]] as the tagged poll.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def lp_row_ag(c, t, trs):
    """lamport: row t's 4 triples a lane from this rank's AG inbox (owner t // G) ->
    [[(lo, hi)]] as the tagged poll."""
    g_rows, tid = c["G"], c["tid"]
    owner = t // g_rows
    r = t % g_rows
    tl = tid % TASK_TRIPLES
    addrs = [lp_seg(c, c["sym"], LP_AG, owner, tr // TASK_TRIPLES, r) for tr in trs]
    if const_expr(c["lpprobe"]):
        lp_probe(c, addrs[-1] + fx.Int64(lp_triple_off(tl)), c["tid"] >= 0)
    return [[p] for p in lp_poll_triples(c, addrs, tl, [True] * len(trs), SITE_AG)]

lp_row_of(c, bid)

(has a row, the row t) of the row stage CTA bid ran (run_glue's placement).

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def lp_row_of(c, bid):
    """(has a row, the row t) of the row stage CTA `bid` ran (run_glue's placement)."""
    s, g_rows = c["S"], c["G"]
    if const_expr(not c["orouter"]):
        return bid < s, bid
    npeer = s - g_rows
    own = (bid >= ROUTER0) & (bid < ROUTER0 + g_rows)
    peer = bid < min(npeer, ROUTER0)
    t = sel(
        own, c["rank"] * g_rows + bid - ROUTER0, peer_row(c, fx.min(bid, npeer - 1))
    )
    has = own | peer
    if const_expr(npeer > ROUTER0):
        peer2 = (bid >= ROUTER0 + g_rows) & (bid < ROUTER0 + g_rows + npeer - ROUTER0)
        t = sel(
            peer2, peer_row(c, fx.max(fx.min(bid - g_rows, npeer - 1), fx.Int32(0))), t
        )
        has = has | peer2
    return has, t

lp_seg(c, base, region, who, task, r)

Byte address of segment (this launch's parity, rank who, seam task, row r) of inbox region in the peer buffer at base.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def lp_seg(c, base, region, who, task, r):
    """Byte address of segment (this launch's parity, rank `who`, seam task, row r) of
    inbox `region` in the peer buffer at `base`."""
    blk = (c["lpar"] * TP + fx.Int64(who)) * SEAM_TASKS + fx.Int64(task)
    return base + fx.Int64(region) + (blk * GMAX + fx.Int64(r)) * LP_SEG

lp_wait_acked(c)

lpack: until every seam CTA of this rank counted its acked RS pushes (thread 0 polls, bounded; then a barrier).

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def lp_wait_acked(c):
    """lpack: until every seam CTA of this rank counted its acked RS pushes (thread 0
    polls, bounded; then a barrier)."""
    if c["tid"] == 0:
        target = fx.ptr_load(c["red"] + LP_ACK_LDS).bitcast(fx.Int32)
        t0 = memrealtime()
        n = load32(c["epoch"], RS_DONE_WORD, "agent")
        late = fx.Int64(0)
        while (n - target < 0) & (late < c["mb"].ticks):
            rocdl.s_sleep(1)
            n = load32(c["epoch"], RS_DONE_WORD, "agent")
            late = memrealtime() - t0
        if n - target < 0:
            c["mb"].fail(SITE_RS)
    cta_sync()

max_pad(s)

sorted_ids length, as aiter.fused_moe.moe_sorting allocates it.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def max_pad(s: int) -> int:
    """sorted_ids length, as aiter.fused_moe.moe_sorting allocates it."""
    n = s * TOPK + EXPERTS * SORT_BLOCK - TOPK
    return -(-n // SORT_BLOCK) * SORT_BLOCK

mfma_bf16(a64, b64, acc)

16x16x16 bf16 MFMA on 64-bit packed operands (4 bf16 each, element 0 = lowest k).

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def mfma_bf16(a64, b64, acc):
    """16x16x16 bf16 MFMA on 64-bit packed operands (4 bf16 each, element 0 =
    lowest k)."""
    a = fx.Vector.from_elements([a64], dtype=fx.Int64).bitcast(fx.Int16)
    b = fx.Vector.from_elements([b64], dtype=fx.Int64).bitcast(fx.Int16)
    return fx.Vector(
        rocdl.mfma_f32_16x16x16bf16_1k(T.vec(4, T.f32), [a, b, acc, 0, 0, 0])
    )

peer_row(c, q)

Peer row q (0 .. 3G - 1) in owner order rank + 1, rank + 2, rank + 3.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def peer_row(c, q):
    """Peer row q (0 .. 3G - 1) in owner order rank + 1, rank + 2, rank + 3."""
    g_rows = c["G"]
    return ((c["rank"] + 1 + q // g_rows) % TP) * g_rows + q % g_rows

policy(rows)

The options a width builds with when its GlueBuild leaves them None (measured, MI355X TP4). Options off: rowvec (+0..0.8 us), agdefer (+1.5..2.5: its RS finishes 7 us earlier but the deferred all-gather then blocks the owner rows' local stores), rnorm (+1..2.5: its local hand-offs queue behind the all-gather's remote stores), ownfirst (unmeasured), prepoll outside PREPOLL_ROWS.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def policy(rows: int) -> dict[str, bool]:
    """The options a width builds with when its GlueBuild leaves them None (measured,
    MI355X TP4). Options off: rowvec (+0..0.8 us), agdefer (+1.5..2.5: its RS finishes 7
    us earlier but the deferred all-gather then blocks the owner rows' local stores),
    rnorm (+1..2.5: its local hand-offs queue behind the all-gather's remote stores),
    ownfirst (unmeasured), prepoll outside PREPOLL_ROWS."""
    pol = {f: f in POLICY_ON for f in OPT_FLAGS}
    pol["prepoll"] = PREPOLL_ROWS[0] <= rows <= PREPOLL_ROWS[1]
    return pol

prepoll(c, addr, idx, site, triples=True)

One lane a wave spins on one pair before the CTA's full poll (a hint only: the full poll checks every tag).

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def prepoll(c, addr, idx, site, triples=True):
    """One lane a wave spins on one pair before the CTA's full poll (a hint only: the
    full poll checks every tag)."""
    if const_expr(c["prepoll"]):  # noqa: SIM102 (compile-time branch)
        if c["lane"] == 0:
            c["mb"].poll([(addr, idx, 1)], site=site, triples=triples)

resolve(key)

key with every None option set by policy.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def resolve(key: GlueBuild) -> GlueBuild:
    """`key` with every None option set by `policy`."""
    from dataclasses import replace

    pol = policy(key.rows)
    of_free = key.ownfirst is None
    key = replace(key, **{f: pol[f] for f in OPT_FLAGS if getattr(key, f) is None})  # type: ignore[arg-type]
    if key.transport is None:
        key = replace(key, transport=TRANSPORT_DEFAULT)
    if key.transport == "lamport_rs":  # alias: the RS untagged, the AG tagged
        key = replace(key, transport="lamport", lpag=False)
    if key.lpag is None or key.transport == "tagged":
        key = replace(
            key, lpag=key.transport == "lamport" and key.rows >= LP_AG_MIN_ROWS
        )
    if key.lpmix is None:
        key = replace(key, lpmix=LP_MIX_DEFAULT if key.mixdst else 0)
    if (
        of_free and key.lpag
        # lamport AG: this rank's own inbox first (its owner rows start the routing
        # chain)
    ):
        key = replace(key, ownfirst=True)
    assert key.transport in TRANSPORTS, key.transport
    if key.transport != "tagged":
        bad = [f for f in ("rnorm", "agdefer") if getattr(key, f)]
        if key.lpag and key.rowvec:
            bad.append("rowvec")
        assert not bad, f"transport {key.transport} does not support {bad}"
    return key

router_weights(c, task)

The task's FP32 weights (a lane's float4 x 3 chunks), issued at kernel entry. rnorm: a lane group's 12 consecutive columns (4 triples: the own AG slot's granularity) instead of 3 x 4 columns 16 apart.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def router_weights(c, task):
    """The task's FP32 weights (a lane's float4 x 3 chunks), issued at kernel entry.
    rnorm: a lane group's 12 consecutive columns (4 triples: the own AG slot's
    granularity) instead of 3 x 4 columns 16 apart."""
    lane, wave, a = c["lane"], c["wave"], c["args"]
    tile, part = task // ROUTER_PARTS, task % ROUTER_PARTS
    e = tile * 16 + lane % 16
    chunks = ROUTER_K // WAVES // 16
    if const_expr(c["rnorm"]):
        k0 = part * ROUTER_K + wave * (ROUTER_K // WAVES) + (lane // 16) * 12
        return [
            fx.Vector(
                buf_load(
                    a["router_w"], e * HIDDEN + k0 + 4 * ch, vec_width=4, dtype=T.f32
                )
            )
            for ch in range(chunks)
        ]
    k0 = part * ROUTER_K + wave * (ROUTER_K // WAVES) + (lane // 16) * 4
    return [
        fx.Vector(
            buf_load(a["router_w"], e * HIDDEN + k0 + 16 * ch, vec_width=4, dtype=T.f32)
        )
        for ch in range(chunks)
    ]

rs_done_wait(c)

Every seam CTA of this launch has reduced: one add each, then wait for the launch's multiple of SEAM_TASKS.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def rs_done_wait(c):
    """Every seam CTA of this launch has reduced: one add each, then wait for the
    launch's multiple of SEAM_TASKS."""
    epoch = c["epoch"]
    cta_sync()
    if c["tid"] == 0:
        ptr = fx.inttoptr(
            fx.PointerType.get(T.i32, fx.AddressSpace.Global, 4),
            epoch + fx.Int64(4 * RS_DONE_WORD),
        )
        old = fx.llvm.atomic_add(ptr, fx.Int32(1), syncscope="agent")
        target = (old // SEAM_TASKS + 1) * SEAM_TASKS
        t0 = memrealtime()
        n = load32(epoch, RS_DONE_WORD, "agent")
        late = fx.Int64(0)
        while (n < target) & (late < c["mb"].ticks):
            rocdl.s_sleep(1)
            n = load32(epoch, RS_DONE_WORD, "agent")
            late = memrealtime() - t0
        if n < target:
            c["mb"].fail(SITE_RS)
    cta_sync()

sort_scan_wave(c, padded)

Exclusive scan of the padded counts by wave 0 (two experts a lane, ds_bpermute Hillis-Steele) -> off[0 .. E], num_valid. fill was zeroed at the start.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def sort_scan_wave(c, padded):
    """Exclusive scan of the padded counts by wave 0 (two experts a lane, ds_bpermute
    Hillis-Steele) -> off[0 .. E], num_valid. `fill` was zeroed at the start."""
    s, tid, a, cnt, off = c["S"], c["tid"], c["args"], c["cnt"], c["off"]
    if tid < 64:
        p0 = padded(fx.ptr_load(cnt + 2 * tid))
        p1 = padded(fx.ptr_load(cnt + 2 * tid + 1))
        v = p0 + p1
        for d in range_constexpr(6):
            sh = 1 << d
            got = lane_gather(v, fx.max(tid - sh, fx.Int32(0)))
            v = v + (tid >= sh).select(got, fx.Int32(0))
        ex = v - p0 - p1
        fx.ptr_store(ex, off + 2 * tid)
        fx.ptr_store(ex + p0, off + 2 * tid + 1)
        if tid == 63:
            fx.ptr_store(v, off + EXPERTS)
            buf_store(v, a["num_valid"], 0)
            buf_store(fx.Int32(s), a["num_valid"], 1)
    gpu.barrier()

split3(w4)

A float4 -> three 64-bit words of 4 bf16 (hi, mid, lo) with w = hi + mid + lo exactly (truncations).

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def split3(w4):
    """A float4 -> three 64-bit words of 4 bf16 (hi, mid, lo) with w = hi + mid + lo
    exactly (truncations)."""
    outs: list[list] = [[], [], []]
    for e in range(4):
        r = w4[e]
        for lvl in range(3):
            b = r.bitcast(fx.Int32) & fx.Int32(-65536)
            outs[lvl].append(b)
            if lvl < 2:
                r = r - b.bitcast(fx.Float32)
    words = []
    for lvl in range(3):
        b = outs[lvl]
        lo = fx.Uint32((b[0] >> 16) & 0xFFFF) | fx.Uint32(b[1])
        hi = fx.Uint32((b[2] >> 16) & 0xFFFF) | fx.Uint32(b[3])
        words.append(fx.Int64(lo) | (fx.Int64(hi) << 32))
    return words

stage_route(c, t)

Row t: top-4 of sigmoid + bias (ties to the lower expert); weights = selected sigmoids x (2.0 / their sum).

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def stage_route(c, t):
    """Row t: top-4 of sigmoid + bias (ties to the lower expert); weights = selected
    sigmoids x (2.0 / their sum)."""
    tid, lane, wave, red, a = c["tid"], c["lane"], c["wave"], c["red"], c["args"]
    quarter = ROUTER_PARTS // 4
    e = tid % EXPERTS
    q = tid // EXPERTS
    # the bias before the poll (a global load round trip off the critical path)
    bias = [
        fx.Float32(buf_load(a["bias"], lane + 64 * i, vec_width=1, dtype=T.f32))
        for i in range(2)
    ]
    got = c["mb"].poll(
        [
            (c["logit"], ((q * quarter + p) * c["RR"] + t) * EXPERTS + e, 1)
            for p in range(quarter)
        ],
        site=SITE_ROUTE,
    )
    c["mb"].acquire(Space.SCRATCH)
    stamp(c, "rt_land")
    part = fx.Float32(0.0)
    for p in range_constexpr(quarter):
        part = part + got[p][0].bitcast(fx.Float32)
    fx.ptr_store(part, red + tid)
    gpu.barrier()
    if wave == 0:
        logits = []
        for i in range_constexpr(2):
            tot = fx.Float32(0.0)
            for qq in range_constexpr(4):
                tot = tot + fx.ptr_load(red + (qq * EXPERTS + lane + 64 * i))
            logits.append(tot)
        sc = [_sigmoid(logits[i]) for i in range(2)]
        neg = fx.Float32(float("-inf"))
        val = [sc[i] + bias[i] for i in range(2)]
        val = [fx.isnan(v).select(neg, v) for v in val]
        sc = [fx.isnan(v).select(fx.Float32(0.0), v) for v in sc]
        taken = [fx.Int32(0), fx.Int32(0)]
        total = fx.Float32(0.0)
        my_id, my_w = fx.Int32(0), fx.Float32(0.0)
        for k in range_constexpr(TOPK):
            v0 = (taken[0] == 0).select(val[0], neg)
            v1 = (taken[1] == 0).select(val[1], neg)
            pick1 = v1 > v0
            v = pick1.select(v1, v0)
            idx = pick1.select(fx.Int32(lane + 64), fx.Int32(lane))
            best = butterfly(v, (32, 16, 8, 4, 2, 1), fx.max)
            cand = (v == best).select(idx, fx.Int32(1 << 20))
            win = butterfly(cand, (32, 16, 8, 4, 2, 1), fx.min)
            mine0 = win == lane
            mine1 = win == lane + 64
            w_mine = mine0.select(sc[0], mine1.select(sc[1], fx.Float32(0.0)))
            w = butterfly(w_mine, (32, 16, 8, 4, 2, 1), fx.max)
            taken[0] = mine0.select(fx.Int32(1), taken[0])
            taken[1] = mine1.select(fx.Int32(1), taken[1])
            total = total + w
            my_id = (lane == k).select(win, my_id)
            my_w = (lane == k).select(w, my_w)
        scale = fx.Float32(ROUTED_SCALE) / fx.max(total, fx.Float32(1e-20))
        if const_expr(c["orouter"]):
            if (
                lane < TOPK
            ):  # row t of this rank's owner rows -> every rank's top-k exchange
                tg = c["row0"] + t
                for p in range_constexpr(TP):
                    c["mb"].put(c["peer"][p], ROUTE_P0 + tg * 2 * TOPK + lane, my_id)
                    c["mb"].put(
                        c["peer"][p],
                        ROUTE_P0 + tg * 2 * TOPK + TOPK + lane,
                        my_w * scale,
                    )
        elif lane < TOPK:
            buf_store(my_id, a["topk_ids"], t * TOPK + lane)
            buf_store(my_w * scale, a["topk_w"], t * TOPK + lane)
            c["mb"].put(c["route"], t * 2 * TOPK + lane, my_id)
            c["mb"].put(c["route"], t * 2 * TOPK + TOPK + lane, my_w * scale)

stage_router(c, task, wv)

Experts 16 tile .. over K part part for every row (16-row tiles): f32 MFMAs, waves summed in order -> LOGIT[row][expert][part]. One wait for every row's XRDY, then every tile's X loads in flight together.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def stage_router(c, task, wv):
    """Experts 16 tile .. over K part `part` for every row (16-row tiles): f32 MFMAs,
    waves summed in order -> `LOGIT[row][expert][part]`. One wait for every row's XRDY,
    then every tile's X loads in flight together."""
    s, tid, lane, wave, red = (
        c["RR"],
        c["tid"],
        c["lane"],
        c["wave"],
        c["red"],
    )  # s: the router's rows
    row0 = c["row0"]  # its first row (orouter: this rank's owner rows)
    tile, part = task // ROUTER_PARTS, task % ROUTER_PARTS
    k0 = part * ROUTER_K + wave * (ROUTER_K // WAVES) + (lane // 16) * 4
    chunks = ROUTER_K // WAVES // 16
    ntiles = -(-s // 16)
    if tid < s:
        c["mb"].poll([(c["xrdy"], row0 + tid, 1)], site=SITE_ROUTER)
        c["mb"].acquire(Space.SCRATCH)
    cta_sync()
    stamp(c, "r_rows")
    xs = []
    for rt in range_constexpr(ntiles):
        n = min(16, s - rt * 16)
        tok = row0 + rt * 16 + fx.min(lane % 16, n - 1)
        xs.append(
            [
                load_dev(c["args"]["xn"], (tok * HIDDEN + k0 + 16 * ch) // 2, 2)
                for ch in range(chunks)
            ]
        )
    if const_expr(c["rbf16"]):
        w3 = [split3(wv[ch]) for ch in range(chunks)]
    for rt in range_constexpr(ntiles):
        acc = fx.Vector.filled(4, 0.0, fx.Float32)
        for ch in range_constexpr(chunks):
            if const_expr(c["rbf16"]):
                x64 = fx.Int64(fx.Uint32(fx.Int32(xs[rt][ch][0]))) | (
                    fx.Int64(fx.Uint32(fx.Int32(xs[rt][ch][1]))) << 32
                )
                for lvl in range_constexpr(3):
                    acc = mfma_bf16(w3[ch][lvl], x64, acc)
            else:
                for h in range_constexpr(2):
                    lo, hi = bf2_f32(fx.Int32(xs[rt][ch][h]))
                    acc = mfma_f32(wv[ch][2 * h], lo, acc)
                    acc = mfma_f32(wv[ch][2 * h + 1], hi, acc)
        fx.ptr_store(acc, red + ((rt * WAVES + wave) * 64 + lane) * 4)
    gpu.barrier()
    stamp(c, "r_mfma")
    for rt in range_constexpr(ntiles):
        n = min(16, s - rt * 16)
        if tid < 16 * n:
            r = tid % 16
            t = tid // 16
            at = (16 * (r // 4) + t) * 4 + r % 4
            tot = fx.Float32(0.0)
            for w in range_constexpr(WAVES):
                tot = tot + fx.ptr_load(red + ((rt * WAVES + w) * 64 * 4 + at))
            c["mb"].put(
                c["logit"], (part * s + rt * 16 + t) * EXPERTS + tile * 16 + r, tot
            )

stage_router_rn(c, task, wv)

stage_router on the owner rows with its own normalization (rnorm): rstd from the seam CTAs' SSQ partials (a row's 128 in a fixed order), v = own AG slot (h) + residual for the lane group's 12 columns, x = bf16(v * rstd * (1 + w)) as the norm CTAs compute it (rstd's summation order differs: a rare 1-ulp x difference).

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def stage_router_rn(c, task, wv):
    """`stage_router` on the owner rows with its own normalization (rnorm): rstd from
    the seam CTAs' SSQ partials (a row's 128 in a fixed order), v = own AG slot (h) +
    residual for the lane group's 12 columns, x = bf16(v * rstd * (1 + w)) as the norm
    CTAs compute it (rstd's summation order differs: a rare 1-ulp x difference)."""
    s, tid, lane, wave, red, a = (
        c["RR"],
        c["tid"],
        c["lane"],
        c["wave"],
        c["red"],
        c["args"],
    )
    rank, row0 = c["rank"], c["row0"]
    tile, part = task // ROUTER_PARTS, task % ROUTER_PARTS
    tr0 = (
        part * (ROUTER_K // 3) + wave * (ROUTER_K // WAVES // 3) + (lane // 16) * 4
    )  # the lane group's 4 triples
    ntiles = -(-s // 16)
    # loads before the polls: residual and norm weight triples of the lane's columns
    res, rls = [], []
    for rt in range_constexpr(ntiles):
        n = min(16, s - rt * 16)
        rl = rt * 16 + fx.min(lane % 16, n - 1)
        rls.append(rl)
        res.append([triple_bits(a["res_in"], row0 + rl, tr0 + j) for j in range(4)])
    nw = [triple_bits(a["norm_w"], 0, tr0 + j) for j in range(4)]
    # every owner row's 128 SSQ partials (rows past G poll row G - 1 again)
    its = -(-(s * SEAM_TASKS) // THREADS)
    specs = []
    for it in range_constexpr(its):
        idx = fx.min(it * THREADS + tid, s * SEAM_TASKS - 1)
        specs.append((c["ssq"], idx, 1))
    a0, i0, _ = specs[-1]
    prepoll(
        c, a0, i0, SITE_ROUTER, triples=False
    )  # prepoll builds: a lane a wave on one pair first
    got = c["mb"].poll(specs, batch=16, site=SITE_ROUTER)
    for it in range_constexpr(its):
        idx = it * THREADS + tid
        if idx < s * SEAM_TASKS:
            fx.ptr_store(got[it][0].bitcast(fx.Float32), red + SSQ_LDS + idx)
    gpu.barrier()
    for rr in range_constexpr(
        -(-s // WAVES)
    ):  # wave w: rows w, w + 8, ...: 2 partials a lane, butterfly
        row = rr * WAVES + wave
        if row < s:
            p2 = fx.ptr_load(red + SSQ_LDS + row * SEAM_TASKS + 2 * lane) + fx.ptr_load(
                red + SSQ_LDS + row * SEAM_TASKS + 2 * lane + 1
            )
            tot = butterfly(p2, (1, 2, 4, 8, 16, 32))
            if lane == 0:
                var = tot / fx.Float32(HIDDEN)
                fx.ptr_store(
                    fx.Float32(1.0) / fx.Float32(fmath.sqrt(var + fx.Float32(EPS))),
                    red + RSTD_LDS + row,
                )
    hs = c["mb"].poll(
        [
            (c["own"], ag_pair(rank, rls[rt], tr0 + j), 1)
            for rt in range(ntiles)
            for j in range(4)
        ],
        batch=16,
        site=SITE_ROUTER,
        triples=True,
    )
    gpu.barrier()
    stamp(c, "r_rows")
    for rt in range_constexpr(ntiles):
        rstd = fx.ptr_load(red + RSTD_LDS + rls[rt])
        xs = []
        for j in range_constexpr(4):
            for k in range_constexpr(3):
                hv = triple_value(hs[rt * 4 + j][0][0], hs[rt * 4 + j][0][1], k)
                v = c["sharded"].select(
                    hv, hv + bf16_bits_f32(fx.Int32(res[rt][j] >> (16 * k)))
                )
                w = bf16_bits_f32(fx.Int32(nw[j] >> (16 * k)))
                xs.append(v * rstd * (fx.Float32(1.0) + w))
        acc = fx.Vector.filled(4, 0.0, fx.Float32)
        if const_expr(c["rbf16"]):
            for ch in range_constexpr(3):
                lo = bf16_pair(xs[4 * ch], xs[4 * ch + 1]).bitcast(fx.Int32)
                hi = bf16_pair(xs[4 * ch + 2], xs[4 * ch + 3]).bitcast(fx.Int32)
                x64 = fx.Int64(fx.Uint32(lo)) | (fx.Int64(fx.Uint32(hi)) << 32)
                for w3 in split3(wv[ch]):
                    acc = mfma_bf16(w3, x64, acc)
        else:
            for m in range_constexpr(12):
                acc = mfma_f32(wv[m // 4][m % 4], bf16_round(xs[m]), acc)
        fx.ptr_store(acc, red + ((rt * WAVES + wave) * 64 + lane) * 4)
    gpu.barrier()
    stamp(c, "r_mfma")
    for rt in range_constexpr(ntiles):
        n = min(16, s - rt * 16)
        if tid < 16 * n:
            r = tid % 16
            t = tid // 16
            at = (16 * (r // 4) + t) * 4 + r % 4
            tot = fx.Float32(0.0)
            for w in range_constexpr(WAVES):
                tot = tot + fx.ptr_load(red + ((rt * WAVES + w) * 64 * 4 + at))
            c["mb"].put(
                c["logit"], (part * s + rt * 16 + t) * EXPERTS + tile * 16 + r, tot
            )

stage_row(c, t)

Row t: its owner's gathered h + residual (f32) -> residual out (bf16); Gemma RMSNorm -> normed (bf16), XRDY.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def stage_row(c, t):
    """Row t: its owner's gathered h + residual (f32) -> residual out (bf16); Gemma
    RMSNorm -> normed (bf16), XRDY."""
    g_rows, tid, a = c["G"], c["tid"], c["args"]
    owner = t // g_rows
    trs = [tid + j * THREADS for j in range(ROW_TRIPLES)]
    res = [triple_bits(a["res_in"], t, tr) for tr in trs]  # loads before the poll
    nw = [triple_bits(a["norm_w"], 0, tr) for tr in trs]
    if const_expr(c["lp_ag"]):
        got = lp_row_ag(c, t, trs)
    else:
        prepoll(c, c["own"], ag_pair(owner, t % g_rows, trs[0]), SITE_AG)
        got = c["mb"].poll(
            [(c["own"], ag_pair(owner, t % g_rows, tr), 1) for tr in trs],
            site=SITE_AG,
            triples=True,
        )
    stamp(c, "ag_land")
    vals = []
    sq = fx.Float32(0.0)
    for j in range_constexpr(ROW_TRIPLES):
        for k in range_constexpr(3):
            hv = triple_value(got[j][0][0], got[j][0][1], k)
            # mode 2: the gathered value is already bf16(h + residual)
            v = c["sharded"].select(
                hv, hv + bf16_bits_f32(fx.Int32(res[j] >> (16 * k)))
            )
            if const_expr(not c["xfirst"]):
                buf_store(v.to(fx.BFloat16), a["res_out"], t * HIDDEN + 3 * trs[j] + k)
            vals.append(v)
            sq = sq + v * v
    var = block_sum(c, sq, 0) / fx.Float32(HIDDEN)
    rstd = fx.Float32(1.0) / fx.Float32(fmath.sqrt(var + fx.Float32(EPS)))
    for j in range_constexpr(ROW_TRIPLES):
        for k in range_constexpr(3):
            w = bf16_bits_f32(fx.Int32(nw[j] >> (16 * k)))
            o = vals[3 * j + k] * rstd * (fx.Float32(1.0) + w)
            store16(a["xn"], t * HIDDEN + 3 * trs[j] + k, o.to(fx.BFloat16), "agent")
    if const_expr(c["xfirst"]):
        cta_sync(drain=True)
        if tid == 0:
            c["mb"].publish(c["xrdy"], t, fx.Int32(1))
        for j in range_constexpr(ROW_TRIPLES):
            for k in range_constexpr(3):
                buf_store(
                    vals[3 * j + k].to(fx.BFloat16),
                    a["res_out"],
                    t * HIDDEN + 3 * trs[j] + k,
                )
    # MXFP8 copy of the normed row (the shared expert's gate_up input), vLLM's
    # fused-norm math on bf16(normed): per 32 columns sb = clamp(ceil(log2(amax / 448))
    # + 127, 0, 254), q = y * 2^(127 - sb) as e4m3.
    xrow = c["red"] + XROW_OFF
    for j in range_constexpr(ROW_TRIPLES):
        for k in range_constexpr(3):
            w = bf16_bits_f32(fx.Int32(nw[j] >> (16 * k)))
            o = vals[3 * j + k] * rstd * (fx.Float32(1.0) + w)
            fx.ptr_store(bf16_round(o), xrow + 3 * trs[j] + k)
    gpu.barrier()
    if tid < GROUPS:
        ys = [fx.ptr_load(xrow + tid * 32 + i) for i in range(32)]
        amax = fx.Float32(F32_TINY)
        for y in ys:
            amax = fx.max(amax, fx.Float32(fmath.absf(y)))
        bits = (amax / fx.Float32(FP8_MAX)).bitcast(fx.Int32)
        # ceil(log2 r) of a normal r: its exponent, + 1 unless the mantissa is 0
        k_exp = (
            ((bits >> 23) & 0xFF)
            - 127
            + ((bits & 0x7FFFFF) != 0).select(fx.Int32(1), fx.Int32(0))
        )
        sb = fx.min(fx.max(k_exp + 127, fx.Int32(0)), fx.Int32(254))
        inv = ((254 - sb) << 23).bitcast(fx.Float32)
        words = [
            fp8_pack4(
                ys[4 * w] * inv,
                ys[4 * w + 1] * inv,
                ys[4 * w + 2] * inv,
                ys[4 * w + 3] * inv,
            )
            for w in range(8)
        ]
        for h in range_constexpr(2):
            buf_store(
                fx.Vector.from_elements(words[4 * h : 4 * h + 4], fx.Int32),
                a["xq"],
                (t * HIDDEN + tid * 32) // 4 + 4 * h,
            )
        fx.ptr_store(sb, c["scl"] + tid)
    gpu.barrier()
    if tid < GROUPS // 4:
        sw = [fx.ptr_load(c["scl"] + 4 * tid + i) for i in range(4)]
        buf_store(
            sw[0] | (sw[1] << 8) | (sw[2] << 16) | (sw[3] << 24),
            a["xsc"],
            t * (GROUPS // 4) + tid,
        )
    if const_expr(not c["xfirst"]):
        cta_sync(drain=True)
        if tid == 0:
            c["mb"].publish(c["xrdy"], t, fx.Int32(1))
    stamp(c, "norm")

stage_row_vec(c, t)

stage_row with vector stores: the f32 sums go through LDS; each thread then owns 12 consecutive columns (24 bytes: res_out and normed as 64-bit stores). Same math and bits.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def stage_row_vec(c, t):
    """`stage_row` with vector stores: the f32 sums go through LDS; each thread then
    owns 12 consecutive columns (24 bytes: res_out and normed as 64-bit stores). Same
    math and bits."""
    g_rows, tid, a = c["G"], c["tid"], c["args"]
    owner = t // g_rows
    trs = [tid + j * THREADS for j in range(ROW_TRIPLES)]
    res = [triple_bits(a["res_in"], t, tr) for tr in trs]  # loads before the poll
    c0 = tid * (3 * ROW_TRIPLES)
    nw = [
        fx.Vector(buf_load(a["norm_w"], c0 // 2 + 2 * i, vec_width=2, dtype=T.i32))
        for i in range(3)
    ]
    prepoll(c, c["own"], ag_pair(owner, t % g_rows, trs[0]), SITE_AG)
    got = c["mb"].poll(
        [(c["own"], ag_pair(owner, t % g_rows, tr), 1) for tr in trs],
        site=SITE_AG,
        triples=True,
    )
    stamp(c, "ag_land")
    xrow = c["red"] + XROW_OFF
    sq = fx.Float32(0.0)
    for j in range_constexpr(ROW_TRIPLES):
        for k in range_constexpr(3):
            hv = triple_value(got[j][0][0], got[j][0][1], k)
            v = c["sharded"].select(
                hv, hv + bf16_bits_f32(fx.Int32(res[j] >> (16 * k)))
            )
            fx.ptr_store(v, xrow + _pad(3 * trs[j] + k))
            sq = sq + v * v
    var = block_sum(c, sq, 0) / fx.Float32(
        HIDDEN
    )  # its barrier also publishes the LDS row
    rstd = fx.Float32(1.0) / fx.Float32(fmath.sqrt(var + fx.Float32(EPS)))
    vs = [fx.ptr_load(xrow + _pad(c0 + i)) for i in range(3 * ROW_TRIPLES)]
    ws = []
    for i in range_constexpr(3):
        for h in range_constexpr(2):
            lo, hi = bf2_f32(fx.Int32(nw[i][h]))
            ws += [lo, hi]
    os_ = [vs[i] * rstd * (fx.Float32(1.0) + ws[i]) for i in range(3 * ROW_TRIPLES)]
    rw = [bf16_pair(vs[2 * i], vs[2 * i + 1]).bitcast(fx.Int32) for i in range(6)]
    ow = [bf16_pair(os_[2 * i], os_[2 * i + 1]).bitcast(fx.Int32) for i in range(6)]
    for i in range_constexpr(3):
        buf_store(
            fx.Vector.from_elements(rw[2 * i : 2 * i + 2], fx.Int32),
            a["res_out"],
            (t * HIDDEN + c0) // 2 + 2 * i,
        )
        store64(
            a["xn"] + fx.Int64(t * HIDDEN + c0) * 2,
            i,
            (fx.Int64(fx.Uint32(ow[2 * i + 1])) << 32) | fx.Int64(fx.Uint32(ow[2 * i])),
            "agent",
        )
    for i in range_constexpr(3 * ROW_TRIPLES):
        fx.ptr_store(bf16_round(os_[i]), xrow + _pad(c0 + i))
    gpu.barrier()
    if tid < GROUPS:
        ys = [fx.ptr_load(xrow + tid * 33 + i) for i in range(32)]
        amax = fx.Float32(F32_TINY)
        for y in ys:
            amax = fx.max(amax, fx.Float32(fmath.absf(y)))
        bits = (amax / fx.Float32(FP8_MAX)).bitcast(fx.Int32)
        k_exp = (
            ((bits >> 23) & 0xFF)
            - 127
            + ((bits & 0x7FFFFF) != 0).select(fx.Int32(1), fx.Int32(0))
        )
        sb = fx.min(fx.max(k_exp + 127, fx.Int32(0)), fx.Int32(254))
        inv = ((254 - sb) << 23).bitcast(fx.Float32)
        words = [
            fp8_pack4(
                ys[4 * w] * inv,
                ys[4 * w + 1] * inv,
                ys[4 * w + 2] * inv,
                ys[4 * w + 3] * inv,
            )
            for w in range(8)
        ]
        for h in range_constexpr(2):
            buf_store(
                fx.Vector.from_elements(words[4 * h : 4 * h + 4], fx.Int32),
                a["xq"],
                (t * HIDDEN + tid * 32) // 4 + 4 * h,
            )
        fx.ptr_store(sb, c["scl"] + tid)
    gpu.barrier()
    if tid < GROUPS // 4:
        sw = [fx.ptr_load(c["scl"] + 4 * tid + i) for i in range(4)]
        buf_store(
            sw[0] | (sw[1] << 8) | (sw[2] << 16) | (sw[3] << 24),
            a["xsc"],
            t * (GROUPS // 4) + tid,
        )
    cta_sync(drain=True)
    if tid == 0:
        c["mb"].publish(c["xrdy"], t, fx.Int32(1))
    stamp(c, "norm")

stage_rs_push(c, task)

Triples 16 task ..: every row of this rank's partial to its owner's RS slot [this rank]; peers' rows first.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def stage_rs_push(c, task):
    """Triples 16 task ..: every row of this rank's partial to its owner's RS slot [this
    rank]; peers' rows first."""
    g_rows, rank, tid = c["G"], c["rank"], c["tid"]
    tr = task * TASK_TRIPLES + tid % TASK_TRIPLES
    loads = []  # every row's load in flight before the first store
    for d in range_constexpr(1, TP + 1):
        g = (rank + d) % TP  # the three peers first, this rank last
        base = c["peer"][0].value
        for p in range_constexpr(1, TP):
            base = sel(g == p, c["peer"][p].value, base)
        for it in range_constexpr(-(-g_rows // ROW_LANES)):
            r = it * ROW_LANES + tid // TASK_TRIPLES
            rr = fx.min(r, g_rows - 1)
            loads.append((base, r, triple_bits(c["args"]["part"], g * g_rows + rr, tr)))
    if const_expr(c["mixdst"]):
        # store j of a lane group q goes to destination (q + j) % 4 of the four: each
        # instruction feeds every link
        q = c["lane"] // 16
        passes = -(-g_rows // ROW_LANES)
        for it in range_constexpr(passes):
            r = it * ROW_LANES + tid // TASK_TRIPLES
            if r < g_rows:
                for j in range_constexpr(TP):
                    dsel = (q + j) % TP
                    base, _, bits = loads[it]
                    for k in range_constexpr(1, TP):
                        bk, _, vk = loads[k * passes + it]
                        base = sel(dsel == k, bk, base)
                        bits = sel(dsel == k, vk, bits)
                    push(c, Addr("peer", Space.PEER, base), rs_pair(rank, r, tr), bits)
    else:
        for base, r, bits in loads:
            if r < g_rows:
                push(c, Addr("peer", Space.PEER, base), rs_pair(rank, r, tr), bits)

stage_rs_push_lp(c, task)

This rank's partial, owner g's rows, the task's 48 columns -> rank g's RS inbox [this rank][task] as plain bf16 (lane = (row, 16-byte vector); every load before the first store); the 3 peers only: the owner reads its own partial from part. mixdst: the lane's j-th store goes to peer (j + row) % 3 (after this rank), so one store instruction feeds all three links.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def stage_rs_push_lp(c, task):
    """This rank's partial, owner g's rows, the task's 48 columns -> rank g's RS inbox
    `[this rank][task]` as plain bf16 (lane = (row, 16-byte vector); every load before
    the first store); the 3 peers only: the owner reads its own partial from `part`.
    mixdst: the lane's j-th store goes to peer (j + row) % 3 (after this rank), so one
    store instruction feeds all three links."""
    g_rows, rank, tid, a = c["G"], c["rank"], c["tid"], c["args"]
    r = tid // LP_VEC
    q = tid % LP_VEC
    rr = fx.min(r, g_rows - 1)
    col = task * 48 + 8 * q
    vals = []
    for d in range_constexpr(1, TP):
        g = (rank + d) % TP
        vals.append(_ld16(a["part"], ((g * g_rows + rr) * HIDDEN + col) // 2))
    if r < g_rows:
        for j in range_constexpr(TP - 1):
            k = (j + r // c["lpmix"]) % (TP - 1) if c["lpmix"] else j
            ws = list(vals[0])
            for kk in range_constexpr(1, TP - 1):
                ws = [sel(k == kk, vals[kk][e], ws[e]) for e in range(4)]
            p = (rank + 1 + k) % TP
            lp_st16(
                lp_seg(c, lp_peer(c, p), LP_RS, rank, task, r)
                + fx.Int64(16) * fx.Int64(q),
                lp_canon(ws),
            )
    if const_expr(c["lpack"]):
        lp_count_acked(c)

stage_rs_reduce(c, task)

Triples 16 task .. of this rank's G rows: the 4 ranks' triples in rank order (f32) -> bf16 h -> every rank's AG slot [this rank].

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def stage_rs_reduce(c, task):
    """Triples 16 task .. of this rank's G rows: the 4 ranks' triples in rank order
    (f32) -> bf16 h -> every rank's AG slot [this rank]."""
    g_rows, rank, tid = c["G"], c["rank"], c["tid"]
    tr = task * TASK_TRIPLES + tid % TASK_TRIPLES
    for it in range_constexpr(-(-g_rows // ROW_LANES)):
        r = it * ROW_LANES + tid // TASK_TRIPLES
        if r < g_rows:
            rb = triple_bits(
                c["args"]["res_in"], rank * g_rows + r, tr
            )  # mode 2: this rank's residual rows
            prepoll(c, c["own"], rs_pair((rank + 1) % TP, r, tr), SITE_RS)
            got = c["mb"].poll(
                [(c["own"], rs_pair(src, r, tr), 1) for src in range(TP)],
                site=SITE_RS,
                triples=True,
            )
            if const_expr(it == 0):
                stamp(c, "rs_land")
            bits = fx.Int64(0)
            for k in range_constexpr(3):
                acc = triple_value(got[0][0][0], got[0][0][1], k)
                for src in range_constexpr(1, TP):
                    acc = acc + triple_value(got[src][0][0], got[src][0][1], k)
                s_k = bf16_round(acc) + bf16_bits_f32(
                    fx.Int32(rb >> (16 * k))
                )  # the stock add of bf16 h
                bits = bits | (
                    c["sharded"].select(bf16_bits(s_k), bf16_bits(acc)) << (16 * k)
                )
            ag_push_all(c, ag_pair(rank, r, tr), bits)

stage_rs_reduce_batched(c, task)

stage_rs_reduce with every pass's poll in one batch (rows past G poll row G - 1 again).

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def stage_rs_reduce_batched(c, task):
    """`stage_rs_reduce` with every pass's poll in one batch (rows past G poll row G - 1
    again)."""
    g_rows, rank, tid = c["G"], c["rank"], c["tid"]
    tr = task * TASK_TRIPLES + tid % TASK_TRIPLES
    passes = -(-g_rows // ROW_LANES)
    rs = [
        fx.min(it * ROW_LANES + tid // TASK_TRIPLES, g_rows - 1) for it in range(passes)
    ]
    rbs = [triple_bits(c["args"]["res_in"], rank * g_rows + r, tr) for r in rs]
    prepoll(
        c, c["own"], rs_pair((rank + 1) % TP, rs[0], tr), SITE_RS
    )  # (rank + 1) pushes to this owner last
    got = c["mb"].poll(
        [(c["own"], rs_pair(src, r, tr), 1) for r in rs for src in range(TP)],
        site=SITE_RS,
        triples=True,
    )
    stamp(c, "rs_land")
    if const_expr(c["rnorm"]):
        hs = [_rs_sum(c, got[it * TP : (it + 1) * TP], rbs[it]) for it in range(passes)]
        for it in range_constexpr(
            passes
        ):  # this rank's AG slot first (the owner rows' input, local)
            r = it * ROW_LANES + tid // TASK_TRIPLES
            if r < g_rows:
                push(c, c["own"], ag_pair(rank, r, tr), hs[it])
        for it in range_constexpr(
            passes
        ):  # the row's partial sum of squares over this task's 48 columns
            r = it * ROW_LANES + tid // TASK_TRIPLES
            sq = fx.Float32(0.0)
            for k in range_constexpr(3):
                hv = bf16_bits_f32(fx.Int32(hs[it] >> (16 * k)))
                v = c["sharded"].select(
                    hv, hv + bf16_bits_f32(fx.Int32(rbs[it] >> (16 * k)))
                )
                sq = sq + v * v
            sq = butterfly(sq, (8, 4, 2, 1))
            if (r < g_rows) & (tid % TASK_TRIPLES == 0):
                c["mb"].put(c["ssq"], r * SEAM_TASKS + task, sq)
        for it in range_constexpr(passes):
            r = it * ROW_LANES + tid // TASK_TRIPLES
            if r < g_rows:
                ag_push_peers(c, ag_pair(rank, r, tr), hs[it])
    elif const_expr(c["agdefer"]):
        hs = []
        for it in range_constexpr(passes):
            r = it * ROW_LANES + tid // TASK_TRIPLES
            hs.append(_rs_sum(c, got[it * TP : (it + 1) * TP], rbs[it]))
            if r < g_rows:
                push(
                    c, c["own"], ag_pair(rank, r, tr), hs[it]
                )  # own rows: this rank's AG slot now
        rs_done_wait(c)
        for it in range_constexpr(passes):
            r = it * ROW_LANES + tid // TASK_TRIPLES
            if r < g_rows:
                ag_push_peers(c, ag_pair(rank, r, tr), hs[it])
    else:
        for it in range_constexpr(passes):
            r = it * ROW_LANES + tid // TASK_TRIPLES
            if r < g_rows:
                bits = _rs_sum(c, got[it * TP : (it + 1) * TP], rbs[it])
                ag_push_all(c, ag_pair(rank, r, tr), bits)

stage_rs_reduce_lp3(c, task)

lamport_rs: this rank's G rows of the task's columns in the tagged reducer's layout (lane = (row, triple); every pass's poll in one batch): each peer's segment read at the lane's triple, plus this rank's own partial (cached), summed in rank order (f32) -> _rs_sum -> tagged triples to every rank's AG slot, as the tagged build.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def stage_rs_reduce_lp3(c, task):
    """lamport_rs: this rank's G rows of the task's columns in the tagged reducer's
    layout (lane = (row, triple); every pass's poll in one batch): each peer's segment
    read at the lane's triple, plus this rank's own partial (cached), summed in rank
    order (f32) -> _rs_sum -> tagged triples to every rank's AG slot, as the tagged
    build."""
    g_rows, rank, tid, a = c["G"], c["rank"], c["tid"], c["args"]
    tl = tid % TASK_TRIPLES
    tr = task * TASK_TRIPLES + tl
    passes = -(-g_rows // ROW_LANES)
    rows = [it * ROW_LANES + tid // TASK_TRIPLES for it in range(passes)]
    rs = [fx.min(r, g_rows - 1) for r in rows]
    owns = [triple_bits(a["part"], rank * g_rows + rr, tr) for rr in rs]
    rbs = [triple_bits(a["res_in"], rank * g_rows + rr, tr) for rr in rs]
    addrs, lives = [], []
    for it in range_constexpr(passes):
        for d in range_constexpr(1, TP):
            addrs.append(lp_seg(c, c["sym"], LP_RS, (rank + d) % TP, task, rs[it]))
            lives.append(rows[it] < g_rows)
    if const_expr(c["lpprobe"]):
        lp_probe(
            c,
            addrs[-1] + fx.Int64(lp_triple_off(tl)),
            (c["wave"] * (64 // TASK_TRIPLES)) < g_rows,
        )
    got = lp_poll_triples(c, addrs, tl, lives, SITE_RS)
    stamp(c, "rs_land")
    for it in range_constexpr(passes):
        r = rows[it]
        if r < g_rows:
            own = [fx.Int32(owns[it]), fx.Int32(owns[it] >> 32)]
            srcs = _by_rank(
                c, own, [list(got[it * (TP - 1) + d]) for d in range(TP - 1)]
            )
            bits = _rs_sum(c, [[(w[0], w[1])] for w in srcs], rbs[it])
            ag_push_all(c, ag_pair(rank, r, tr), bits)

stage_rs_reduce_lpv(c, task)

lamport: this rank's G rows of the task's columns, lane = (row, 16-byte vector): the 3 peers' slots + this rank's own partial (cached), summed in rank order (f32) -> bf16 h (mode 2: bf16(h + this rank's residual rows)) -> every rank's AG inbox [this rank][task] as plain bf16 (ownfirst: this rank's own inbox first; mixdst: the peers rotated by row).

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def stage_rs_reduce_lpv(c, task):
    """lamport: this rank's G rows of the task's columns, lane = (row, 16-byte vector):
    the 3 peers' slots + this rank's own partial (cached), summed in rank order (f32) ->
    bf16 h (mode 2: bf16(h + this rank's residual rows)) -> every rank's AG inbox
    `[this rank][task]` as plain bf16 (ownfirst: this rank's own inbox first; mixdst:
    the peers rotated by row)."""
    g_rows, rank, tid, a = c["G"], c["rank"], c["tid"], c["args"]
    r = tid // LP_VEC
    q = tid % LP_VEC
    live = r < g_rows
    rr = fx.min(r, g_rows - 1)
    dw = ((rank * g_rows + rr) * HIDDEN + task * 48 + 8 * q) // 2
    own = _ld16(a["part"], dw)
    rb = _ld16(a["res_in"], dw)
    q16 = fx.Int64(16) * fx.Int64(q)
    addrs = [
        lp_seg(c, c["sym"], LP_RS, (rank + d) % TP, task, rr) + q16
        for d in range(1, TP)
    ]
    if const_expr(c["lpprobe"]):
        lp_probe(c, addrs[-1], ((c["wave"] * 64) // LP_VEC) < g_rows)
    got = lp_poll_vecs(c, addrs, live, SITE_RS)
    stamp(c, "rs_land")
    srcs = _by_rank(c, own, got)
    hw = []
    for e in range_constexpr(4):
        lo, hi = bf2_f32(srcs[0][e])
        for s in range_constexpr(1, TP):
            l2, h2 = bf2_f32(srcs[s][e])
            lo = lo + l2
            hi = hi + h2
        rlo, rhi = bf2_f32(rb[e])
        blo = c["sharded"].select(
            bf16_bits(bf16_round(lo) + rlo), bf16_bits(lo)
        )  # the stock add of bf16 h (mode 2)
        bhi = c["sharded"].select(bf16_bits(bf16_round(hi) + rhi), bf16_bits(hi))
        hw.append(fx.Int32(blo | (bhi << 16)))
    hv = lp_canon(hw)
    if const_expr(c["lpack"]):  # own AG first, then the gate, then the peers
        if live:
            lp_st16(lp_seg(c, c["sym"], LP_AG, rank, task, r) + q16, hv)
        lp_wait_acked(c)
    if live:
        if const_expr(c["ownfirst"] and not c["lpack"]):
            lp_st16(lp_seg(c, c["sym"], LP_AG, rank, task, r) + q16, hv)
        for j in range_constexpr(TP - 1):
            k = (j + r // c["lpmix"]) % (TP - 1) if c["lpmix"] else j
            p = (rank + 1 + k) % TP
            lp_st16(lp_seg(c, lp_peer(c, p), LP_AG, rank, task, r) + q16, hv)
        if const_expr(not c["ownfirst"] and not c["lpack"]):
            lp_st16(lp_seg(c, c["sym"], LP_AG, rank, task, r) + q16, hv)

stage_sort(c)

Every row's top-k -> AITER moe_sorting's outputs (block 16): expert segments in ascending expert id, a pick's place in its segment by (row, slot), each segment padded to 16 with MOCK = (TOPK << 24) | S at weight 0, an expert id a 16-row block, num_valid = [padded rows, S]. Experts without picks get no block.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def stage_sort(c):
    """Every row's top-k -> AITER moe_sorting's outputs (block 16): expert segments in
    ascending expert id, a pick's place in its segment by (row, slot), each segment
    padded to 16 with MOCK = (TOPK << 24) | S at weight 0, an expert id a 16-row block,
    num_valid = [padded rows, S]. Experts without picks get no block."""
    s, tid, a = c["S"], c["tid"], c["args"]
    npick = s * TOPK
    pe, pw, cnt, off, fill = c["pe"], c["pw"], c["cnt"], c["off"], c["fill"]
    if tid < EXPERTS:
        fx.ptr_store(fx.Int32(0), cnt + tid)
        if const_expr(c["fastsort"]):
            fx.ptr_store(fx.Int32(0), fill + tid)
    if const_expr(
        c["orouter"]
    ):  # every pick's pairs in one poll batch (S > 128: one round trip, not two)
        its = -(-npick // THREADS)
        specs = []
        for it in range_constexpr(its):
            pc = fx.min(it * THREADS + tid, npick - 1)
            base = ROUTE_P0 + (pc // TOPK) * 2 * TOPK + pc % TOPK
            specs += [(c["own"], base, 1), (c["own"], base + TOPK, 1)]
        picks = c["mb"].poll(specs, site=SITE_ROUTE)
        for it in range_constexpr(its):
            pp = it * THREADS + tid
            if pp < npick:
                buf_store(picks[2 * it][0], a["topk_ids"], pp)
                buf_store(picks[2 * it + 1][0].bitcast(fx.Float32), a["topk_w"], pp)
                fx.ptr_store(picks[2 * it][0], pe + pp)
                fx.ptr_store(picks[2 * it + 1][0].bitcast(fx.Float32), pw + pp)
    for it in range_constexpr(0 if c["orouter"] else -(-npick // THREADS)):
        p = it * THREADS + tid
        if p < npick:
            t = p // TOPK
            k = p % TOPK
            if const_expr(c["orouter"]):
                got = c["mb"].poll(
                    [
                        (c["own"], ROUTE_P0 + t * 2 * TOPK + k, 1),
                        (c["own"], ROUTE_P0 + t * 2 * TOPK + TOPK + k, 1),
                    ],
                    site=SITE_ROUTE,
                )
                buf_store(got[0][0], a["topk_ids"], p)
                buf_store(got[1][0].bitcast(fx.Float32), a["topk_w"], p)
            else:
                got = c["mb"].poll(
                    [
                        (c["route"], t * 2 * TOPK + k, 1),
                        (c["route"], t * 2 * TOPK + TOPK + k, 1),
                    ],
                    site=SITE_ROUTE,
                )
            fx.ptr_store(got[0][0], pe + p)
            fx.ptr_store(got[1][0].bitcast(fx.Float32), pw + p)
    gpu.barrier()
    stamp(c, "s_picks")
    for it in range_constexpr(-(-npick // THREADS)):
        p = it * THREADS + tid
        if p < npick:
            fx.llvm.atomic_add(
                cnt + fx.ptr_load(pe + p), fx.Int32(1), syncscope="workgroup"
            )
    gpu.barrier()

    def padded(n):
        return ((n + (SORT_BLOCK - 1)) // SORT_BLOCK) * SORT_BLOCK

    if const_expr(c["fastsort"]):
        sort_scan_wave(c, padded)
    else:
        sort_scan_serial(c, padded)
    stamp(c, "s_scan")
    for it in range_constexpr(-(-npick // THREADS)):
        p = it * THREADS + tid
        if p < npick:
            e = fx.ptr_load(pe + p)
            # a slot in the expert's segment (the order inside a segment only moves
            # gemm2's bf16 atomic adds, whose order is not fixed anyway)
            rank = fx.llvm.atomic_add(fill + e, fx.Int32(1), syncscope="workgroup")
            pos = fx.ptr_load(off + e) + rank
            buf_store(fx.Int32(((p % TOPK) << 24) | (p // TOPK)), a["sorted_ids"], pos)
            buf_store(fx.ptr_load(pw + p), a["sorted_w"], pos)
    stamp(c, "s_placed")
    if const_expr(c["fastsort"]):
        # thread (expert e, slot j): pad slot o + n + j of e's segment, expert id of e's
        # block j
        for it in range_constexpr(EXPERTS * SORT_BLOCK // THREADS):
            idx = it * THREADS + tid
            e = idx // SORT_BLOCK
            j = idx % SORT_BLOCK
            n = fx.ptr_load(cnt + e)
            o = fx.ptr_load(off + e)
            o_end = fx.ptr_load(off + e + 1)
            if o + n + j < o_end:
                buf_store(fx.Int32((TOPK << 24) | s), a["sorted_ids"], o + n + j)
                buf_store(fx.Float32(0.0), a["sorted_w"], o + n + j)
            if o // SORT_BLOCK + j < o_end // SORT_BLOCK:
                buf_store(e, a["sorted_eids"], o // SORT_BLOCK + j)
    else:
        sort_pad_serial(c)

stage_zero(c, part)

moe_buf [S, HIDDEN] bf16 := 0, slice part of ZERO_CTAS (gemm2 adds into it atomically).

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
@traced
def stage_zero(c, part):
    """moe_buf [S, HIDDEN] bf16 := 0, slice `part` of ZERO_CTAS (gemm2 adds into it
    atomically)."""
    words = c["S"] * HIDDEN // 2
    per = -(-words // ZERO_CTAS)
    per = -(-per // 4) * 4
    z = fx.Vector.filled(4, 0, fx.Int32)
    for it in range_constexpr(-(-per // (4 * THREADS))):
        w = part * per + (it * THREADS + c["tid"]) * 4
        if (w < (part + 1) * per) & (w < words):
            buf_store(z, c["args"]["moe_buf"], w)

stamp(c, name)

s_memrealtime -> tl[CTA][point] (timeline builds); name a TL name or a raw index (post_kernel.py).

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def stamp(c, name):
    """s_memrealtime -> `tl[CTA][point]` (timeline builds); `name` a TL name or a raw
    index (post_kernel.py)."""
    if const_expr(c["timeline"]):
        k = TL[name] if isinstance(name, str) else name
        buf_store(memrealtime(), c["args"]["tl"], fx.block_idx.x * TL_POINTS + k)

tl_participants(s, orouter=False, afinish=False)

The CTAs that do the work a stamp closes (the harness reduces min / median / max over them).

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def tl_participants(
    s: int, orouter: bool = False, afinish: bool = False
) -> dict[str, list[int]]:
    """The CTAs that do the work a stamp closes (the harness reduces min / median / max
    over them)."""
    seam = list(range(SEAM0, SEAM0 + SEAM_TASKS))
    rows = orouter_row_ctas(s) if orouter else list(range(s))
    route = list(range(ROUTER0, ROUTER0 + s // TP)) if orouter else rows
    router = list(range(ROUTER0, ROUTER0 + ROUTER_TASKS))
    out = {
        "entry": list(range(BLOCKS)),
        "zero": router,
        "push": seam,
        "rs_land": seam,
        "reduce": seam,
        "ag_land": rows,
        "norm": rows,
        "r_rows": router,
        "r_mfma": router,
        "router": router,
        "rt_land": route,
        "route": route,
        "done": list(range(BLOCKS)),
        "epoch": [0],
    }
    for k in ("s_picks", "s_scan", "s_placed", "sort"):
        out[k] = [SORT_CTA]
    if afinish:
        out["epoch"] = list(range(BLOCKS))
    if orouter:
        own = list(range(ROUTER0, ROUTER0 + s // TP))
        out["own_land"] = out["own_norm"] = own
        out["ag_land"] = out["norm"] = [b for b in rows if b not in own]
    return out

triple_bits(buf, t, triple)

Row t's bf16 at columns 3 triple .. of a [*, HIDDEN] bf16 buffer, in bits 0 .. 47.

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def triple_bits(buf, t, triple):
    """Row t's bf16 at columns 3 triple .. of a [*, HIDDEN] bf16 buffer, in bits
    0 .. 47."""
    v = lp.load_dwords(buf, t * (HIDDEN // 2) + (3 * triple) // 2, 2)
    w = (fx.Int64(fx.Uint32(fx.Int32(v[1]))) << 32) | fx.Int64(
        fx.Uint32(fx.Int32(v[0]))
    )
    return w >> fx.Int64(16 * (triple & 1))

triple_value(lo, hi, k)

Value k (0 .. 2) of a triple pair's (low word, high word).

Source code in vllm/models/minimax_m3/amd/glue/kernel.py
def triple_value(lo, hi, k):
    """Value k (0 .. 2) of a triple pair's (low word, high word)."""
    if const_expr(k == 2):
        return bf16_bits_f32(hi)
    return bf16_bits_f32(lo >> (16 * k))