vllm.models.mistral.mistral_large4.amd.specialized_decode.mono.layout
¶
Compile-time storage layout and CTA schedule for the ML4 MonoKernel.
Functions:
-
dn_tile–Return rows per expert-down/FFN-reduce task for the tuned schedule.
-
down_prefetch_batch–How many expert-down weight tiles a wave keeps in flight.
-
index_cp_capacity–Maximum context shard, rounded to a complete scoring tile.
-
index_cp_world–Use two independent CP4 groups for TP8's single-query medium contexts.
-
index_fan–Lists one merge level consumes per list it emits.
-
index_fp8_data_dword–Dword offset of FP8 index elements
k..k + 3of one cache row. -
index_fp8_scale_dword–Dword offset of the fp32 scale of one FP8 index cache row.
-
index_levels–Merge levels a build needs: the depth at which one list remains.
-
index_lists–Lists the tree holds at
levelfor a build ofindex_max_seq. -
index_seg–Keys one select task stages in LDS: the leaf size, not a capacity.
-
index_select_capacity–Context the LOCAL (merge-tree) selector must serve in this build.
-
index_select_stage–Stage name of tree
level; the final level keepsindex_select. -
indexer_uses_tiles–Use the distributed tiled radix only to shard a context ACROSS RANKS.
-
layout–Return byte offsets for per-rank scratch and symmetric peer buffers.
-
paged_row_contract–Reference the device contract for one ATOM-layout request row.
-
ptpc_stage_chunk–How many (sample, group) cells
stage_x_pairs_ptpcstages together. -
sparse_cache_rows–Resolve the cache rows consumed by one flat or paged attention row.
-
sparse_keys_per_task–Use narrower sparse tiles when samples or attention heads fill LDS.
-
split_acc_head–Map one split-PV MFMA row to its local attention head.
-
split_score_column–Locate this wave's head column in a 16x16 MFMA score tile.
-
stage_tasks–Return
(stage name, task count)pairs in execution order. -
ug_split–Return the balanced up/gate leftover split for batches two and four.
dn_tile(samples, expert_mxfp4=False)
¶
Return rows per expert-down/FFN-reduce task for the tuned schedule.
Source code in vllm/models/mistral/mistral_large4/amd/specialized_decode/mono/layout.py
down_prefetch_batch(samples, native_fp4_mfma=False, atom_experts=False)
¶
How many expert-down weight tiles a wave keeps in flight.
run_units issues round r+1's buffer loads before consuming round r's,
so this sets the stage's memory-level parallelism. Changing it is
bit-identical -- run_units walks the units in index order whatever the
batch size, so the MFMA accumulation order does not move.
Measured upstream on a TP4 shard, this stage is not latency-starved: sweeping 2..8 at S=6 and S=12 is monotone the wrong way (bigger is slower, presumably register pressure), and 2 is the best of the values tried at both widths.
MONO_DN_BATCH overrides it for an A/B.
Source code in vllm/models/mistral/mistral_large4/amd/specialized_decode/mono/layout.py
index_cp_capacity(index_max_seq, npes)
¶
Maximum context shard, rounded to a complete scoring tile.
Source code in vllm/models/mistral/mistral_large4/amd/specialized_decode/mono/layout.py
index_cp_world(npes, samples, index_max_seq)
¶
Use two independent CP4 groups for TP8's single-query medium contexts.
Source code in vllm/models/mistral/mistral_large4/amd/specialized_decode/mono/layout.py
index_fan(index_max_seq, topk)
¶
index_fp8_data_dword(page, row, k, page_size)
¶
Dword offset of FP8 index elements k .. k + 3 of one cache row.
vLLM's ROCm DSA indexer page (indexer_k_quant_and_cache_triton for
page_size > 1): the page_size x INDEX_DIM E4M3 keys in 16 x 16-byte
SHUFFLE tiles [row // 16, k // 16, row % 16, k % 16], then page_size
fp32 row scales (INDEX_PAGE_ROW bytes per row). k % 4 == 0, so the
four bytes never straddle a tile.
Source code in vllm/models/mistral/mistral_large4/amd/specialized_decode/mono/layout.py
index_fp8_scale_dword(page, row, page_size)
¶
Dword offset of the fp32 scale of one FP8 index cache row.
index_levels(index_max_seq, topk)
¶
Merge levels a build needs: the depth at which one list remains.
index_max_seq <= INDEX_SEG gives 0, i.e. today's single flat select.
Source code in vllm/models/mistral/mistral_large4/amd/specialized_decode/mono/layout.py
index_lists(index_max_seq, level, topk)
¶
Lists the tree holds at level for a build of index_max_seq.
Source code in vllm/models/mistral/mistral_large4/amd/specialized_decode/mono/layout.py
index_seg(index_max_seq)
¶
Keys one select task stages in LDS: the leaf size, not a capacity.
index_select_capacity(index_max_seq, indexer_cp)
¶
Context the LOCAL (merge-tree) selector must serve in this build.
A tiled build keeps the local selector as its short-query fallback, bounded
by use_short_index; it reads the peer-exchanged score plane, which is
one leaf wide, so its tree is always a bare leaf. Every other build sends
the whole context through the tree.
Source code in vllm/models/mistral/mistral_large4/amd/specialized_decode/mono/layout.py
index_select_stage(level, depth)
¶
Stage name of tree level; the final level keeps index_select.
indexer_uses_tiles(index_max_seq, indexer_cp)
¶
Use the distributed tiled radix only to shard a context ACROSS RANKS.
It used to answer "is this context past what one selector CTA can stage in
LDS", because the local selector staged every key of the context. It no
longer does -- the selector is a merge tree over fixed INDEX_SEG leaves
(index_levels), so one CTA serves any context. Tiles are therefore
purely the context-parallel feature now: shard the scorer and the selection
over index_cp_world ranks, which only a CP build can do.
Source code in vllm/models/mistral/mistral_large4/amd/specialized_decode/mono/layout.py
layout(samples, heads, npes, sparse_attention_topk, with_indexer=False, index_max_seq=4096, indexer_cp=False, inter=INTER, output_heads=None, dcp_size=1, native_fp4_mfma=False)
¶
Return byte offsets for per-rank scratch and symmetric peer buffers.
Source code in vllm/models/mistral/mistral_large4/amd/specialized_decode/mono/layout.py
371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 | |
paged_row_contract(sparse_kv_indices, sparse_kv_indptr, sample, slot_mapping=None)
¶
Reference the device contract for one ATOM-layout request row.
Source code in vllm/models/mistral/mistral_large4/amd/specialized_decode/mono/layout.py
ptpc_stage_chunk(samples)
¶
How many (sample, group) cells stage_x_pairs_ptpc stages together.
The helper's two per-cell costs are per-call costs: poll resolves a
whole batch in one round trip and block_maxs reduces a whole list in
two barriers. Batching cells is bit-identical and removes a clean O(S)
term from q_b, uk and o -- stages whose task count and weight
bytes do not depend on S at all.
Measured upstream on a TP4 shard (MONO_PTPC_CHUNK=...,
7 trials, interleaved, spread <= 0.27%):
========== =============== ================ chunk S=6 us/launch S=12 us/launch ========== =============== ================ 1 (old) 113.785 201.790 2 112.507 196.527 4 113.238 194.113 6 116.048 194.134 all cells 116.012 214.911 ========== =============== ================
So the optimum moves with the width, and batching every cell is much
worse than batching four -- it holds every cell's values live across one
reduction and spills. MONO_PTPC_CHUNK overrides this for an A/B;
1 restores the per-cell walk exactly.
Source code in vllm/models/mistral/mistral_large4/amd/specialized_decode/mono/layout.py
sparse_cache_rows(sparse_kv_indices, *, sample, topk, cur_pos=0, sparse_kv_indptr=None)
¶
Resolve the cache rows consumed by one flat or paged attention row.
Source code in vllm/models/mistral/mistral_large4/amd/specialized_decode/mono/layout.py
sparse_keys_per_task(samples, heads=WAVES)
¶
Use narrower sparse tiles when samples or attention heads fill LDS.
Source code in vllm/models/mistral/mistral_large4/amd/specialized_decode/mono/layout.py
split_acc_head(head_group, lane_group, element)
¶
Map one split-PV MFMA row to its local attention head.
split_score_column(wave, lane)
¶
Locate this wave's head column in a 16x16 MFMA score tile.
stage_tasks(samples, heads, sparse_attention_topk, with_indexer=False, index_max_seq=4096, expert_mxfp4=False, inter=INTER, indexer_cp=False, npes=1, split_head_merge=False)
¶
Return (stage name, task count) pairs in execution order.
split_head_merge folds the two head groups of an (sample, key-chunk)
cell into one task, halving split. It must be given the same value the
kernel was built with: these counts set the CTA placement rotation and the
timeline's per-stage offsets, so a disagreement silently misplaces both.
Source code in vllm/models/mistral/mistral_large4/amd/specialized_decode/mono/layout.py
526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 | |
ug_split(samples, inter=INTER)
¶
Return the balanced up/gate leftover split for batches two and four.