llmcompressor.modifiers.pruning.mone.utils
Utilities for MoNe (Mixture of Novices): expert collapse.
The lowest-saliency experts are collapsed into constant "novice" vectors
(each expert's mean output, an unbiased estimator of its contribution).
Retained experts are left-shifted while the router keeps all R + N rows,
so novices continue to receive their original router weights at inference
time.
Functions:
-
mone_collapse_moe_layer–Collapse a MoE block into retained experts + novices (MoNe).
mone_collapse_moe_layer
mone_collapse_moe_layer(
moe_name: str,
moe_module: Module,
retained: list[int],
dropped: list[int],
novices: Tensor,
) -> list[int]
Collapse a MoE block into retained experts + novices (MoNe).
The retained experts are left-shifted into a contiguous ModuleList
exactly as in plain expert pruning (via
:func:~llmcompressor.modifiers.pruning.expert_sparsity.utils.prune_moe_layer
with prune_router=False). Unlike plain pruning, the router is not
shrunk: its rows are reordered so the retained experts occupy rows
0..R-1 and the collapsed experts occupy rows R..R+N-1 (matching
the order of novices). The collapsed experts' mean outputs are
installed as experts.novices, which the experts module serves in place
of the collapsed experts' outputs for tokens the router sends there.
Parameters:
-
moe_name(str) –name of the MoE block, used for error reporting
-
moe_module(Module) –the MoE block module
-
retained(list[int]) –indices of the experts to keep, in their new order
-
dropped(list[int]) –indices of the experts to collapse, ordered to match the rows of
novices -
novices(Tensor) –(N, hidden_dim)tensor of novice vectors, ordered to matchdropped(i.e. the router rows that follow the retained block)
Returns:
-
list[int]–the retained expert indices