llmcompressor.modifiers.pruning.mone.base
MoNe (Mixture of Novices) pruning modifier for MoE models.
MoNe mirrors REAP's collapse structure (see
:class:~llmcompressor.modifiers.pruning.reap.REAPPruningModifier) but ranks
experts with its own "fusion" saliency metric
S_j = ||std_tokens(f_j)||_2 * E_tokens[g_j * 1(j in top-k)] accumulated via
calibration hooks. It selects the lowest-saliency experts per layer and,
instead of pruning them, collapses them into constant "novice" vectors -- each
expert's mean output, an unbiased estimator of its contribution -- leaving the
router untouched so the novices keep receiving their original routing weights.
See: https://arxiv.org/abs/2507.00390
Classes:
-
MoNePruningModifier–Collapses low-saliency experts from MoE layers into constant "novice" vectors
MoNePruningModifier
Bases: ExpertPruningModifierBase
Collapses low-saliency experts from MoE layers into constant "novice" vectors
using the MoNe saliency metric. For each expert j the saliency is
``S_j = ||std_tokens(f_j)||_2 * E_tokens[g_j * 1(j in top-k)]``
the product of the expert's output fluctuation (the L2 norm of the
per-dimension standard deviation of expert j's output f_j) and its
routing score (the frequency-weighted mean router gate weight g_j,
averaged over all tokens and zero when j is not in the top-k). The
lowest-saliency experts per layer are collapsed: their mean
output over calibration tokens becomes a constant novice vector, retained
experts are left-shifted, and the router is kept at its original width so
novices continue to receive their routing weights. The per-layer retained
counts are recorded in the checkpoint's
quantization_config.moe_sparsity_config with method="novice".
MoNe runs during the sequential calibration pipeline: routing and expert
output statistics are accumulated via hooks on the MoE experts, and the
structural collapse for a layer is executed when it completes
(SEQUENTIAL_EPOCH_END).
Parameters:
-
sparsity–fraction of experts to collapse per layer, in
[0, 1). When0.0, no experts are collapsed; saliency is still computed so that areport_pathcan be written, but the model weights and config are never modified. -
ignore–module name patterns to skip during MoE layer pruning.
-
report_path–optional path to a
.jsonfile where the per-expert statistics are written after calibration completes. The report is a JSON object mappingcountandsaliencyto one inner list per MoE layer (ordered by layer). Required whensparsityis0.0, since the report is then the modifier's only output.Example recipe::
MoNePruningModifier: sparsity: 0.25
Methods:
-
expert_hook–Record the expert's full output for every token in this batch.
-
experts_block_hook–Update the routing and novice accumulators for the batch.
-
init_moe_stats–Allocate the per-layer MoNe accumulators.
-
prune_moe_layer–Collapse the layer's dropped experts into novices.
-
reduce_moe_stats–Reduce a layer's statistics across ranks, in place.
-
saliency–Return the MoNe fusion saliency
fluctuation * routing_score.
expert_hook
Record the expert's full output for every token in this batch.
Captures expert f_j's output for :meth:experts_block_hook to
consume at the end of the batch; it feeds the novice mean and the
output-fluctuation term of the saliency.
Parameters:
-
moe_name(str) –Name of the MoE layer the expert belongs to
-
expert_idx(int) –Index of the expert within its layer
-
module(ExpertMLP) –The expert module the hook is attached to
-
args(tuple) –The expert's forward arguments
-
output(Tensor) –The expert's output for the current batch
Source code in src/llmcompressor/modifiers/pruning/mone/base.py
experts_block_hook
Update the routing and novice accumulators for the batch.
Combines the router's top-k decisions with the captured expert outputs, then resets the per-batch output buffers.
Parameters:
-
moe_name(str) –Name of the MoE layer the block belongs to
-
module(Module) –The experts block the hook is attached to
-
args(tuple) –The block's forward arguments, where
args[1]andargs[2]are the router's top-k indices and weights -
output(Tensor) –The block's output for the current batch
Source code in src/llmcompressor/modifiers/pruning/mone/base.py
init_moe_stats
Allocate the per-layer MoNe accumulators.
Output statistics (sum_output, sum_sq_output, count):
the mean output is the novice vector, and the per-dimension variance
gives the fluctuation term of the saliency metric. Routing statistics
(sum_gates, token_count): the frequency-weighted mean gate
weight gives the routing score term.
count (per-expert routed-token counts) is all-reduced by
:meth:reduce_moe_stats so that every rank decides identically
whether a layer saw calibration data; the remaining accumulators,
including the scalar token_count, are reduced to the source rank,
which is the only rank that consumes them (saliency, novices, report).
Parameters:
-
moe_name(str) –Name of the MoE layer
-
num_experts(int) –Number of experts in the layer
-
hidden_dim(int) –Hidden dimension of the layer, sizing the output accumulators
-
device(device) –The layer's execution device
Source code in src/llmcompressor/modifiers/pruning/mone/base.py
prune_moe_layer
Collapse the layer's dropped experts into novices.
Their mean outputs (computed on the source rank, which holds the globally reduced statistics, and broadcast to every rank) are installed as constant novice vectors while the router keeps its full width.
Parameters:
-
moe_name(str) –Name of the MoE layer to collapse
-
moe_module(Module) –The MoE block module to collapse
-
retained(list[int]) –Indices of the experts to keep, in their new order
Source code in src/llmcompressor/modifiers/pruning/mone/base.py
reduce_moe_stats
Reduce a layer's statistics across ranks, in place.
All-reduces the per-expert counts, which the base reads to decide
whether the layer saw calibration data, so every rank makes the same
decision; reduces the remaining accumulators (including the scalar
token_count) to the source rank, which is the only rank that
consumes them (saliency, novices, report).
Parameters:
-
moe_name(str) –Name of the MoE layer whose stats to reduce
Source code in src/llmcompressor/modifiers/pruning/mone/base.py
saliency
Return the MoNe fusion saliency fluctuation * routing_score.
Parameters:
-
moe_name(str) –Name of the MoE layer
Returns:
-
Tensor–Per-expert saliency, indexed by expert