llmcompressor.entrypoints.converter.expert_pruner
Classes:
-
ExpertPruner–Prune MoE experts based on a per-expert score. Experts are scored either by
ExpertPruner
Bases: Converter
Prune MoE experts based on a per-expert score. Experts are scored either by
router weight magnitude (router.weight[i].abs().sum(), L1 magnitude) or
by a statistic read from a saliency report produced by
REAPPruningModifier (see its report_path argument). The
lowest-scoring sparsity fraction of experts is pruned, retaining the
rest. With uniform, the same fraction is pruned from every layer.
Otherwise, the lowest-scoring experts across all layers are pruned, so layers
may retain different numbers of experts.
Supports both 2D expert weights (one tensor per expert, e.g.
model.layers.0.mlp.experts.3.gate_proj.weight) and 3D stacked expert
weights (model.layers.0.mlp.experts.gate_proj.weight with shape
[num_experts, out_features, in_features]).
Pruning removes expert weight tensors (or slices stacked tensors), adjusts
router weights, and updates the expert count. Uniform pruning writes the new
count to the model config, while non-uniform pruning records per-layer
counts in the quantization config's moe_sparsity_config.
Besides the router weight, a layer may carry auxiliary router tensors whose shape or contents depend on the expert count. These are pruned alongside the router:
- per-expert vectors (
aux_pattern), e.g. a routing-bias such as DeepSeek'sffn.gate.bias(e_score_correction_bias) with shape[num_experts]. Sliced along dim 0 by the retained expert indices, exactly like the router weight's rows. - expert-index tables (
index_pattern), e.g. DeepSeek's hash-routingffn.gate.tid2eidwith shape[vocab, num_experts_per_tok]whose values are expert indices. The values are remapped from old to new expert indices after renumbering. A layer carrying such a table is a hash (statically routed) layer: see the hash-layer handling below.
Hash layers route each token to a fixed set of experts via a lookup table
rather than a learned top-k, so experts cannot be redistributed after
pruning. With uniform=True any hash layer makes pruning ambiguous and
raises. With uniform=False every expert in a hash layer is protected
(never pruned), so its table stays valid and the global budget is spent on
the remaining learned-routing layers.
Group-limited routers (declared in config.json via n_group /
topk_group, e.g. DeepSeek-V3) partition each layer's experts into
contiguous groups and only route tokens to experts of the top
topk_group groups, so every group must retain the same number of
experts. Uniform pruning drops the same number of experts from every
group (the per-layer drop count is rounded to a whole number of experts
per group, like the in-memory REAP pruning path), while non-uniform
pruning, which would unbalance the groups, is rejected.
Parameters:
-
router_pattern–regex matching router weight tensor names
-
expert_pattern–regex matching expert weight tensor names
-
aux_pattern–regex matching per-expert router vectors (e.g. routing bias) that must be sliced along dim 0 by retained expert indices
-
index_pattern–regex matching expert-index tables (e.g. a hash routing table) whose values are expert indices to be remapped
-
sparsity–fraction of experts to prune, in [0, 1]
-
uniform–whether the same number of experts is pruned from every layer
-
num_experts_config_key–config.json attribute name for expert count
-
retained_experts–pre-computed mapping of router_name -> retained expert indices (sorted). Built by :meth:
from_pretrained. -
expert_to_router–mapping of expert tensor name -> associated router tensor name. Built by :meth:
from_pretrained. -
aux_to_router–mapping of auxiliary tensor name (bias or index table) -> associated router tensor name. Built by :meth:
from_pretrained. -
hash_routers–set of router names whose layer routes statically via an expert-index table (
index_pattern). Built by :meth:from_pretrained. -
expert_indices–for 2D experts, mapping of tensor name -> expert index. Empty for 3D experts.
-
is_3d–whether expert weights are 3D stacked tensors
-
router_layer_indices–mapping of router name -> decoder layer index, parsed from the router tensor name. Only populated for non-uniform pruning, which records per-layer expert counts
-
num_decoder_layers–total number of decoder layers in the model. Only populated for non-uniform pruning
Methods:
-
from_pretrained–Build the converter by scanning the checkpoint for router weights,
Source code in src/llmcompressor/entrypoints/converter/expert_pruner.py
from_pretrained
classmethod
from_pretrained(
model_name_or_path: str | PathLike,
sparsity: float,
metric: Literal[
"magnitude",
"saliency",
"layerwise_saliency",
"count",
],
uniform: bool = True,
router_pattern: str = "mlp\\.gate\\.weight$",
expert_pattern: str = "mlp\\.experts\\.(gate_up_proj|down_proj)",
aux_pattern: str | None = "(?:mlp|ffn)\\.gate\\.bias$",
index_pattern: str
| None = "(?:mlp|ffn)\\.gate\\.tid2eid$",
num_experts_config_key: str | None = None,
saliency_report_path: str | None = None,
) -> ExpertPruner
Build the converter by scanning the checkpoint for router weights, scoring experts, and determining which to retain.
Parameters:
-
model_name_or_path(str | PathLike) –HuggingFace stub or local checkpoint path
-
sparsity(float) –fraction of experts to prune, in [0, 1]
-
metric(Literal['magnitude', 'saliency', 'layerwise_saliency', 'count']) –how experts are scored, lowest scores are pruned first: -
"magnitude": L1 magnitude of each expert's router weight row -"saliency": raw REAP saliency from the saliency report -"layerwise_saliency": REAP saliency divided by the highest saliency of its layer. Normalizing makes saliency comparable across layers, which matters for non-uniform pruning -"count": number of tokens routed to each expert, from the saliency report -
uniform(bool, default:True) –if True, prune
round(sparsity * num_experts)experts from every layer and write the new count tonum_experts_config_keyin the model config. If False, prune theround(sparsity * total_experts)lowest-scoring experts across all layers, never taking a layer belownum_experts_per_tok, and record the per-layer counts in the quantization config'smoe_sparsity_config. Group-limited routers (n_group > 1) must prune uniformly, dropping the same number of experts from every group, souniform=Falseis rejected for them -
router_pattern(str, default:'mlp\\.gate\\.weight$') –regex matching router weight tensor names
-
expert_pattern(str, default:'mlp\\.experts\\.(gate_up_proj|down_proj)') –regex matching expert weight tensor names
-
aux_pattern(str | None, default:'(?:mlp|ffn)\\.gate\\.bias$') –regex matching per-expert router vectors (e.g.
ffn.gate.bias) sliced along dim 0 by retained experts.Nonedisables auxiliary-vector pruning -
index_pattern(str | None, default:'(?:mlp|ffn)\\.gate\\.tid2eid$') –regex matching expert-index tables (e.g.
ffn.gate.tid2eid) whose values are remapped to new expert indices. A layer with such a table is treated as a hash layer.Nonedisables index-table handling -
num_experts_config_key(str | None, default:None) –config.json key holding the expert count. If None, auto-detected from config.json.
-
saliency_report_path(str | None, default:None) –path to the
.jsonreport written byREAPPruningModifier. Required for every metric but"magnitude". Its layers must be ordered by layer index
Source code in src/llmcompressor/entrypoints/converter/expert_pruner.py
127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 | |