Skip to content

llmcompressor.entrypoints.converter.expert_pruner

Classes:

  • ExpertPruner –

    Prune MoE experts based on a per-expert score. Experts are scored either by

ExpertPruner

ExpertPruner(*args, **kwargs)

Bases: Converter

Prune MoE experts based on a per-expert score. Experts are scored either by router weight magnitude (router.weight[i].abs().sum(), L1 magnitude) or by a statistic read from a saliency report produced by REAPPruningModifier (see its report_path argument). The lowest-scoring sparsity fraction of experts is pruned, retaining the rest. With uniform, the same fraction is pruned from every layer. Otherwise, the lowest-scoring experts across all layers are pruned, so layers may retain different numbers of experts.

Supports both 2D expert weights (one tensor per expert, e.g. model.layers.0.mlp.experts.3.gate_proj.weight) and 3D stacked expert weights (model.layers.0.mlp.experts.gate_proj.weight with shape [num_experts, out_features, in_features]).

Pruning removes expert weight tensors (or slices stacked tensors), adjusts router weights, and updates the expert count. Uniform pruning writes the new count to the model config, while non-uniform pruning records per-layer counts in the quantization config's moe_sparsity_config.

Besides the router weight, a layer may carry auxiliary router tensors whose shape or contents depend on the expert count. These are pruned alongside the router:

  • per-expert vectors (aux_pattern), e.g. a routing-bias such as DeepSeek's ffn.gate.bias (e_score_correction_bias) with shape [num_experts]. Sliced along dim 0 by the retained expert indices, exactly like the router weight's rows.
  • expert-index tables (index_pattern), e.g. DeepSeek's hash-routing ffn.gate.tid2eid with shape [vocab, num_experts_per_tok] whose values are expert indices. The values are remapped from old to new expert indices after renumbering. A layer carrying such a table is a hash (statically routed) layer: see the hash-layer handling below.

Hash layers route each token to a fixed set of experts via a lookup table rather than a learned top-k, so experts cannot be redistributed after pruning. With uniform=True any hash layer makes pruning ambiguous and raises. With uniform=False every expert in a hash layer is protected (never pruned), so its table stays valid and the global budget is spent on the remaining learned-routing layers.

Group-limited routers (declared in config.json via n_group / topk_group, e.g. DeepSeek-V3) partition each layer's experts into contiguous groups and only route tokens to experts of the top topk_group groups, so every group must retain the same number of experts. Uniform pruning drops the same number of experts from every group (the per-layer drop count is rounded to a whole number of experts per group, like the in-memory REAP pruning path), while non-uniform pruning, which would unbalance the groups, is rejected.

Parameters:

  • router_pattern –

    regex matching router weight tensor names

  • expert_pattern –

    regex matching expert weight tensor names

  • aux_pattern –

    regex matching per-expert router vectors (e.g. routing bias) that must be sliced along dim 0 by retained expert indices

  • index_pattern –

    regex matching expert-index tables (e.g. a hash routing table) whose values are expert indices to be remapped

  • sparsity –

    fraction of experts to prune, in [0, 1]

  • uniform –

    whether the same number of experts is pruned from every layer

  • num_experts_config_key –

    config.json attribute name for expert count

  • retained_experts –

    pre-computed mapping of router_name -> retained expert indices (sorted). Built by :meth:from_pretrained.

  • expert_to_router –

    mapping of expert tensor name -> associated router tensor name. Built by :meth:from_pretrained.

  • aux_to_router –

    mapping of auxiliary tensor name (bias or index table) -> associated router tensor name. Built by :meth:from_pretrained.

  • hash_routers –

    set of router names whose layer routes statically via an expert-index table (index_pattern). Built by :meth:from_pretrained.

  • expert_indices –

    for 2D experts, mapping of tensor name -> expert index. Empty for 3D experts.

  • is_3d –

    whether expert weights are 3D stacked tensors

  • router_layer_indices –

    mapping of router name -> decoder layer index, parsed from the router tensor name. Only populated for non-uniform pruning, which records per-layer expert counts

  • num_decoder_layers –

    total number of decoder layers in the model. Only populated for non-uniform pruning

Methods:

  • from_pretrained –

    Build the converter by scanning the checkpoint for router weights,

Source code in src/llmcompressor/entrypoints/converter/expert_pruner.py
def __init__(self, *args, **kwargs):
    raise ValueError("Please use `from_pretrained`")

from_pretrained classmethod

from_pretrained(
    model_name_or_path: str | PathLike,
    sparsity: float,
    metric: Literal[
        "magnitude",
        "saliency",
        "layerwise_saliency",
        "count",
    ],
    uniform: bool = True,
    router_pattern: str = "mlp\\.gate\\.weight$",
    expert_pattern: str = "mlp\\.experts\\.(gate_up_proj|down_proj)",
    aux_pattern: str | None = "(?:mlp|ffn)\\.gate\\.bias$",
    index_pattern: str
    | None = "(?:mlp|ffn)\\.gate\\.tid2eid$",
    num_experts_config_key: str | None = None,
    saliency_report_path: str | None = None,
) -> ExpertPruner

Build the converter by scanning the checkpoint for router weights, scoring experts, and determining which to retain.

Parameters:

  • model_name_or_path (str | PathLike) –

    HuggingFace stub or local checkpoint path

  • sparsity (float) –

    fraction of experts to prune, in [0, 1]

  • metric (Literal['magnitude', 'saliency', 'layerwise_saliency', 'count']) –

    how experts are scored, lowest scores are pruned first: - "magnitude": L1 magnitude of each expert's router weight row - "saliency": raw REAP saliency from the saliency report - "layerwise_saliency": REAP saliency divided by the highest saliency of its layer. Normalizing makes saliency comparable across layers, which matters for non-uniform pruning - "count": number of tokens routed to each expert, from the saliency report

  • uniform (bool, default: True ) –

    if True, prune round(sparsity * num_experts) experts from every layer and write the new count to num_experts_config_key in the model config. If False, prune the round(sparsity * total_experts) lowest-scoring experts across all layers, never taking a layer below num_experts_per_tok, and record the per-layer counts in the quantization config's moe_sparsity_config. Group-limited routers (n_group > 1) must prune uniformly, dropping the same number of experts from every group, so uniform=False is rejected for them

  • router_pattern (str, default: 'mlp\\.gate\\.weight$' ) –

    regex matching router weight tensor names

  • expert_pattern (str, default: 'mlp\\.experts\\.(gate_up_proj|down_proj)' ) –

    regex matching expert weight tensor names

  • aux_pattern (str | None, default: '(?:mlp|ffn)\\.gate\\.bias$' ) –

    regex matching per-expert router vectors (e.g. ffn.gate.bias) sliced along dim 0 by retained experts. None disables auxiliary-vector pruning

  • index_pattern (str | None, default: '(?:mlp|ffn)\\.gate\\.tid2eid$' ) –

    regex matching expert-index tables (e.g. ffn.gate.tid2eid) whose values are remapped to new expert indices. A layer with such a table is treated as a hash layer. None disables index-table handling

  • num_experts_config_key (str | None, default: None ) –

    config.json key holding the expert count. If None, auto-detected from config.json.

  • saliency_report_path (str | None, default: None ) –

    path to the .json report written by REAPPruningModifier. Required for every metric but "magnitude". Its layers must be ordered by layer index

Source code in src/llmcompressor/entrypoints/converter/expert_pruner.py
@classmethod
def from_pretrained(
    cls,
    model_name_or_path: str | os.PathLike,
    sparsity: float,
    metric: Literal["magnitude", "saliency", "layerwise_saliency", "count"],
    uniform: bool = True,
    router_pattern: str = r"mlp\.gate\.weight$",
    expert_pattern: str = r"mlp\.experts\.(gate_up_proj|down_proj)",
    aux_pattern: str | None = r"(?:mlp|ffn)\.gate\.bias$",
    index_pattern: str | None = r"(?:mlp|ffn)\.gate\.tid2eid$",
    num_experts_config_key: str | None = None,
    saliency_report_path: str | None = None,
) -> ExpertPruner:
    """
    Build the converter by scanning the checkpoint for router weights,
    scoring experts, and determining which to retain.

    :param model_name_or_path: HuggingFace stub or local checkpoint path
    :param sparsity: fraction of experts to prune, in [0, 1]
    :param metric: how experts are scored, lowest scores are pruned first:
        - ``"magnitude"``: L1 magnitude of each expert's router weight row
        - ``"saliency"``: raw REAP saliency from the saliency report
        - ``"layerwise_saliency"``: REAP saliency divided by the highest
          saliency of its layer. Normalizing makes saliency comparable
          across layers, which matters for non-uniform pruning
        - ``"count"``: number of tokens routed to each expert, from the
          saliency report
    :param uniform: if True, prune ``round(sparsity * num_experts)`` experts
        from every layer and write the new count to
        ``num_experts_config_key`` in the model config. If False, prune the
        ``round(sparsity * total_experts)`` lowest-scoring experts across all
        layers, never taking a layer below ``num_experts_per_tok``, and
        record the per-layer counts in the quantization config's
        ``moe_sparsity_config``. Group-limited routers (``n_group > 1``)
        must prune uniformly, dropping the same number of experts from
        every group, so ``uniform=False`` is rejected for them
    :param router_pattern: regex matching router weight tensor names
    :param expert_pattern: regex matching expert weight tensor names
    :param aux_pattern: regex matching per-expert router vectors (e.g.
        ``ffn.gate.bias``) sliced along dim 0 by retained experts. ``None``
        disables auxiliary-vector pruning
    :param index_pattern: regex matching expert-index tables (e.g.
        ``ffn.gate.tid2eid``) whose values are remapped to new expert
        indices. A layer with such a table is treated as a hash layer.
        ``None`` disables index-table handling
    :param num_experts_config_key: config.json key holding the expert count.
        If None, auto-detected from config.json.
    :param saliency_report_path: path to the ``.json`` report written by
        ``REAPPruningModifier``. Required for every metric but
        ``"magnitude"``. Its layers must be ordered by layer index
    """
    if not 0 <= sparsity <= 1:
        raise ValueError(f"sparsity must be in [0, 1], got {sparsity}")
    if metric != "magnitude" and metric not in REPORT_METRICS:
        raise ValueError(
            f"metric must be one of {('magnitude',) + REPORT_METRICS}, "
            f"got {metric!r}"
        )
    if metric in REPORT_METRICS and saliency_report_path is None:
        raise ValueError(f"metric={metric!r} requires a saliency_report_path")

    router_re = re.compile(router_pattern)
    expert_re = re.compile(expert_pattern)
    aux_re = re.compile(aux_pattern) if aux_pattern else None
    index_re = re.compile(index_pattern) if index_pattern else None

    model_files = get_checkpoint_files(model_name_or_path)
    weight_map = get_weight_map(model_files)
    config_data = load_model_config(model_files)

    # auto-detect num_experts_config_key from config.json
    if num_experts_config_key is None:
        num_experts_config_key = get_num_experts_config_key(config_data)

    # detect num_experts_per_tok for routing-floor validation
    num_experts_per_tok = get_num_experts_per_tok(config_data)

    # detect group-limited routing (e.g. DeepSeek-V3): experts are
    # partitioned into ``n_group`` contiguous groups of which only the top
    # ``topk_group`` are routable per token, so every group must retain the
    # same number of experts
    n_group, topk_group = _get_group_routing(config_data)
    grouped = n_group is not None and n_group > 1
    if grouped and not uniform:
        raise ValueError(
            f"non-uniform expert pruning is not supported for group-limited "
            f"routers (config.json n_group={n_group}): every group must "
            "retain the same number of experts, so the pruning budget "
            "cannot be redistributed across layers. Use uniform=True, "
            "which drops the same number of experts from every group"
        )
    if grouped and topk_group is None:
        raise ValueError(
            f"config.json declares n_group={n_group} but none of "
            f"{TOP_K_GROUP_CONFIG_KEYS}; cannot determine how many groups "
            "the router makes routable per token"
        )

    # collect router and expert tensor names. Routers are ordered by natural
    # (numeric) layer order to match the saliency report
    router_names = sorted(
        (n for n in weight_map if router_re.search(n)), key=_natural_sort_key
    )
    expert_names = [n for n in weight_map if expert_re.search(n)]

    if not router_names:
        raise ValueError(f"No tensors matched router_pattern {router_pattern!r}")
    if not expert_names:
        raise ValueError(f"No tensors matched expert_pattern {expert_pattern!r}")

    # associate each expert tensor with a router tensor by longest common
    # dot-separated prefix
    expert_to_router = build_expert_to_router_map(expert_names, router_names)

    # collect auxiliary router tensors (per-expert vectors and expert-index
    # tables) and associate each with its router by common prefix
    aux_names = [n for n in weight_map if aux_re and aux_re.search(n)]
    index_names = [n for n in weight_map if index_re and index_re.search(n)]
    aux_to_router = build_expert_to_router_map(
        aux_names + index_names, router_names
    )

    # a router whose layer carries an expert-index table routes statically
    # (hash routing), so its experts cannot be redistributed after pruning
    hash_routers = {aux_to_router[n] for n in index_names}

    # detect 2D vs 3D and extract expert indices for 2D
    expert_indices, is_3d = _detect_moe_layout(
        expert_names, weight_map, model_files
    )

    # score experts and determine which to retain
    if metric == "magnitude":
        scores_by_router = _magnitude_scores(router_names, weight_map, model_files)
    else:
        scores_by_router = _report_scores(
            router_names, weight_map, model_files, saliency_report_path, metric
        )

    if uniform:
        if hash_routers:
            raise ValueError(
                f"{len(hash_routers)} layer(s) route statically via an "
                f"expert-index table ({index_pattern!r}); these experts are "
                "selected discretely and cannot be redistributed after "
                "pruning. Use uniform=False to prune only the learned-routing "
                "layers while protecting the hash layers"
            )
        if grouped:
            # the drop count must split evenly across groups, which can
            # round the effective count (mirrors the in-memory REAP path)
            num_experts = next(iter(scores_by_router.values())).shape[0]
            num_prune = round(sparsity * num_experts)
            prune_per_group = round(num_prune / n_group)
            if num_prune > 0 and prune_per_group * n_group != num_prune:
                logger.warning(
                    f"group-limited routing requires an equal drop per "
                    f"group; dropping {prune_per_group * n_group} of "
                    f"{num_experts} experts instead of the requested "
                    f"{num_prune} (n_group={n_group})"
                )
        retained_experts = _compute_retained_experts(
            scores_by_router, sparsity, n_group if grouped else None
        )
    else:
        retained_experts = _compute_retained_experts_nonuniform(
            scores_by_router,
            sparsity,
            floor=num_experts_per_tok or 1,
            protected=hash_routers,
        )

    if num_experts_per_tok is not None:
        if grouped:
            # only experts of the top ``topk_group`` groups are routable
            # per token, so the floor applies to the reachable experts,
            # not the retained count of the whole layer
            k = min(
                topk_group * (len(retained) // n_group)
                for retained in retained_experts.values()
            )
        else:
            k = min(len(v) for v in retained_experts.values())
        if k < num_experts_per_tok:
            raise ValueError(
                f"sparsity={sparsity} retains {k} reachable expert(s) per "
                f"layer but num_experts_per_tok={num_experts_per_tok}; "
                "cannot prune below the per-token routing floor"
            )

    # non-uniform pruning records per-layer expert counts in the
    # quantization config, which requires placing each MoE layer at its
    # decoder layer index
    router_layer_indices: dict[str, int] = {}
    num_decoder_layers: int | None = None
    if not uniform:
        router_layer_indices = {
            rname: _extract_layer_index(rname) for rname in router_names
        }
        num_decoder_layers = _get_num_decoder_layers(
            config_data, max(router_layer_indices.values()) + 1
        )

    for rname, retained in retained_experts.items():
        logger.debug(
            f"{rname}: retaining {len(retained)}/{len(scores_by_router[rname])} "
            f"experts {retained}"
        )

    # bypass `__init__`, which is reserved to direct users to this method
    instance = cls.__new__(cls)
    instance.router_pattern = router_re
    instance.expert_pattern = expert_re
    instance.aux_pattern = aux_re
    instance.index_pattern = index_re
    instance.sparsity = sparsity
    instance.uniform = uniform
    instance.metric = metric
    instance.num_experts_config_key = num_experts_config_key
    instance.retained_experts = retained_experts
    instance.expert_to_router = expert_to_router
    instance.aux_to_router = aux_to_router
    instance.hash_routers = hash_routers
    instance.expert_indices = expert_indices
    instance.is_3d = is_3d
    instance.num_experts_per_tok = num_experts_per_tok
    instance.router_layer_indices = router_layer_indices
    instance.num_decoder_layers = num_decoder_layers
    return instance