Skip to content

llmcompressor.observers.fused_mappings

Classes:

  • FusedMapping –

    Layers that vLLM loads into a single packed weight. Packed layers must be

Functions:

  • get_fused_layers –

    Find the fused groups among the layers of one parent module, as defined by

  • resolve_fused_names –

    Resolve the fused groups among the layers present under one parent module.

FusedMapping dataclass

FusedMapping(
    fused_name: str,
    names: tuple[str, ...],
    optional_names: tuple[str, ...] = (),
    context_names: tuple[str, ...] = (),
)

Layers that vLLM loads into a single packed weight. Packed layers must be quantized the same way and, for TENSOR_GROUP schemes (e.g. NVFP4), must share one global_scale.

Names are paths relative to the layers' common parent module. They are matched against transformers module attributes (oneshot) and against checkpoint weight names (model_free_ptq). Where the two namings differ, the fusion is listed once per naming.

Parameters:

  • fused_name (str) –

    name of the packed module in vLLM, for reference

  • names (tuple[str, ...]) –

    layers that are always packed. The mapping applies to a parent only when all of these are present

  • optional_names (tuple[str, ...], default: () ) –

    layers that are packed only in some model configurations. When the mapping applies, those present join the group

  • context_names (tuple[str, ...], default: () ) –

    layers that are not packed, but must be present for the mapping to apply. Use them when another model has layers with the same names that vLLM does not pack

get_fused_layers

get_fused_layers(module: Module) -> list[dict[str, Module]]

Find the fused groups among the layers of one parent module, as defined by FUSED_MODULE_MAPPINGS.

Parameters:

  • module (Module) –

    parent module of the layers, e.g. an attention module

Returns:

  • list[dict[str, Module]] –

    one dict per fused group, mapping each layer name to its layer

Source code in src/llmcompressor/observers/fused_mappings.py
def get_fused_layers(module: Module) -> list[dict[str, Module]]:
    """
    Find the fused groups among the layers of one parent module, as defined by
    FUSED_MODULE_MAPPINGS.

    :param module: parent module of the layers, e.g. an attention module
    :return: one dict per fused group, mapping each layer name to its layer
    """
    present_layers = {}
    for name in FUSED_MODULE_NAMES:
        layer = module
        for part in name.split("."):
            layer = getattr(layer, part, None)
        if isinstance(layer, Module):
            present_layers[name] = layer

    return [
        {name: present_layers[name] for name in group}
        for group in resolve_fused_names(present_layers.keys())
    ]

resolve_fused_names

resolve_fused_names(
    present_names: Collection[str],
) -> list[tuple[str, ...]]

Resolve the fused groups among the layers present under one parent module.

A mapping applies when all of its names and context_names are present, and those of its optional_names that are present join the group. This follows vLLM, which packs a layer into whichever fused module exists on that parent. For example, Kimi g_proj is packed into in_proj_qkvgfab on delta attention layers and into fused_qkv_a_g_proj on MLA layers.

Larger groups take precedence, and a group that shares a layer with an already chosen group is dropped. For example, a Kimi delta attention layer resolves to one q/k/v/b/f_a group rather than a q/k/v group.

Parameters:

  • present_names (Collection[str]) –

    layer names present under one parent, relative to it

Returns:

  • list[tuple[str, ...]] –

    disjoint groups of layer names. The first name of each group is always one of its mapping's required names

Source code in src/llmcompressor/observers/fused_mappings.py
def resolve_fused_names(present_names: Collection[str]) -> list[tuple[str, ...]]:
    """
    Resolve the fused groups among the layers present under one parent module.

    A mapping applies when all of its `names` and `context_names` are present,
    and those of its `optional_names` that are present join the group. This
    follows vLLM, which packs a layer into whichever fused module exists on
    that parent. For example, Kimi `g_proj` is packed into `in_proj_qkvgfab` on
    delta attention layers and into `fused_qkv_a_g_proj` on MLA layers.

    Larger groups take precedence, and a group that shares a layer with an
    already chosen group is dropped. For example, a Kimi delta attention layer
    resolves to one q/k/v/b/f_a group rather than a q/k/v group.

    :param present_names: layer names present under one parent, relative to it
    :return: disjoint groups of layer names. The first name of each group is
        always one of its mapping's required `names`
    """
    candidates: list[tuple[str, ...]] = []
    for mapping in FUSED_MODULE_MAPPINGS:
        if all(n in present_names for n in mapping.names + mapping.context_names):
            optional = tuple(n for n in mapping.optional_names if n in present_names)
            candidates.append(mapping.names + optional)

    # stable sort: larger groups first, ties keep FUSED_MODULE_MAPPINGS order
    candidates.sort(key=len, reverse=True)

    groups: list[tuple[str, ...]] = []
    claimed: set[str] = set()
    for group in candidates:
        if claimed.isdisjoint(group):
            claimed.update(group)
            groups.append(group)
    return groups