llmcompressor.observers.fused_mappings
Classes:
-
FusedMapping–Layers that vLLM loads into a single packed weight. Packed layers must be
Functions:
-
get_fused_layers–Find the fused groups among the layers of one parent module, as defined by
-
resolve_fused_names–Resolve the fused groups among the layers present under one parent module.
FusedMapping
dataclass
FusedMapping(
fused_name: str,
names: tuple[str, ...],
optional_names: tuple[str, ...] = (),
context_names: tuple[str, ...] = (),
)
Layers that vLLM loads into a single packed weight. Packed layers must be quantized the same way and, for TENSOR_GROUP schemes (e.g. NVFP4), must share one global_scale.
Names are paths relative to the layers' common parent module. They are matched against transformers module attributes (oneshot) and against checkpoint weight names (model_free_ptq). Where the two namings differ, the fusion is listed once per naming.
Parameters:
-
fused_name(str) –name of the packed module in vLLM, for reference
-
names(tuple[str, ...]) –layers that are always packed. The mapping applies to a parent only when all of these are present
-
optional_names(tuple[str, ...], default:()) –layers that are packed only in some model configurations. When the mapping applies, those present join the group
-
context_names(tuple[str, ...], default:()) –layers that are not packed, but must be present for the mapping to apply. Use them when another model has layers with the same names that vLLM does not pack
get_fused_layers
Find the fused groups among the layers of one parent module, as defined by FUSED_MODULE_MAPPINGS.
Parameters:
-
module(Module) –parent module of the layers, e.g. an attention module
Returns:
-
list[dict[str, Module]]–one dict per fused group, mapping each layer name to its layer
Source code in src/llmcompressor/observers/fused_mappings.py
resolve_fused_names
Resolve the fused groups among the layers present under one parent module.
A mapping applies when all of its names and context_names are present,
and those of its optional_names that are present join the group. This
follows vLLM, which packs a layer into whichever fused module exists on
that parent. For example, Kimi g_proj is packed into in_proj_qkvgfab on
delta attention layers and into fused_qkv_a_g_proj on MLA layers.
Larger groups take precedence, and a group that shares a layer with an already chosen group is dropped. For example, a Kimi delta attention layer resolves to one q/k/v/b/f_a group rather than a q/k/v group.
Parameters:
-
present_names(Collection[str]) –layer names present under one parent, relative to it
Returns:
-
list[tuple[str, ...]]–disjoint groups of layer names. The first name of each group is always one of its mapping's required
names