Skip to content

llmcompressor.pipelines

Compression pipelines for orchestrating different compression strategies.

Provides various compression pipelines including basic, sequential, independent, layer-sequential, and data-free approaches. Each pipeline coordinates different compression techniques and workflows for optimal model optimization based on specific requirements and constraints.

Modules:

Classes:

Functions:

  • handle_sequential_oom –

    Catch ooms and suggest changing sequential targets

  • trace_subgraphs –

    Trace a model to produce subgraphs, where each sequential target belongs to exactly

BasicPipeline

CalibrationPipeline

Bases: ABC, RegistryMixin

Methods:

  • from_modifiers –

    Infer which calibration pipeline to use based on the available modifiers and

from_modifiers classmethod

from_modifiers(
    modifiers: list[Modifier], user: str | None = None
) -> CalibrationPipeline

Infer which calibration pipeline to use based on the available modifiers and any user specifications

Parameters:

  • modifiers (list[Modifier]) –

    modifiers to apply to model

  • user (str | None, default: None ) –

    pipeline name passed by user

Returns:

  • CalibrationPipeline –

    CalibrationPipeline instance to be called with data (if not datafree)

Source code in src/llmcompressor/pipelines/registry.py
@classmethod
def from_modifiers(
    cls, modifiers: list[Modifier], user: str | None = None
) -> "CalibrationPipeline":
    """
    Infer which calibration pipeline to use based on the available modifiers and
    any user specifications

    :param modifiers: modifiers to apply to model
    :param user: pipeline name passed by user
    :return: CalibrationPipeline instance to be called with data (if not datafree)
    """
    user = standardize_lookup_name(user) if user else None
    inferred = standardize_lookup_name(cls._infer_pipeline(modifiers))
    independent = standardize_lookup_name("independent")

    if user == independent:
        inferred = independent

    if user is not None and user != inferred:
        logger.warning(
            f"Calibration pipeline is set to `{user}`, but it is recommended to "
            f"use `{inferred}`"
        )

    pipeline = user or inferred
    return cls.load_from_registry(pipeline)

DataFreePipeline

IndependentPipeline

SequentialPipeline

Subgraph dataclass

Subgraph(
    graph: Graph,
    input_names: set[str],
    consumed_names: set[str],
    _code: PythonCode | None = None,
)

Dataclass specifying an executable subgraph of a model graph

Parameters:

  • graph (Graph) –

    subgraph of model graph

  • input_names (set[str]) –

    argument names of the compiled forward function

  • consumed_names (set[str]) –

    argument names which are not used by any subsequent subgraphs and can therefore be deleted from the intermediates cache

Methods:

  • forward –

    Execute the operations within the subgraph

  • submodule_dict –

    Return subgraph modules keyed by their fully qualified model names.

forward

forward(*args, **kwargs) -> dict[str, Any]

Execute the operations within the subgraph

Parameters:

  • \*args –

    argument inputs to subgraph forward function

  • \**kwargs –

    keyword inputs to subgraph forward function

Returns:

  • dict[str, Any] –
Source code in src/llmcompressor/pipelines/sequential/helpers.py
def forward(self, *args, **kwargs) -> dict[str, Any]:
    """
    Execute the operations within the subgraph

    :param \\*args: argument inputs to subgraph forward function
    :param \\**kwargs: keyword inputs to subgraph forward function
    :return keyword outputs of subgraph forward function (non-consumed variables):
    """
    if self._code is None:
        self._code = self.graph.python_code("self")
        exec(self._code.src, self._code.globals)

    forward_fn = self._code.globals.get("forward")

    with append_autowrap_source_on_fail():
        return forward_fn(*args, **kwargs)

submodule_dict

submodule_dict(
    model: Module, recurse: bool = True
) -> dict[str, Module]

Return subgraph modules keyed by their fully qualified model names.

Source code in src/llmcompressor/pipelines/sequential/helpers.py
def submodule_dict(self, model: Module, recurse: bool = True) -> dict[str, Module]:
    """Return subgraph modules keyed by their fully qualified model names."""
    return self._subgraph_module_dict(model, recurse=recurse)

handle_sequential_oom

handle_sequential_oom(func)

Catch ooms and suggest changing sequential targets

Source code in src/llmcompressor/pipelines/sequential/helpers.py
def handle_sequential_oom(func):
    """Catch ooms and suggest changing sequential targets"""

    @wraps(func)
    def wrapper(*args, **kwargs):
        try:
            return func(*args, **kwargs)
        except torch.OutOfMemoryError as e:
            dataset_args = kwargs.get(
                "dataset_args", args[2] if len(args) > 2 else None
            )
            targets_per_subgraph = getattr(
                dataset_args, "sequential_targets_per_subgraph", None
            )
            if targets_per_subgraph == "auto":
                grouping_hint = (
                    "sequential_targets_per_subgraph='auto' estimates how many targets "
                    "fit in free memory; lowering `gpu_memory_utilization` leaves more "
                    "headroom for what the estimate misses. "
                )
            elif isinstance(targets_per_subgraph, int) and targets_per_subgraph > 1:
                grouping_hint = (
                    "sequential_targets_per_subgraph='auto' chooses as many targets "
                    "per subgraph as fit in free memory. "
                )
            else:
                # "auto" never puts fewer than one target in a subgraph, so it cannot
                # use less memory than one target per subgraph
                grouping_hint = ""

            raise torch.OutOfMemoryError(
                "Sequential pipeline ran out of memory. "
                "Please consider choosing a smaller module for `sequential_targets`, "
                "ex. 'Linear' for dense models. "
                "For MoE models with untraceable attention, "
                "use the attention module and individual expert class instead, "
                "ex. sequential_targets=['AttentionClass', 'ExpertClass'] with "
                "sequential_targets_per_subgraph set to batch multiple experts per "
                "subgraph and reduce memory overhead. "
                f"{grouping_hint}Choosing a smaller "
                "calibration dataset can also help: reduce `max_seq_length` "
                "(when unset, calibration samples are not truncated, so a few "
                "long samples can dominate memory) or `num_calibration_samples` "
                "(memory scales with sample count for modifiers that cache "
                "activations, such as AWQ)."
            ) from e

    return wrapper

trace_subgraphs

trace_subgraphs(
    model: PreTrainedModel,
    sample_input: dict[str, Any],
    sequential_targets: list[str],
    ignore: list[str],
    targets_per_subgraph: int = 1,
    eager_attention: bool = False,
) -> list[Subgraph]

Trace a model to produce subgraphs, where each sequential target belongs to exactly one subgraph and where executing each subgraph in order is equivalent to executing the original model

Parameters:

  • model (PreTrainedModel) –

    model being traced

  • sample_input (dict[str, Any]) –

    inputs whose values will change during execution but whose len, bool, and contains values are assumed constant across batches

  • sequential_targets (list[str]) –

    list of patterns matching sequential targets. Detected attention modules are also used as targets when not contained by a requested target.

  • ignore (list[str]) –

    function and method names to skip during tracing

  • targets_per_subgraph (int, default: 1 ) –

    number of targets to include per subgraph

  • eager_attention (bool, default: False ) –

    whether to force eager attention while tracing

Returns:

  • list[Subgraph] –

    a list of Subgraphs in order of execution

Source code in src/llmcompressor/pipelines/sequential/helpers.py
def trace_subgraphs(
    model: PreTrainedModel,
    sample_input: dict[str, Any],
    sequential_targets: list[str],
    ignore: list[str],
    targets_per_subgraph: int = 1,
    eager_attention: bool = False,
) -> list[Subgraph]:
    """
    Trace a model to produce subgraphs, where each sequential target belongs to exactly
    one subgraph and where executing each subgraph in order is equivalent to executing
    the original model

    :param model: model being traced
    :param sample_input: inputs whose values will change during execution but whose
        __len__, __bool__, and __contains__ values are assumed constant across batches
    :param sequential_targets: list of patterns matching sequential targets. Detected
        attention modules are also used as targets when not contained by a requested
        target.
    :param ignore: function and method names to skip during tracing
    :param targets_per_subgraph: number of targets to include per subgraph
    :param eager_attention: whether to force eager attention while tracing
    :return: a list of Subgraphs in order of execution
    """
    graph, targets = trace_graph(
        model, sample_input, sequential_targets, ignore, eager_attention
    )
    return partition_subgraphs(model, graph, targets, targets_per_subgraph)