Skip to content

llmcompressor.pipelines.sequential

Modules:

Classes:

Functions:

  • handle_sequential_oom –

    Catch ooms and suggest changing sequential targets

  • trace_subgraphs –

    Trace a model to produce subgraphs, where each sequential target belongs to exactly

SequentialPipeline

Subgraph dataclass

Subgraph(
    graph: Graph,
    input_names: set[str],
    consumed_names: set[str],
    _code: PythonCode | None = None,
)

Dataclass specifying an executable subgraph of a model graph

Parameters:

  • graph (Graph) –

    subgraph of model graph

  • input_names (set[str]) –

    argument names of the compiled forward function

  • consumed_names (set[str]) –

    argument names which are not used by any subsequent subgraphs and can therefore be deleted from the intermediates cache

Methods:

  • forward –

    Execute the operations within the subgraph

  • submodule_dict –

    Return subgraph modules keyed by their fully qualified model names.

forward

forward(*args, **kwargs) -> dict[str, Any]

Execute the operations within the subgraph

Parameters:

  • \*args –

    argument inputs to subgraph forward function

  • \**kwargs –

    keyword inputs to subgraph forward function

Returns:

  • dict[str, Any] –
Source code in src/llmcompressor/pipelines/sequential/helpers.py
def forward(self, *args, **kwargs) -> dict[str, Any]:
    """
    Execute the operations within the subgraph

    :param \\*args: argument inputs to subgraph forward function
    :param \\**kwargs: keyword inputs to subgraph forward function
    :return keyword outputs of subgraph forward function (non-consumed variables):
    """
    if self._code is None:
        self._code = self.graph.python_code("self")
        exec(self._code.src, self._code.globals)

    forward_fn = self._code.globals.get("forward")

    with append_autowrap_source_on_fail():
        return forward_fn(*args, **kwargs)

submodule_dict

submodule_dict(
    model: Module, recurse: bool = True
) -> dict[str, Module]

Return subgraph modules keyed by their fully qualified model names.

Source code in src/llmcompressor/pipelines/sequential/helpers.py
def submodule_dict(self, model: Module, recurse: bool = True) -> dict[str, Module]:
    """Return subgraph modules keyed by their fully qualified model names."""
    return self._subgraph_module_dict(model, recurse=recurse)

handle_sequential_oom

handle_sequential_oom(func)

Catch ooms and suggest changing sequential targets

Source code in src/llmcompressor/pipelines/sequential/helpers.py
def handle_sequential_oom(func):
    """Catch ooms and suggest changing sequential targets"""

    @wraps(func)
    def wrapper(*args, **kwargs):
        try:
            return func(*args, **kwargs)
        except torch.OutOfMemoryError as e:
            dataset_args = kwargs.get(
                "dataset_args", args[2] if len(args) > 2 else None
            )
            targets_per_subgraph = getattr(
                dataset_args, "sequential_targets_per_subgraph", None
            )
            if targets_per_subgraph == "auto":
                grouping_hint = (
                    "sequential_targets_per_subgraph='auto' estimates how many targets "
                    "fit in free memory; lowering `gpu_memory_utilization` leaves more "
                    "headroom for what the estimate misses. "
                )
            elif isinstance(targets_per_subgraph, int) and targets_per_subgraph > 1:
                grouping_hint = (
                    "sequential_targets_per_subgraph='auto' chooses as many targets "
                    "per subgraph as fit in free memory. "
                )
            else:
                # "auto" never puts fewer than one target in a subgraph, so it cannot
                # use less memory than one target per subgraph
                grouping_hint = ""

            raise torch.OutOfMemoryError(
                "Sequential pipeline ran out of memory. "
                "Please consider choosing a smaller module for `sequential_targets`, "
                "ex. 'Linear' for dense models. "
                "For MoE models with untraceable attention, "
                "use the attention module and individual expert class instead, "
                "ex. sequential_targets=['AttentionClass', 'ExpertClass'] with "
                "sequential_targets_per_subgraph set to batch multiple experts per "
                "subgraph and reduce memory overhead. "
                f"{grouping_hint}Choosing a smaller "
                "calibration dataset can also help: reduce `max_seq_length` "
                "(when unset, calibration samples are not truncated, so a few "
                "long samples can dominate memory) or `num_calibration_samples` "
                "(memory scales with sample count for modifiers that cache "
                "activations, such as AWQ)."
            ) from e

    return wrapper

trace_subgraphs

trace_subgraphs(
    model: PreTrainedModel,
    sample_input: dict[str, Any],
    sequential_targets: list[str],
    ignore: list[str],
    targets_per_subgraph: int = 1,
    eager_attention: bool = False,
) -> list[Subgraph]

Trace a model to produce subgraphs, where each sequential target belongs to exactly one subgraph and where executing each subgraph in order is equivalent to executing the original model

Parameters:

  • model (PreTrainedModel) –

    model being traced

  • sample_input (dict[str, Any]) –

    inputs whose values will change during execution but whose len, bool, and contains values are assumed constant across batches

  • sequential_targets (list[str]) –

    list of patterns matching sequential targets. Detected attention modules are also used as targets when not contained by a requested target.

  • ignore (list[str]) –

    function and method names to skip during tracing

  • targets_per_subgraph (int, default: 1 ) –

    number of targets to include per subgraph

  • eager_attention (bool, default: False ) –

    whether to force eager attention while tracing

Returns:

  • list[Subgraph] –

    a list of Subgraphs in order of execution

Source code in src/llmcompressor/pipelines/sequential/helpers.py
def trace_subgraphs(
    model: PreTrainedModel,
    sample_input: dict[str, Any],
    sequential_targets: list[str],
    ignore: list[str],
    targets_per_subgraph: int = 1,
    eager_attention: bool = False,
) -> list[Subgraph]:
    """
    Trace a model to produce subgraphs, where each sequential target belongs to exactly
    one subgraph and where executing each subgraph in order is equivalent to executing
    the original model

    :param model: model being traced
    :param sample_input: inputs whose values will change during execution but whose
        __len__, __bool__, and __contains__ values are assumed constant across batches
    :param sequential_targets: list of patterns matching sequential targets. Detected
        attention modules are also used as targets when not contained by a requested
        target.
    :param ignore: function and method names to skip during tracing
    :param targets_per_subgraph: number of targets to include per subgraph
    :param eager_attention: whether to force eager attention while tracing
    :return: a list of Subgraphs in order of execution
    """
    graph, targets = trace_graph(
        model, sample_input, sequential_targets, ignore, eager_attention
    )
    return partition_subgraphs(model, graph, targets, targets_per_subgraph)