llmcompressor.pipelines
Compression pipelines for orchestrating different compression strategies.
Provides various compression pipelines including basic, sequential, independent, layer-sequential, and data-free approaches. Each pipeline coordinates different compression techniques and workflows for optimal model optimization based on specific requirements and constraints.
Modules:
-
basic– -
cache– -
data_free– -
independent– -
registry– -
sequential–
Classes:
-
BasicPipeline– -
CalibrationPipeline– -
DataFreePipeline– -
IndependentPipeline– -
SequentialPipeline– -
Subgraph–Dataclass specifying an executable subgraph of a model graph
Functions:
-
handle_sequential_oom–Catch ooms and suggest changing sequential targets
-
trace_subgraphs–Trace a model to produce subgraphs, where each sequential target belongs to exactly
BasicPipeline
Bases: CalibrationPipeline
CalibrationPipeline
Bases: ABC, RegistryMixin
Methods:
-
from_modifiers–Infer which calibration pipeline to use based on the available modifiers and
from_modifiers
classmethod
Infer which calibration pipeline to use based on the available modifiers and any user specifications
Parameters:
-
modifiers(list[Modifier]) –modifiers to apply to model
-
user(str | None, default:None) –pipeline name passed by user
Returns:
-
CalibrationPipeline–CalibrationPipeline instance to be called with data (if not datafree)
Source code in src/llmcompressor/pipelines/registry.py
DataFreePipeline
Bases: CalibrationPipeline
IndependentPipeline
Bases: CalibrationPipeline
SequentialPipeline
Bases: CalibrationPipeline
Subgraph
dataclass
Subgraph(
graph: Graph,
input_names: set[str],
consumed_names: set[str],
_code: PythonCode | None = None,
)
Dataclass specifying an executable subgraph of a model graph
Parameters:
-
graph(Graph) –subgraph of model graph
-
input_names(set[str]) –argument names of the compiled forward function
-
consumed_names(set[str]) –argument names which are not used by any subsequent subgraphs and can therefore be deleted from the intermediates cache
Methods:
-
forward–Execute the operations within the subgraph
-
submodule_dict–Return subgraph modules keyed by their fully qualified model names.
forward
Execute the operations within the subgraph
Parameters:
-
\*args–argument inputs to subgraph forward function
-
\**kwargs–keyword inputs to subgraph forward function
Returns:
-
dict[str, Any]–
Source code in src/llmcompressor/pipelines/sequential/helpers.py
submodule_dict
Return subgraph modules keyed by their fully qualified model names.
handle_sequential_oom
Catch ooms and suggest changing sequential targets
Source code in src/llmcompressor/pipelines/sequential/helpers.py
trace_subgraphs
trace_subgraphs(
model: PreTrainedModel,
sample_input: dict[str, Any],
sequential_targets: list[str],
ignore: list[str],
targets_per_subgraph: int = 1,
eager_attention: bool = False,
) -> list[Subgraph]
Trace a model to produce subgraphs, where each sequential target belongs to exactly one subgraph and where executing each subgraph in order is equivalent to executing the original model
Parameters:
-
model(PreTrainedModel) –model being traced
-
sample_input(dict[str, Any]) –inputs whose values will change during execution but whose len, bool, and contains values are assumed constant across batches
-
sequential_targets(list[str]) –list of patterns matching sequential targets. Detected attention modules are also used as targets when not contained by a requested target.
-
ignore(list[str]) –function and method names to skip during tracing
-
targets_per_subgraph(int, default:1) –number of targets to include per subgraph
-
eager_attention(bool, default:False) –whether to force eager attention while tracing
Returns:
-
list[Subgraph]–a list of Subgraphs in order of execution