llmcompressor.pipelines.sequential
Modules:
-
ast_helpers– -
ast_utils– -
error_logging– -
grouping–Estimate the memory each node of a traced graph needs while its subgraph is calibrated,
-
helpers– -
pipeline– -
transformers_helpers–
Classes:
-
SequentialPipeline– -
Subgraph–Dataclass specifying an executable subgraph of a model graph
Functions:
-
handle_sequential_oom–Catch ooms and suggest changing sequential targets
-
trace_subgraphs–Trace a model to produce subgraphs, where each sequential target belongs to exactly
SequentialPipeline
Bases: CalibrationPipeline
Subgraph
dataclass
Subgraph(
graph: Graph,
input_names: set[str],
consumed_names: set[str],
_code: PythonCode | None = None,
)
Dataclass specifying an executable subgraph of a model graph
Parameters:
-
graph(Graph) –subgraph of model graph
-
input_names(set[str]) –argument names of the compiled forward function
-
consumed_names(set[str]) –argument names which are not used by any subsequent subgraphs and can therefore be deleted from the intermediates cache
Methods:
-
forward–Execute the operations within the subgraph
-
submodule_dict–Return subgraph modules keyed by their fully qualified model names.
forward
Execute the operations within the subgraph
Parameters:
-
\*args–argument inputs to subgraph forward function
-
\**kwargs–keyword inputs to subgraph forward function
Returns:
-
dict[str, Any]–
Source code in src/llmcompressor/pipelines/sequential/helpers.py
submodule_dict
Return subgraph modules keyed by their fully qualified model names.
handle_sequential_oom
Catch ooms and suggest changing sequential targets
Source code in src/llmcompressor/pipelines/sequential/helpers.py
trace_subgraphs
trace_subgraphs(
model: PreTrainedModel,
sample_input: dict[str, Any],
sequential_targets: list[str],
ignore: list[str],
targets_per_subgraph: int = 1,
eager_attention: bool = False,
) -> list[Subgraph]
Trace a model to produce subgraphs, where each sequential target belongs to exactly one subgraph and where executing each subgraph in order is equivalent to executing the original model
Parameters:
-
model(PreTrainedModel) –model being traced
-
sample_input(dict[str, Any]) –inputs whose values will change during execution but whose len, bool, and contains values are assumed constant across batches
-
sequential_targets(list[str]) –list of patterns matching sequential targets. Detected attention modules are also used as targets when not contained by a requested target.
-
ignore(list[str]) –function and method names to skip during tracing
-
targets_per_subgraph(int, default:1) –number of targets to include per subgraph
-
eager_attention(bool, default:False) –whether to force eager attention while tracing
Returns:
-
list[Subgraph]–a list of Subgraphs in order of execution