llmcompressor.pipelines.sequential.grouping
Estimate the memory each node of a traced graph needs while its subgraph is calibrated,
so that topological_partition can group consecutive sequential targets into
subgraphs that fit in device memory
Functions:
-
log_partitions–Log the number of targets in each subgraph, and warn about subgraphs which need
-
plan_subgraph_memory–Estimate the memory each node onloads and holds until its subgraph is compressed,
estimate_activation_memory
estimate_activation_memory(
model: Module,
shapes: InputShapes | None,
targets: set[Module] | None = None,
) -> int | None
Estimate the activation memory of a subgraph as max_batch_size * max_seq_len * width * ACTIVATION_BYTES, since subgraphs run one batch at a time
Parameters:
-
model(Module) –model being calibrated
-
shapes(InputShapes | None) –input shapes of the calibration data
-
targets(set[Module] | None, default:None) –sequential target modules, see
get_activation_width
Returns:
-
int | None–bytes, or None if there are no input shapes or the model config has no
hidden_size
Source code in src/llmcompressor/pipelines/sequential/grouping.py
estimate_cached_activations
estimate_cached_activations(
model: Module,
shapes: InputShapes,
dataset_args: DatasetArguments,
device: device,
targets: set[Module] | None = None,
) -> int
Estimate the calibration activations which the pipeline caches on device between
subgraphs: the activations of every token at the width of get_activation_width,
kept twice when the sequential error is logged. Other cached values are not
counted, such as position embeddings or attention masks
Parameters:
-
model(Module) –model being calibrated
-
shapes(InputShapes) –input shapes of the calibration data
-
dataset_args(DatasetArguments) –arguments which set where activations are cached
-
device(device) –device which subgraphs are onloaded to
-
targets(set[Module] | None, default:None) –sequential target modules, see
get_activation_width
Returns:
-
int–bytes, or 0 if activations are cached on another device or the model config has no
hidden_size
Source code in src/llmcompressor/pipelines/sequential/grouping.py
get_activation_width
Get the width per token of the activations a subgraph holds. ACTIVATION_BYTES
is measured for whole decoder layers, whose subgraphs start and end on the hidden
states. Smaller targets can start or end a subgraph on a wider intermediate, so
they are sized by the widest intermediate in the config instead
Parameters:
-
model(Module) –model being calibrated
-
targets(set[Module] | None, default:None) –sequential target modules, or None if unknown
Returns:
-
int | None–width, or None if the config has no
hidden_size
Source code in src/llmcompressor/pipelines/sequential/grouping.py
get_hidden_size
get_input_shapes
Read the shapes of the calibration batches with an extra pass over the dataloader, without running the model. Streamed (iterable) datasets are not read, since a second pass may yield other batches, or none
Parameters:
-
dataloader(DataLoader) –calibration data
Returns:
-
InputShapes | None–largest batch size, longest sequence and number of tokens including padding of the batches'
input_ids, or None if the dataset is streamed or a batch has no 2Dinput_ids
Source code in src/llmcompressor/pipelines/sequential/grouping.py
get_memory_budget
Get the device memory available to a subgraph: the fraction of total memory that calibration may use, less what is already in use by this or other processes
Parameters:
-
device(device) –device which subgraphs are onloaded to
-
gpu_memory_utilization(float) –fraction of the device's total memory that calibration may use
Returns:
-
int | None–bytes, or None if memory cannot be measured
Source code in src/llmcompressor/pipelines/sequential/grouping.py
get_module_costs
get_module_costs(
model: Module,
names: set[str],
modifiers: list[Modifier],
num_tokens: int,
) -> tuple[ModuleCosts, set[str]]
Get the memory each module onloads and holds until its subgraph is compressed, and the scratch memory needed to compress it, without onloading any weights
Parameters:
-
model(Module) –model which owns the modules
-
names(set[str]) –names of the modules to cost
-
modifiers(list[Modifier]) –modifiers which will calibrate the modules
-
num_tokens(int) –number of calibration tokens, including padding
Returns:
-
tuple[ModuleCosts, set[str]]–map of module name to (weight and held, scratch) bytes, and the names of modifiers which could not estimate their memory
Source code in src/llmcompressor/pipelines/sequential/grouping.py
get_onload_bytes
Get the device memory added by onloading a module's own parameters and buffers. Modules which are not offloaded, or are offloaded to their onload device, add nothing. Tied tensors count once per module, since each module is onloaded separately
Parameters:
-
module(Module) –module whose tensors are counted (not including submodules)
Returns:
-
int–number of bytes
Source code in src/llmcompressor/pipelines/sequential/grouping.py
log_partitions
log_partitions(
partitions: list[list[Node]],
target_nodes: set[Node],
node_memory: dict[Node, int],
max_memory: int,
)
Log the number of targets in each subgraph, and warn about subgraphs which need more memory than is available: a node which does not fit on its own, or a target which joins a costly node that started its subgraph
Parameters:
-
partitions(list[list[Node]]) –partitions produced by
topological_partition -
target_nodes(set[Node]) –nodes which call sequential targets
-
node_memory(dict[Node, int]) –map of node to bytes
-
max_memory(int) –bytes available to the nodes of a subgraph
Source code in src/llmcompressor/pipelines/sequential/grouping.py
plan_subgraph_memory
plan_subgraph_memory(
model: Module,
node_modules: dict[Node, list[str]],
dataloader: DataLoader,
modifiers: list[Modifier],
dataset_args: DatasetArguments,
targets: set[Module] | None = None,
) -> tuple[dict[Node, int], int] | None
Estimate the memory each node onloads and holds until its subgraph is compressed, and how much of it fits in a subgraph. Calibration activations cached on the device are taken out of the budget. Activations are freed before modules are compressed, so the larger of the activation estimate and the largest modifier scratch is reserved from the budget of every subgraph
Called once persistent modules are onloaded, so the memory they use is already taken out of the budget
Parameters:
-
model(Module) –model being calibrated
-
node_modules(dict[Node, list[str]]) –map of each node to the names of the modules it onloads
-
dataloader(DataLoader) –calibration data
-
modifiers(list[Modifier]) –modifiers which will calibrate the subgraphs
-
dataset_args(DatasetArguments) –arguments which set the budget and where activations are cached
-
targets(set[Module] | None, default:None) –sequential target modules, which set how wide the activations of a subgraph are (see
get_activation_width)
Returns:
-
tuple[dict[Node, int], int] | None–map of node to bytes, and the bytes available to the nodes of a subgraph, or None to fall back to one target per subgraph
Source code in src/llmcompressor/pipelines/sequential/grouping.py
52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 | |
reduce_across_ranks
reduce_across_ranks(
budget: int | None,
shapes: InputShapes | None,
device: device,
) -> tuple[int | None, InputShapes | None]
Agree on the inputs to the estimate across ranks, which hold different calibration data and may see different free memory: the smallest budget and the largest input shapes. Values which any rank could not measure become None on every rank
Parameters:
-
budget(int | None) –this rank's memory budget in bytes, or None
-
shapes(InputShapes | None) –this rank's input shapes, or None
-
device(device) –device used to communicate between ranks
Returns:
-
tuple[int | None, InputShapes | None]–budget and input shapes shared by all ranks