llmcompressor.utils.helpers
General utility helper functions. Common functions for interfacing with python primitives and directories/files.
Functions:
-
DisableQuantization–Disable quantization during forward passes after applying a quantization config
-
calibration_forward_context–Context in which all calibration forward passes should occur.
-
disable_cache–Temporarily disable the key-value cache for transformer models. Used to prevent
-
disable_hf_kernels–In transformers>=4.50.0, some module forward methods may be
-
disable_lm_head–Disable the lm_head of a model by moving it to the meta device. This function
-
eval_context–Disable pytorch training mode for the given module
-
import_from_path–Import the module and the name of the function/class separated by :
-
numeric_sorted_names–Sort names by their numeric components, so that e.g.
layers.10sorts -
values_by_decoder_layer–Position values keyed by module name onto a per-decoder-layer list.
DisableQuantization
Disable quantization during forward passes after applying a quantization config
Source code in src/llmcompressor/utils/helpers.py
calibration_forward_context
Context in which all calibration forward passes should occur.
- Remove gradient calculations
- Disable the KV cache
- Disable train mode and enable eval mode
- Disable hf kernels which could bypass hooks
- Disable lm head (input and weights can still be calibrated, output will be meta)
- Optionally force eager attention
Source code in src/llmcompressor/utils/helpers.py
disable_cache
Temporarily disable the key-value cache for transformer models. Used to prevent excess memory use in one-shot cases where the model only performs the prefill phase and not the generation phase.
Example:
model = AutoModel.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0") input = torch.randint(0, 32, size=(1, 32)) with disable_cache(model): ... output = model(input)
Source code in src/llmcompressor/utils/helpers.py
disable_hf_kernels
In transformers>=4.50.0, some module forward methods may be replaced by calls to hf hub kernels. This has the potential to bypass hooks added by LLM Compressor
Source code in src/llmcompressor/utils/helpers.py
disable_lm_head
Disable the lm_head of a model by moving it to the meta device. This function does not untie parameters and restores the model proper loading upon exit
Source code in src/llmcompressor/utils/helpers.py
eval_context
Disable pytorch training mode for the given module
Source code in src/llmcompressor/utils/helpers.py
import_from_path
Import the module and the name of the function/class separated by : Examples: path = "/path/to/file.py:func_or_class_name" path = "/path/to/file:focn" path = "path.to.file:focn"
Parameters:
-
path(str) –path including the file path and object name
Source code in src/llmcompressor/utils/helpers.py
numeric_sorted_names
Sort names by their numeric components, so that e.g. layers.10 sorts
after layers.2 rather than before it (as a plain lexicographic sort
would do).
MoE layer names must be ordered like the model's layers, since per-layer lists (saliency reports, expert-sparsity configs) are matched positionally against the model's layer order.
Parameters:
-
names(Iterable[str]) –an iterable of names, or a mapping keyed by name
Returns:
-
list[str]–the names in numeric order
Source code in src/llmcompressor/utils/helpers.py
use_eager_attention
Temporarily force eager attention.
Keep this active across tracing and calibration when enabled: the attention implementation used at trace time must match the implementation used to execute the resulting graph.
Source code in src/llmcompressor/utils/helpers.py
values_by_decoder_layer
values_by_decoder_layer(
values: Mapping[str, _T], num_layers: int | None = None
) -> list[_T | None]
Position values keyed by module name onto a per-decoder-layer list.
Each name is matched against layers.(\d+) to find the decoder layer
its module belongs to, and its value is placed at that index of the
returned list; every other index holds None (e.g. for dense layers).
The result is matched positionally against the model's layer order, as
required by per-layer checkpoint fields such as
ExpertSparsityConfig.num_experts.
Parameters:
-
values(Mapping[str, _T]) –mapping from module name to that layer's value, where a value of
Nonemarks a layer which was tracked but not modified -
num_layers(int | None, default:None) –total number of decoder layers, which sizes the list. When
None, the list is instead sized to cover the largest layer index parsed fromvalues(with a warning), leaving any trailing decoder layers unrepresented
Returns:
-
list[_T | None]–a list holding each value at its layer index and
Noneeverywhere else
Raises:
-
ValueError–if a name carries no
layers.(\d+)component, if a layer index falls outside[0, num_layers), or if two names map to the same layer index -- which can happen when e.g. a vision tower holds its own MoE layers namedlayers.N