llmcompressor.utils.pytorch.module
Utility / helper functions
Functions:
-
get_no_split_params–Get list of module classes that shouldn't be split when sharding. For
-
infer_sequential_targets–Infer or validate sequential targets for layer-wise processing.
-
qat_active–Determines if any layers in the model have quantization enabled by checking for
get_no_split_params
Get list of module classes that shouldn't be split when sharding. For Hugging Face Transformer models, this is the decoder layer type. For other types of models, this just returns all module names.
Returns:
-
str | list[str]–list of class names that shouldn't be split
Source code in src/llmcompressor/utils/pytorch/module.py
infer_sequential_targets
infer_sequential_targets(
model: Module,
sequential_targets: str | list[str] | None = None,
) -> str | list[str]
Infer or validate sequential targets for layer-wise processing.
When sequential_targets is None, automatically infers targets using get_no_split_params(). When provided as a string, wraps it in a list. Otherwise, returns the provided list as-is.
Parameters:
-
model(Module) –The model to infer targets from
-
sequential_targets(str | list[str] | None, default:None) –Optional sequential targets to use. If None, targets are inferred from the model. If a string, it's wrapped in a list.
Returns:
-
str | list[str]–List of sequential target class names or patterns
Source code in src/llmcompressor/utils/pytorch/module.py
qat_active
Determines if any layers in the model have quantization enabled by checking for weight_fake_quant attributes
Parameters:
-
module(Module) –PyTorch model to check for quantization
Returns:
-
bool–True if quantization is active anywhere in the model, False otherwise