llmcompressor.entrypoints.oneshot
Oneshot compression entrypoint for post-training model optimization.
Provides the main oneshot compression entry point for applying quantization, pruning, and other compression techniques to pre-trained models without additional training. Supports calibration-based compression with various pipeline configurations for efficient model optimization.
Classes:
-
Oneshot–Class responsible for carrying out one-shot calibration on a pretrained model.
Functions:
-
oneshot–Performs oneshot calibration on a model.
Oneshot
Class responsible for carrying out one-shot calibration on a pretrained model.
This class handles the entire lifecycle of one-shot calibration, including preprocessing (model and tokenizer/processor initialization), model optimization (quantization or sparsification), and postprocessing (saving outputs). The instructions for model optimization can be specified by using a recipe.
-
Input Keyword Arguments:
kwargsare parsed into:model_args: Arguments for loading and configuring a pretrained model (e.g.,AutoModelForCausalLM).dataset_args: Arguments for dataset-related configurations, such as calibration dataloaders.recipe_args: Arguments for defining and configuring recipes that specify optimization actions.
Parsers are defined in
src/llmcompressor/args/. -
Lifecycle Overview: The oneshot calibration lifecycle consists of three steps:
- Preprocessing:
- Instantiates a pretrained model and tokenizer/processor.
- Ensures input and output embedding layers are untied if they share tensors.
- Patches the model to include additional functionality for saving with quantization configurations.
- Oneshot Calibration:
- Optimizes the model using a global
CompressionSessionand applies recipe-defined modifiers (e.g.,GPTQModifier,QuantizationModifier)
- Optimizes the model using a global
- Postprocessing:
- Saves the model, tokenizer/processor, and configuration to the specified
output_dir.
- Saves the model, tokenizer/processor, and configuration to the specified
- Preprocessing:
-
Usage:
Methods:
init(**kwargs):
Initializes the Oneshot object by parsing input arguments, performing
preprocessing, and setting instance attributes.
__call__(**kwargs):
Performs the one-shot calibration process by preparing a calibration
dataloader, applying recipe modifiers to the model, and executing
postprocessing steps.
save():
Saves the calibrated model and tokenizer/processor to the specified
`output_dir`. Supports saving in compressed formats based on model
arguments.
apply_recipe_modifiers(calibration_dataloader, **kwargs):
Applies lifecycle actions (e.g., `initialize`, `finalize`) using modifiers
defined in the recipe. Each action is executed via the global
`CompressionSession`.
Initializes the Oneshot class with provided arguments.
Parses the input keyword arguments into model_args, dataset_args, and
recipe_args. Performs preprocessing to initialize the model and
tokenizer/processor.
Parameters:
-
model_args–ModelArguments parameters, responsible for controlling model loading and saving logic
-
dataset_args–DatasetArguments parameters, responsible for controlling dataset loading, preprocessing and dataloader loading
-
recipe_args–RecipeArguments parameters, responsible for containing recipe-related parameters
-
output_dir–Path to save the output model after carrying out oneshot
-
log_dir(str | None, default:None) –Path to save logs during oneshot run. Nothing is logged to file if None.
Methods:
-
apply_recipe_modifiers–Applies recipe modifiers to the model during the lifecycle.
-
handle_error–Decorator which removes the file log handler added for this run if
func -
validate_model–Validate that oneshot can be applied to model.
Source code in src/llmcompressor/entrypoints/oneshot.py
148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 | |
apply_recipe_modifiers
apply_recipe_modifiers(
calibration_dataloader: DataLoader | None,
recipe_stage: str | None = None,
)
Applies recipe modifiers to the model during the lifecycle.
The modifiers are defined in the recipe and executed via lifecycle actions
(initialize, finalize) through the global CompressionSession.
Source code in src/llmcompressor/entrypoints/oneshot.py
handle_error
staticmethod
Decorator which removes the file log handler added for this run if func
raises, so that later runs in the same process do not write to this run's
log file
Source code in src/llmcompressor/entrypoints/oneshot.py
validate_model
Validate that oneshot can be applied to model. Raise warning if model is quantized with compressed-tensors quant method. Raise error if model is quantized with any other quant method.
Source code in src/llmcompressor/entrypoints/oneshot.py
oneshot
oneshot(
model: str | PreTrainedModel,
config_name: str | None = None,
tokenizer: str | PreTrainedTokenizerBase | None = None,
processor: str | ProcessorMixin | None = None,
precision: str = "auto",
tie_word_embeddings: bool = True,
trust_remote_code_model: bool = False,
save_compressed: bool = True,
model_revision: str = "main",
recipe: RecipeInput | None = None,
recipe_args: list[str] | None = None,
clear_sparse_session: bool = False,
stage: str | None = None,
dataset: str
| Dataset
| DatasetDict
| DataLoader
| None = None,
dataset_config_name: str | None = None,
dataset_path: str | None = None,
splits: str | list[str] | dict[str, str] | None = None,
batch_size: int = 1,
data_collator: str | Callable = "truncation",
num_calibration_samples: int = 512,
shuffle_calibration_samples: bool = True,
max_seq_length: int | None = None,
pad_to_max_length: bool = False,
text_column: str = "text",
concatenate_data: bool = False,
streaming: bool = False,
overwrite_cache: bool = False,
preprocessing_num_workers: int | None = None,
dataloader_num_workers: int = 0,
min_tokens_per_module: float | None = None,
moe_calibrate_all_experts: bool = True,
pipeline: str | None = "independent",
use_eager_attention: bool = False,
tracing_ignore: list[str] = [
"_update_causal_mask",
"create_causal_mask",
"_update_mamba_mask",
"make_causal_mask",
"get_causal_mask",
"mask_interface",
"mask_function",
"_prepare_4d_causal_attention_mask",
"_prepare_fsmt_decoder_inputs",
"_prepare_4d_causal_attention_mask_with_cache_position",
"_update_linear_attn_mask",
"project_per_layer_inputs",
"_apply_attn_res",
],
sequential_targets: list[str] | None = None,
sequential_offload_device: str = "cpu",
quantization_aware_calibration: bool = True,
stage_weights_in_pinned_memory: bool = False,
log_sequential_error: bool = False,
output_dir: str | None = None,
log_dir: str | None = None,
**kwargs,
) -> PreTrainedModel
Performs oneshot calibration on a model.
Model arguments
Parameters:
-
model(str | PreTrainedModel) –A pretrained model identifier from huggingface.co/models or a path to a local model. Required parameter.
-
distill_teacher–Teacher model (a trained text generation model) for distillation.
-
config_name(str | None, default:None) –Pretrained config name or path if not the same as model_name.
-
tokenizer(str | PreTrainedTokenizerBase | None, default:None) –Pretrained tokenizer name or path if not the same as model_name.
-
processor(str | ProcessorMixin | None, default:None) –Pretrained processor name or path if not the same as model_name.
-
precision(str, default:'auto') –Precision to cast model weights to, default to auto.
-
tie_word_embeddings(bool, default:True) –Whether the model's input and output word embeddings should be left tied if possible. False means always untie.
-
trust_remote_code_model(bool, default:False) –Whether to allow for custom models to execute their own modeling files.
-
save_compressed(bool, default:True) –Whether to compress quantized weights during save.
-
model_revision(str, default:'main') –The specific model version to use (can be branch name, tag, or commit id).
Recipe arguments
-
recipe(RecipeInput | None, default:None) –A LLM Compressor recipe. Accepts a path (or list of paths) to recipe YAML file(s), a Modifier instance (or list), or a Recipe object (or list).
-
recipe_args(list[str] | None, default:None) –List of recipe arguments to evaluate, in the format "key1=value1", "key2=value2".
-
clear_sparse_session(bool, default:False) –Whether to clear CompressionSession/ CompressionLifecycle data between runs.
-
stage(str | None, default:None) –The stage of the recipe to use for oneshot.
Dataset arguments
-
dataset(str | Dataset | DatasetDict | DataLoader | None, default:None) –The dataset to use for calibration. Can be a dataset name (str, via the datasets library), a HuggingFace Dataset or DatasetDict, or a pre-built PyTorch DataLoader. When a DataLoader is passed, the internal dataset-to-dataloader conversion is skipped.
-
dataset_config_name(str | None, default:None) –The configuration name of the dataset to use.
-
dataset_path(str | None, default:None) –Path to a custom dataset. Supports json, csv, dvc.
-
splits(str | list[str] | dict[str, str] | None, default:None) –Optional percentages of each split to download.
-
batch_size(int, default:1) –calibration dataset batch size. During calibration, LLM Compressor disables lm_head output computations to reduce memory usage from large calibration batch sizes. Large batch sizes may result excess padding or truncation, depending on the data_collator
-
data_collator(str | Callable, default:'truncation') –The function to use to form a batch from the dataset. Can also specify 'truncation' or 'padding' to truncate or pad non-uniform sequence lengths in a batch. Defaults to 'truncation'.
-
num_calibration_samples(int, default:512) –Number of samples to use for one-shot calibration.
-
shuffle_calibration_samples(bool, default:True) –Whether to shuffle the dataset before calibration.
-
max_seq_length(int | None, default:None) –Maximum total input sequence length after tokenization.
-
pad_to_max_length(bool, default:False) –Whether to pad all samples to
max_seq_length. -
text_column(str, default:'text') –Key to use as the
textinput to tokenizer/processor. -
concatenate_data(bool, default:False) –Whether to concatenate datapoints to fill max_seq_length.
-
streaming(bool, default:False) –True to stream data from a cloud dataset.
-
overwrite_cache(bool, default:False) –Whether to overwrite the cached preprocessed datasets.
-
preprocessing_num_workers(int | None, default:None) –Number of processes for dataset preprocessing.
-
dataloader_num_workers(int, default:0) –Number of worker processes for data loading. Default is 0 (safe for low CPU/GPU memory). Set to 2 or more for faster calibration if you have sufficient RAM. Custom data collators may not work with multiprocessing.
-
min_tokens_per_module(float | None, default:None) –Minimum percentage of tokens per module, relevant for MoE models.
-
moe_calibrate_all_experts(bool, default:True) –Whether to calibrate all experts during MoE model calibration. When True, all experts will see all tokens during calibration, ensuring proper quantization statistics. When False, only routed experts will be used. Only relevant for MoE models. Default is True.
-
pipeline(str | None, default:'independent') –Calibration pipeline used to calibrate model Options: ['basic', 'datafree', 'sequential', 'independent']
-
use_eager_attention(bool, default:False) –Whether to force eager attention during tracing and calibration. By default, the model's configured attention implementation is used for both.
-
tracing_ignore(list[str], default:['_update_causal_mask', 'create_causal_mask', '_update_mamba_mask', 'make_causal_mask', 'get_causal_mask', 'mask_interface', 'mask_function', '_prepare_4d_causal_attention_mask', '_prepare_fsmt_decoder_inputs', '_prepare_4d_causal_attention_mask_with_cache_position', '_update_linear_attn_mask', 'project_per_layer_inputs', '_apply_attn_res']) –List of functions to ignore during tracing, either {module}.{method_name} or {function_name}
-
sequential_targets(list[str] | None, default:None) –List of layer targets for the sequential pipeline. This is typically a single DecoderLayer. Not specifying this argument will cause the sequential pipeline to default to using the
no_split_paramsspecified by the HF model definition -
sequential_offload_device(str, default:'cpu') –Device used to offload intermediate activations between sequential layers. It is recommended to use
cuda:1if using more than one gpu. Default is cpu. -
quantization_aware_calibration(bool, default:True) –Deprecated. This argument has no effect and will be removed in a future release.
-
stage_weights_in_pinned_memory(bool, default:False) –When using the sequential pipeline, stage offloaded module tensors in pinned CPU memory before onloading them to the execution device. Default False.
-
log_sequential_error(bool, default:False) –Only relevant for the sequential pipeline. If True, compute and log the SQNR between each subgraph's pre-compression and post-compression outputs, independent of propagate_error. Default is False.
Miscellaneous arguments
-
output_dir(str | None, default:None) –Path to save the output model after calibration. Nothing is saved if None.
-
log_dir(str | None, default:None) –Path to save logs during oneshot run. Nothing is logged to file if None.
Returns:
-
PreTrainedModel–The calibrated PreTrainedModel
Source code in src/llmcompressor/entrypoints/oneshot.py
409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 | |