Skip to content

llmcompressor.modifiers.quantization

Modules:

Classes:

Functions:

  • quantize_weight –

    Quantize a batch of weights according to the GPTQ algorithm.

GPTQModifier

Bases: Modifier, QuantizationMixin

Implements the GPTQ algorithm from https://arxiv.org/abs/2210.17323. This modifier uses activations to calibrate a hessian matrix, which is then used to determine optimal quantization values and orderings for the model weights.

Sample yaml:

test_stage:
  obcq_modifiers:
    GPTQModifier:
      block_size: 128
      dampening_frac: 0.001
      actorder: static
      config_groups:
        group_0:
          targets:
            - "Linear"
          input_activations: null
          output_activations: null
          weights:
            num_bits: 8
            type: "int"
            symmetric: true
            strategy: group
            group_size: 128

Lifecycle:

  • on_initialize
    • apply config to model
  • on_calibration_start
    • add activation calibration hooks
    • add gptq weight calibration hooks
  • on_sequential_epoch_end
    • quantize_weight
  • on_calibration_end
    • remove_hooks()
    • model.apply(freeze_module_quantization)

Parameters:

  • block_size –

    Used to determine number of columns to compress in one pass

  • dampening_frac –

    Amount of dampening to apply to H, as a fraction of the diagonal norm

  • actorder –

    order in which weight columns are quantized. Defaults to "static" activation ordering, which achieves best accuracy recovery with no runtime cost. For more information, see https://github.com/vllm-project/vllm/pull/8135.

  • batched_quantization –

    Controls batching of same-shape modules (e.g. linearized MoE experts). "auto" (the default) limits batches to 75% of available CUDA memory. A positive integer sets a maximum batch size without consulting available memory. None disables batching.

  • config_groups –

    dictionary specifying quantization schemes to apply to target modules. Modules not matching a scheme target will NOT be quantized.

  • targets –

    list of layer names to quantize if a scheme is provided. Defaults to Linear layers

  • ignore –

    optional list of module class names or submodule names to not quantize even if they match a target in config_groups. Defaults to empty list.

  • scheme –

    a single quantization scheme to apply to the model. This is a dictionary that supports all keys from QuantizationScheme except targets, which will be set to the targets parameter set at the modifier level. Can also be set to a dictionary of the format preset_scheme_name: targets for example: W8A8: ['Linear'] for weight and activation 8-bit.

  • kv_cache_scheme –

    optional QuantizationArgs, that specify the quantization of the kv cache. If None, kv cache is not quantized. When applying kv cache quantization to transformer AutoModelForCausalLM, the kv_cache_scheme gets converted into a QuantizationScheme that: - targets the q_proj and k_proj modules of the model. The outputs of those modules are the keys and values that might be cached - quantizes the outputs of the aforementioned layers, so that keys and values are compressed before storing them in the cache There is an explicit assumption that the model contains modules with k_proj and v_proj in their names. If this is not the case and kv_cache_scheme != None, the quantization of kv cache will fail

Methods:

  • calibrate_module –

    Calibration hook used to accumulate the hessian of the input to the module

  • compress_modules –

    Quantize modules which have been calibrated

  • estimate_sequential_memory –

    Estimate the Hessian held for a module until its subgraph is compressed, and

  • on_calibration_end –

    Finish calibrating by removing observers and calibration hooks

  • on_finalize –

    disable the quantization observers used by the OBCQ algorithm

  • on_initialize –

    Initialize and run the GPTQ algorithm on the current state

calibrate_module

calibrate_module(
    module: Module,
    args: tuple[Tensor, ...],
    _output: Tensor,
)

Calibration hook used to accumulate the hessian of the input to the module

Parameters:

  • module (Module) –

    module being calibrated

  • args (tuple[Tensor, ...]) –

    inputs to the module, the first element of which is the canonical input

  • _output (Tensor) –

    uncompressed module output, unused

Source code in src/llmcompressor/modifiers/gptq/base.py
def calibrate_module(
    self,
    module: torch.nn.Module,
    args: tuple[torch.Tensor, ...],
    _output: torch.Tensor,
):
    """
    Calibration hook used to accumulate the hessian of the input to the module

    :param module: module being calibrated
    :param args: inputs to the module, the first element of which is the
        canonical input
    :param _output: uncompressed module output, unused
    """
    # Assume that first argument is the input
    inp = args[0]

    # Initialize hessian if not present
    if module not in self._num_samples:
        self._hessians[module] = make_empty_hessian(
            module, device=get_execution_device(module)
        )
        self._num_samples[module] = torch.zeros(
            tuple(), device=get_execution_device(module)
        )

    self._hessians[module], self._num_samples[module] = accumulate_hessian(
        inp,
        module,
        self._hessians[module],
        self._num_samples[module],
    )

compress_modules

compress_modules()

Quantize modules which have been calibrated

Source code in src/llmcompressor/modifiers/gptq/base.py
def compress_modules(self):
    """
    Quantize modules which have been calibrated
    """
    ### Not Distributed
    if not is_distributed():
        self.compress_module_list(list(self._num_samples.keys()))
        return

    ### Distributed
    rank = dist.get_rank()
    world_size = dist.get_world_size()

    # Assign modules to ranks
    module_list, rank_to_modules, module_to_rank = greedy_bin_packing(
        list(self._hessians.keys()),
        world_size,
        item_weight_fn=lambda mod: self._hessians[mod].shape[0],
    )

    # send hessians to assigned ranks
    self._reduce_hessian_to_target_rank(module_list, module_to_rank)

    self.compress_module_list(rank_to_modules[rank])

    # broadcast compressed modules to each rank
    broadcast_qparams_and_cleanup(module_list, module_to_rank, _GPTQ_Q_PARAMS)

estimate_sequential_memory

estimate_sequential_memory(
    module: Module, num_tokens: int
) -> tuple[int, int]

Estimate the Hessian held for a module until its subgraph is compressed, and the temporary memory needed to compress it

Parameters:

  • module (Module) –

    module belonging to the subgraph being calibrated

  • num_tokens (int) –

    number of calibration tokens, unused

Returns:

  • tuple[int, int] –

    tuple of (held, scratch) bytes

Source code in src/llmcompressor/modifiers/gptq/base.py
def estimate_sequential_memory(
    self, module: torch.nn.Module, num_tokens: int
) -> tuple[int, int]:
    """
    Estimate the Hessian held for a module until its subgraph is compressed, and
    the temporary memory needed to compress it

    :param module: module belonging to the subgraph being calibrated
    :param num_tokens: number of calibration tokens, unused
    :return: tuple of (held, scratch) bytes
    """
    if not self._is_calibrated(module):
        return 0, 0

    # the Hessian is allocated from the weight's second dimension whatever its
    # shape, see `make_empty_hessian`
    num_columns = module.weight.shape[1]
    held = num_columns * num_columns * GPTQ_PRECISION.itemsize

    # batched_quantization="auto" sizes batches to the memory left over at
    # compression time, so only a batch of one module has to fit
    batch_size = (
        self.batched_quantization
        if isinstance(self.batched_quantization, int)
        else 1
    )
    scratch = batch_size * quantization_peak_bytes(module, self.block_size)

    return held, scratch

on_calibration_end

on_calibration_end(state: State, event: Event, **kwargs)

Finish calibrating by removing observers and calibration hooks

Source code in src/llmcompressor/modifiers/gptq/base.py
def on_calibration_end(self, state: State, event: Event, **kwargs):
    """
    Finish calibrating by removing observers and calibration hooks
    """
    QuantizationMixin.end_calibration(self, state.model)
    self.remove_hooks()  # remove gptq hooks

on_finalize

on_finalize(state: State, **kwargs) -> bool

disable the quantization observers used by the OBCQ algorithm

Parameters:

  • state (State) –

    session state storing input model and calibration data

Source code in src/llmcompressor/modifiers/gptq/base.py
def on_finalize(self, state: State, **kwargs) -> bool:
    """
    disable the quantization observers used by the OBCQ algorithm

    :param state: session state storing input model and calibration data
    """
    if not self.ended_:
        self.on_end(state, None)

    self._log_rtn_fallback_summary()

    if len(self._num_samples) > 0:
        raise ValueError(f"Failed to compress {len(self._num_samples)} modules")

    self._hessians = dict()
    self._num_samples = dict()
    self._num_compressed_modules = 0
    self._rtn_fallback_module_names = []

    return True

on_initialize

on_initialize(state: State, **kwargs) -> bool

Initialize and run the GPTQ algorithm on the current state

Parameters:

  • state (State) –

    session state storing input model and calibration data

Source code in src/llmcompressor/modifiers/gptq/base.py
def on_initialize(self, state: State, **kwargs) -> bool:
    """
    Initialize and run the GPTQ algorithm on the current state

    :param state: session state storing input model and calibration data
    """
    # apply config to model and prepare calibration hooks
    if QuantizationMixin.has_config(self):
        QuantizationMixin.initialize_quantization(self, state)

    # prepare module names
    self._module_names = {
        m: name
        for name, m in match_named_modules(
            state.model, self.resolved_targets, self.ignore
        )
    }

    return True

QuantizationMixin

Bases: HooksMixin

Mixin which enables a Modifier to act as a quantization config, attaching observers, calibration hooks, and compression wrappers to modifiers

Lifecycle:

  • on_initialize: QuantizationMixin.initialize_quantization
    • Attach schemes to modules
    • Attach observers to modules
    • Disable quantization until calibration starts/finishes
  • on_calibration_start: QuantizationMixin.start_calibration
    • Attach calibration hooks
    • Apply calibration status
    • Enable quantization during calibration
  • on_calibration_end: QuantizationMixin.end_calibration
    • Remove calibration hooks
    • Apply freeze status
    • Keep quantization enabled for future steps

NOTE: QuantizationMixin does not update scales and zero-points on its own, as this is not desired for all Modifiers inheriting from it. Modifier must explicitly call observe(modules, base_name="weight") then update_qparams(modules, base_name="weight"). See QuantizationModifier.on_event method for example

NOTE: When no observer is specified, weight observers default to imatrix_mse if calibration data is available to the session and memoryless_minmax otherwise. Activation observers default to minmax.

Parameters:

  • config_groups –

    dictionary specifying quantization schemes to apply to target modules. Modules not matching a scheme target will NOT be quantized.

  • targets –

    list of layer names to quantize if a scheme is provided. If unset, will contain all targets listed in config_groups. If config_groups is also unset, will default to ["Linear"] (i.e. all Linear layers will be targeted). This field is not the source of truth for finding all matching target layers in a model. Additional information can be stored in config_groups. Use self.resolved_targets instead.

  • ignore –

    optional list of module class names or submodule names to not quantize even if they match a target in config_groups. Defaults to empty list.

  • scheme –

    a single quantization scheme to apply to the model. This is a dictionary that supports all keys from QuantizationScheme except targets, which will be set to the targets parameter set at the modifier level. Can also be set to a dictionary of the format preset_scheme_name: targets for example: W8A8: ['Linear'] for weight and activation 8-bit.

  • kv_cache_scheme –

    optional QuantizationArgs, that specify the quantization of the kv cache. If None, kv cache is not quantized. When applying kv cache quantization to transformer AutoModelForCausalLM, the kv_cache_scheme gets converted into a QuantizationScheme that: - targets the q_proj and k_proj modules of the model. The outputs of those modules are the keys and values that might be cached - quantizes the outputs of the aforementioned layers, so that keys and values are compressed before storing them in the cache There is an explicit assumption that the model contains modules with k_proj and v_proj in their names. If this is not the case and kv_cache_scheme != None, the quantization of kv cache will fail

  • weight_observer –

    optional observer name for weight quantization. Overrides the default observer specified in the scheme. Valid values include "minmax", "mse", "static_minmax", "memoryless_minmax", "memoryless_mse", and "imatrix_mse". Defaults to "imatrix_mse" when calibration data is available and "memoryless_minmax" otherwise.

  • input_observer –

    optional observer name for input activation quantization. Overrides the default observer specified in the scheme. Valid values include "minmax", "mse", "static_minmax", "memoryless_minmax", "memoryless_mse".

  • output_observer –

    optional observer name for output activation quantization. Overrides the default observer specified in the scheme. Valid values include "minmax", "mse", "static_minmax", "memoryless_minmax", "memoryless_mse".

  • observer –

    optional dictionary to specify observers for multiple quantization types at once. Keys can be "weights", "input", or "output". Values are observer names. Example: {"weights": "MSE", "input": "MSE"}. If both individual observer parameters (weight_observer, input_observer, output_observer) and observer dict are provided, the observer dict takes precedence.

  • bypass_divisibility_checks –

    if True, skip the check that weight columns are divisible by group_size for GROUP/TENSOR_GROUP. Use when your runtime (e.g. vLLM) supports non-divisible dimensions. Defaults to False.

Methods:

Attributes:

  • resolved_targets (set[str]) –

    Set of all resolved targets, i.e. all unique targets listed

resolved_targets property

resolved_targets: set[str]

Set of all resolved targets, i.e. all unique targets listed in resolved quantization config. Use this property instead of the targets field, as targets can also come from config_groups depending on how recipe is configured.

end_calibration

end_calibration(model: Module)

Remove calibration hooks and observers, and set the model status to frozen. Keep quantization enabled for future operations

Parameters:

  • model (Module) –

    model to end calibration for

Source code in src/llmcompressor/modifiers/quantization/quantization/mixin.py
def end_calibration(self, model: torch.nn.Module):
    """
    Remove calibration hooks and observers, and set the model status to frozen.
    Keep quantization enabled for future operations

    :param model: model to end calibration for
    """
    self.remove_hooks(self._calibration_hooks)
    modules = [
        module
        for _, module in match_named_modules(
            model, self.resolved_targets, self.ignore
        )
    ]

    # observer statistics are consumed when qparams are updated. Modules which
    # still have statistics were never passed to `on_sequential_epoch_end`, for
    # example modules which are traced through rather than called by a sequential
    # subgraph (e.g. attention when using `sequential_targets="Linear"`)
    remaining = [module for module in modules if _has_observer_statistics(module)]
    if len(remaining) > 0:
        module_types = sorted({type(module).__name__ for module in remaining})
        logger.warning(
            f"Found {len(remaining)} modules of type(s) {module_types} which were "
            "calibrated but not quantized. This can occur when setting highly "
            "granular sequential targets. Consider setting less granular "
            "sequential targets for better performance. Quantizing remaining "
            "modules now."
        )
        self.on_sequential_epoch_end(
            active_session().state,
            Event(type_=EventType.SEQUENTIAL_EPOCH_END),
            modules=remaining,
        )

    for module in modules:
        freeze_module_quantization(module)  # remove observers

    model.apply(enable_quantization)  # keep quantization enabled

has_config

has_config() -> bool

Determine if the user has specified a quantization config on this modifier

Source code in src/llmcompressor/modifiers/quantization/quantization/mixin.py
def has_config(self) -> bool:
    """
    Determine if the user has specified a quantization config on this modifier
    """
    return not (
        self.config_groups is None
        and self.targets == ["Linear"]
        and self.ignore == []
        and self.scheme is None
        and self.kv_cache_scheme is None
    )

initialize_quantization

initialize_quantization(state: State)

Attach quantization schemes to modules in the model according to the quantization config specified on this modifier

Parameters:

  • state (State) –

    session state holding the model to attach schemes and observers to. Observer defaults are resolved according to state.has_calibration_data

Source code in src/llmcompressor/modifiers/quantization/quantization/mixin.py
def initialize_quantization(self, state: State):
    """
    Attach quantization schemes to modules in the model according to
    the quantization config specified on this modifier

    :param state: session state holding the model to attach schemes and
        observers to. Observer defaults are resolved according to
        ``state.has_calibration_data``
    """
    model = state.model
    config = self.resolve_quantization_config(
        has_calibration_data=state.has_calibration_data
    )
    targets = _resolved_targets(config)

    for _, module in match_named_modules(model, targets, self.ignore):
        reset_quantization_status(module)  # reset any previously applied qconfigs

    apply_quantization_config(model, config)

    if not self.bypass_divisibility_checks:
        validate_group_size_divisibility(model, targets, self.ignore)

    # disable quantization until calibration
    model.apply(disable_quantization)

resolve_quantization_config

resolve_quantization_config(
    has_calibration_data: bool = False,
) -> QuantizationConfig

Returns the quantization config specified by this modifier

Parameters:

  • has_calibration_data (bool, default: False ) –

    whether calibration data will be available to this modifier. If True, the default weight observer is imatrix_mse, which weights quantization error by per-channel activation importance; otherwise the default is memoryless_minmax

Source code in src/llmcompressor/modifiers/quantization/quantization/mixin.py
def resolve_quantization_config(
    self, has_calibration_data: bool = False
) -> QuantizationConfig:
    """
    Returns the quantization config specified by this modifier

    :param has_calibration_data: whether calibration data will be available
        to this modifier. If True, the default weight observer is
        ``imatrix_mse``, which weights quantization error by per-channel
        activation importance; otherwise the default is
        ``memoryless_minmax``
    """
    scheme = self.scheme
    targets = self.targets
    # work on a copy so that the fields specified on this modifier are
    # left untouched and resolution can be repeated with different
    # `has_calibration_data` values
    config_groups = deepcopy(self.config_groups)
    kv_cache_scheme = self.kv_cache_scheme
    ignore = self.ignore

    if scheme is not None and config_groups is not None:
        raise ValueError("Please specify either `scheme` or `config_groups`")

    if scheme is not None:
        # takes precedence over config_groups

        if isinstance(scheme, str) and is_preset_scheme(scheme):
            # attach targets to scheme
            scheme = {scheme: targets}

        config_groups = {}
        for idx, key in enumerate(scheme.keys()):
            if is_preset_scheme(key):
                scheme_obj = preset_name_to_scheme(key, scheme[key])
            else:
                scheme_obj = QuantizationScheme.model_validate(
                    {"targets": scheme[key], **scheme}
                )

            # Apply observer overrides if specified
            scheme_obj = self._apply_observer_overrides(
                scheme_obj, has_calibration_data
            )

            group_name = f"group_{idx}"
            config_groups[group_name] = scheme_obj

    if config_groups is None or len(config_groups) == 0:
        default_quant_scheme = QuantizationScheme(targets=targets)
        # Apply observer overrides to default scheme as well
        default_quant_scheme = self._apply_observer_overrides(
            default_quant_scheme, has_calibration_data
        )
        config_groups = {"group_0": default_quant_scheme}
    elif scheme is None:
        # Apply observer overrides to all config groups when config_groups
        # was provided directly (not derived from scheme)
        for scheme_obj in config_groups.values():
            self._apply_observer_overrides(scheme_obj, has_calibration_data)

    return QuantizationConfig(
        config_groups=config_groups,
        kv_cache_scheme=kv_cache_scheme,
        quantization_status=QuantizationStatus.INITIALIZED,
        ignore=ignore,
    )

start_calibration

start_calibration(model: Module)

Attach observers, register activation calibration hooks (including kv_cache quantization) and enable quantization as we calibrate

Parameters:

  • model (Module) –

    model to prepare for calibration

Source code in src/llmcompressor/modifiers/quantization/quantization/mixin.py
def start_calibration(self, model: torch.nn.Module):
    """
    Attach observers, register activation calibration hooks (including
    kv_cache quantization) and enable quantization as we calibrate

    :param model: model to prepare for calibration
    """
    targets = match_named_modules(model, self.resolved_targets, self.ignore)
    if targets_embeddings(model, targets):
        untie_word_embeddings(model)
    warn_inference_mode_forwards(model)

    for _, module in match_named_modules(model, self.resolved_targets, self.ignore):
        self._initialize_observers(module)
        self._calibration_hooks |= self._initialize_hooks(module)
        apply_calibration_status(module)

    # Link weight observers in fused groups (Q/K/V, gate/up) for shared global_scale
    fuse_weight_observers(model)

start_layerwise_calibration

start_layerwise_calibration(
    state: State, model: Module, modules: list[Module]
)

Set up quantization for a subset of modules after layerwise decompression. Applies quantization config scoped to the given modules, then initializes observers, calibration hooks, and fuses weight observers.

Parameters:

  • state (State) –

    session state, used to resolve observer defaults according to state.has_calibration_data

  • model (Module) –

    the full model (needed for config application)

  • modules (list[Module]) –

    modules in the current subgraph to prepare

Source code in src/llmcompressor/modifiers/quantization/quantization/mixin.py
def start_layerwise_calibration(
    self, state: State, model: torch.nn.Module, modules: list[torch.nn.Module]
):
    """
    Set up quantization for a subset of modules after layerwise
    decompression. Applies quantization config scoped to the given
    modules, then initializes observers, calibration hooks, and fuses
    weight observers.

    :param state: session state, used to resolve observer defaults
        according to ``state.has_calibration_data``
    :param model: the full model (needed for config application)
    :param modules: modules in the current subgraph to prepare
    """
    config = self.resolve_quantization_config(
        has_calibration_data=state.has_calibration_data
    )
    apply_quantization_config(model, config, allowed_modules=modules)
    for module in modules:
        if is_module_quantized(module):
            self._initialize_observers(module)
            self._calibration_hooks |= self._initialize_hooks(module)
            apply_calibration_status(module)
    fuse_weight_observers(model, modules)

sync_obs_act_stats

sync_obs_act_stats(modules: Iterator[Module])

Synchronize the activation statistics for observers across DDP ranks. Iterates all observers (weight, input, output, q, k, v); note: No-op when not distributed and most weight observers don't have activation statistics and thus are no-ops as well.

Parameters:

  • modules (Iterator[Module]) –

    iterable of modules to sync (e.g., from a sequential chunk)

Source code in src/llmcompressor/modifiers/quantization/quantization/mixin.py
def sync_obs_act_stats(self, modules: Iterator[torch.nn.Module]):
    """
    Synchronize the activation statistics for observers
    across DDP ranks. Iterates all observers
    (weight, input, output, q, k, v);
    note: No-op when not distributed and
    most weight observers don't have activation
    statistics and thus are no-ops as well.

    :param modules: iterable of modules to sync (e.g., from a sequential chunk)
    """
    if not is_distributed():
        return

    pending_comms = []
    synced_obs = {None}  # ignore None observers
    for module in modules:
        for base_name in ACTIVATION_OBS + ("weight",):
            observer = getattr(module, f"{base_name}_observer", None)
            if observer not in synced_obs:
                synced_obs.add(observer)
                pending_comms.extend(observer.sync_activation_stats())
    wait_for_comms(pending_comms)

validate_observer

validate_observer(value: Any) -> dict[str, str] | None

Validate observer dictionary format. Accepts keys: 'weights', 'input', 'output'

Source code in src/llmcompressor/modifiers/quantization/quantization/mixin.py
@field_validator("observer", mode="before")
def validate_observer(cls, value: Any) -> dict[str, str] | None:
    """
    Validate observer dictionary format. Accepts keys: 'weights', 'input', 'output'
    """
    if value is None:
        return value

    if not isinstance(value, dict):
        raise ValueError("`observer` must be a dictionary")

    valid_keys = {"weights", "input", "output"}
    for key in value.keys():
        if key not in valid_keys:
            raise ValueError(
                f"Invalid observer key '{key}'. Valid keys are: {valid_keys}"
            )
        if not isinstance(value[key], str):
            raise ValueError(f"Observer value for '{key}' must be a string")

    return value

QuantizationModifier

Bases: Modifier, QuantizationMixin

Enables post training quantization (PTQ) and quantization aware training (QAT) for a given module or its submodules. After calibration (PTQ) or the start epoch (QAT), the specified module(s) forward pass will emulate quantized execution and the modifier will be enabled until training is completed.

In DDP mode, activation observer statistics are all-reduced across ranks at sequential layer boundaries so all ranks share identical quantization parameters.

Parameters:

  • config_groups –

    dictionary specifying quantization schemes to apply to target modules. Modules not matching a scheme target will NOT be quantized.

  • targets –

    list of layer names to quantize if a scheme is provided. Defaults to Linear layers

  • ignore –

    optional list of module class names or submodule names to not quantize even if they match a target in config_groups. Defaults to empty list.

  • scheme –

    a single quantization scheme to apply to the model. This is a dictionary that supports all keys from QuantizationScheme except targets, which will be set to the targets parameter set at the modifier level. Can also be set to a dictionary of the format preset_scheme_name: targets for example: W8A8: ['Linear'] for weight and activation 8-bit.

  • kv_cache_scheme –

    optional QuantizationArgs, that specify the quantization of the kv cache. If None, kv cache is not quantized. When applying kv cache quantization to transformer AutoModelForCausalLM, the kv_cache_scheme gets converted into a QuantizationScheme that: - targets the q_proj and k_proj modules of the model. The outputs of those modules are the keys and values that might be cached - quantizes the outputs of the aforementioned layers, so that keys and values are compressed before storing them in the cache There is an explicit assumption that the model contains modules with k_proj and v_proj in their names. If this is not the case and kv_cache_scheme != None, the quantization of kv cache will fail

Methods:

estimate_sequential_memory

estimate_sequential_memory(
    module: Module, num_tokens: int
) -> tuple[int, int]

Observers keep only small per-module statistics, but observing a weight when its subgraph ends may need a full precision copy of it

Parameters:

  • module (Module) –

    module belonging to the subgraph being calibrated

  • num_tokens (int) –

    number of calibration tokens, unused

Returns:

  • tuple[int, int] –

    tuple of (held, scratch) bytes

Source code in src/llmcompressor/modifiers/quantization/quantization/base.py
def estimate_sequential_memory(
    self, module: torch.nn.Module, num_tokens: int
) -> tuple[int, int]:
    """
    Observers keep only small per-module statistics, but observing a weight when
    its subgraph ends may need a full precision copy of it

    :param module: module belonging to the subgraph being calibrated
    :param num_tokens: number of calibration tokens, unused
    :return: tuple of (held, scratch) bytes
    """
    weight = getattr(module, "weight", None)
    if weight is None or not is_module_quantized(module):
        return 0, 0

    return 0, weight.numel() * torch.float32.itemsize

on_calibration_end

on_calibration_end(state: State, event: Event, **kwargs)

Finish calibrating by removing observers and calibration hooks.

Skipped when layerwise_decompression is enabled; per-layer cleanup is handled in on_sequential_epoch_end.

Source code in src/llmcompressor/modifiers/quantization/quantization/base.py
def on_calibration_end(self, state: State, event: Event, **kwargs):
    """
    Finish calibrating by removing observers and calibration hooks.

    Skipped when layerwise_decompression is enabled; per-layer cleanup
    is handled in on_sequential_epoch_end.
    """
    if not getattr(state, "layerwise_decompression", False):
        QuantizationMixin.end_calibration(self, state.model)

on_calibration_start

on_calibration_start(state: State, event: Event, **kwargs)

Begin calibrating activations.

Skipped when layerwise_decompression is enabled; per-layer calibration setup is handled by start_layerwise_calibration in the pipeline.

Source code in src/llmcompressor/modifiers/quantization/quantization/base.py
def on_calibration_start(self, state: State, event: Event, **kwargs):
    """
    Begin calibrating activations.

    Skipped when layerwise_decompression is enabled; per-layer calibration
    setup is handled by start_layerwise_calibration in the pipeline.
    """
    if not getattr(state, "layerwise_decompression", False):
        QuantizationMixin.start_calibration(self, state.model)

on_initialize

on_initialize(state: State, **kwargs) -> bool

Prepare to calibrate activations and weights

According to the quantization config, a quantization scheme is attached to each targeted module. The module's forward call is also overwritten to perform quantization to inputs, weights, and outputs.

Then, according to the module's quantization scheme, observers and calibration hooks are added. These hooks are disabled until the modifier starts.

Skipped when layerwise_decompression is enabled because modules are still compressed; per-layer setup happens in start_layerwise_calibration instead.

Source code in src/llmcompressor/modifiers/quantization/quantization/base.py
def on_initialize(self, state: State, **kwargs) -> bool:
    """
    Prepare to calibrate activations and weights

    According to the quantization config, a quantization scheme is attached to each
    targeted module. The module's forward call is also overwritten to perform
    quantization to inputs, weights, and outputs.

    Then, according to the module's quantization scheme, observers and calibration
    hooks are added. These hooks are disabled until the modifier starts.

    Skipped when layerwise_decompression is enabled because modules are still
    compressed; per-layer setup happens in start_layerwise_calibration instead.
    """
    if not QuantizationMixin.has_config(self):
        raise ValueError(
            "QuantizationModifier requires that quantization fields be specified"
        )
    if not getattr(state, "layerwise_decompression", False):
        QuantizationMixin.initialize_quantization(self, state)

    return True

quantize_weight

quantize_weight(
    weights: Tensor,
    hessians: Tensor,
    scale: Tensor,
    zero_point: Tensor,
    global_scale: Tensor | None,
    quant_args: QuantizationArgs,
    blocksize: int = 128,
    percdamp: float = 0.01,
) -> tuple[torch.Tensor, torch.Tensor]

Quantize a batch of weights according to the GPTQ algorithm.

Parameters:

  • weights (Tensor) –

    weights with shape [batch, rows, columns]

  • hessians (Tensor) –

    normalized Hessians with shape [batch, columns, columns]

  • scale (Tensor) –

    stacked observer scales

  • zero_point (Tensor) –

    stacked observer zero points

  • global_scale (Tensor | None) –

    optional stacked observer global scales

  • quant_args (QuantizationArgs) –

    quantization arguments used to find quantization parameters

  • blocksize (int, default: 128 ) –

    chunk size of quantization updates

  • percdamp (float, default: 0.01 ) –

    dampening factor on hessian diagonal

Returns:

  • tuple[Tensor, Tensor] –

    quantized weights, per-batch losses, and RTN fallback flags

Source code in src/llmcompressor/modifiers/gptq/gptq_quantize.py
def quantize_weight(
    weights: torch.Tensor,
    hessians: torch.Tensor,
    scale: torch.Tensor,
    zero_point: torch.Tensor,
    global_scale: torch.Tensor | None,
    quant_args: QuantizationArgs,
    blocksize: int = 128,
    percdamp: float = 0.01,
) -> tuple[torch.Tensor, torch.Tensor]:
    """
    Quantize a batch of weights according to the GPTQ algorithm.

    :param weights: weights with shape [batch, rows, columns]
    :param hessians: normalized Hessians with shape [batch, columns, columns]
    :param scale: stacked observer scales
    :param zero_point: stacked observer zero points
    :param global_scale: optional stacked observer global scales
    :param quant_args: quantization arguments used to find quantization parameters
    :param blocksize: chunk size of quantization updates
    :param percdamp: dampening factor on hessian diagonal
    :return: quantized weights, per-batch losses, and RTN fallback flags
    """
    if weights.ndim != 3 or hessians.ndim != 3:
        raise ValueError("weights and hessians must have shape [batch, ...]")
    if weights.shape[0] != hessians.shape[0]:
        raise ValueError("weights and hessians must have matching batch sizes")

    batch_size, num_rows, num_columns = weights.shape
    strategy = quant_args.strategy
    actorder = quant_args.actorder
    final_dtype = weights.dtype
    device = weights.device
    # The caller provides a disposable stacked weight tensor, so use it as the
    # working buffer when it is already FP32 instead of allocating another copy.
    W = weights.to(device=device, dtype=GPTQ_PRECISION)
    # The stacked Hessian is the disposable working buffer.
    H = hessians.to(device=device, dtype=GPTQ_PRECISION)
    del weights, hessians
    scale = scale.to(device=device)
    zero_point = zero_point.to(device=device)
    if global_scale is not None:
        global_scale = global_scale.to(device=device)

    W, H, perm = apply_activation_ordering(W, H, quant_args.actorder)
    # handle g_idx
    g_idx = None
    if strategy in (
        QuantizationStrategy.GROUP,
        QuantizationStrategy.TENSOR_GROUP,
        QuantizationStrategy.BLOCK,
    ):
        # mapping from column index to group index
        divisor = (
            quant_args.group_size
            if strategy != QuantizationStrategy.BLOCK
            else quant_args.block_structure[1]
        )
        g_idx = torch.arange(num_columns, device=device, dtype=torch.int) // divisor

        if actorder == ActivationOrdering.WEIGHT:
            g_idx = torch.gather(g_idx.unsqueeze(0).expand(batch_size, -1), 1, perm)

    losses = torch.zeros(batch_size, num_rows, device=device)
    used_rtn_fallback = torch.zeros(batch_size, dtype=torch.bool, device=device)
    Hinv = factorize_hessian(W, H, percdamp, used_rtn_fallback)
    # See section 3.4 of https://arxiv.org/abs/2203.07259
    for i1 in range(0, num_columns, blocksize):
        i2 = min(i1 + blocksize, num_columns)

        W1 = W[:, :, i1:i2].clone()
        Q1 = torch.zeros_like(W1)
        Err1 = torch.zeros_like(W1)
        losses1 = torch.zeros_like(W1)
        Hinv1 = Hinv[:, i1:i2, i1:i2]

        gptq_block_update(
            W1,
            Hinv1,
            Q1,
            Err1,
            losses1,
            scale=scale,
            zero_point=zero_point,
            global_scale=global_scale,
            g_idx=g_idx,
            quant_args=quant_args,
            i1=i1,
        )

        # propagate block error
        W[:, :, i1:i2] = Q1
        losses += torch.sum(losses1, 2) / 2

        w_err = torch.bmm(Err1, Hinv[:, i1:i2, i2:])
        W[:, :, i2:] -= w_err

    if perm is not None:
        # restore original permutation
        # Release block-local tensors before allocating the restored output.
        del W1, Q1, Err1, losses1, Hinv1, w_err
        invperm = torch.argsort(perm, dim=-1)
        W = torch.gather(W, -1, invperm.unsqueeze(-2).expand(-1, num_rows, -1))

    return W.to(final_dtype), losses.sum(dim=1), used_rtn_fallback