Skip to content

llmcompressor.modifiers.quantization.gptq

Classes:

Functions:

  • quantize_weight –

    Quantize a batch of weights according to the GPTQ algorithm.

GPTQModifier

Bases: Modifier, QuantizationMixin

Implements the GPTQ algorithm from https://arxiv.org/abs/2210.17323. This modifier uses activations to calibrate a hessian matrix, which is then used to determine optimal quantization values and orderings for the model weights.

Sample yaml:

test_stage:
  obcq_modifiers:
    GPTQModifier:
      block_size: 128
      dampening_frac: 0.001
      actorder: static
      config_groups:
        group_0:
          targets:
            - "Linear"
          input_activations: null
          output_activations: null
          weights:
            num_bits: 8
            type: "int"
            symmetric: true
            strategy: group
            group_size: 128

Lifecycle:

  • on_initialize
    • apply config to model
  • on_calibration_start
    • add activation calibration hooks
    • add gptq weight calibration hooks
  • on_sequential_epoch_end
    • quantize_weight
  • on_calibration_end
    • remove_hooks()
    • model.apply(freeze_module_quantization)

Parameters:

  • block_size –

    Used to determine number of columns to compress in one pass

  • dampening_frac –

    Amount of dampening to apply to H, as a fraction of the diagonal norm

  • actorder –

    order in which weight columns are quantized. Defaults to "static" activation ordering, which achieves best accuracy recovery with no runtime cost. For more information, see https://github.com/vllm-project/vllm/pull/8135.

  • batched_quantization –

    Controls batching of same-shape modules (e.g. linearized MoE experts). "auto" (the default) limits batches to 75% of available CUDA memory. A positive integer sets a maximum batch size without consulting available memory. None disables batching.

  • config_groups –

    dictionary specifying quantization schemes to apply to target modules. Modules not matching a scheme target will NOT be quantized.

  • targets –

    list of layer names to quantize if a scheme is provided. Defaults to Linear layers

  • ignore –

    optional list of module class names or submodule names to not quantize even if they match a target in config_groups. Defaults to empty list.

  • scheme –

    a single quantization scheme to apply to the model. This is a dictionary that supports all keys from QuantizationScheme except targets, which will be set to the targets parameter set at the modifier level. Can also be set to a dictionary of the format preset_scheme_name: targets for example: W8A8: ['Linear'] for weight and activation 8-bit.

  • kv_cache_scheme –

    optional QuantizationArgs, that specify the quantization of the kv cache. If None, kv cache is not quantized. When applying kv cache quantization to transformer AutoModelForCausalLM, the kv_cache_scheme gets converted into a QuantizationScheme that: - targets the q_proj and k_proj modules of the model. The outputs of those modules are the keys and values that might be cached - quantizes the outputs of the aforementioned layers, so that keys and values are compressed before storing them in the cache There is an explicit assumption that the model contains modules with k_proj and v_proj in their names. If this is not the case and kv_cache_scheme != None, the quantization of kv cache will fail

Methods:

  • calibrate_module –

    Calibration hook used to accumulate the hessian of the input to the module

  • compress_modules –

    Quantize modules which have been calibrated

  • estimate_sequential_memory –

    Estimate the Hessian held for a module until its subgraph is compressed, and

  • on_calibration_end –

    Finish calibrating by removing observers and calibration hooks

  • on_finalize –

    disable the quantization observers used by the OBCQ algorithm

  • on_initialize –

    Initialize and run the GPTQ algorithm on the current state

calibrate_module

calibrate_module(
    module: Module,
    args: tuple[Tensor, ...],
    _output: Tensor,
)

Calibration hook used to accumulate the hessian of the input to the module

Parameters:

  • module (Module) –

    module being calibrated

  • args (tuple[Tensor, ...]) –

    inputs to the module, the first element of which is the canonical input

  • _output (Tensor) –

    uncompressed module output, unused

Source code in src/llmcompressor/modifiers/gptq/base.py
def calibrate_module(
    self,
    module: torch.nn.Module,
    args: tuple[torch.Tensor, ...],
    _output: torch.Tensor,
):
    """
    Calibration hook used to accumulate the hessian of the input to the module

    :param module: module being calibrated
    :param args: inputs to the module, the first element of which is the
        canonical input
    :param _output: uncompressed module output, unused
    """
    # Assume that first argument is the input
    inp = args[0]

    # Initialize hessian if not present
    if module not in self._num_samples:
        self._hessians[module] = make_empty_hessian(
            module, device=get_execution_device(module)
        )
        self._num_samples[module] = torch.zeros(
            tuple(), device=get_execution_device(module)
        )

    self._hessians[module], self._num_samples[module] = accumulate_hessian(
        inp,
        module,
        self._hessians[module],
        self._num_samples[module],
    )

compress_modules

compress_modules()

Quantize modules which have been calibrated

Source code in src/llmcompressor/modifiers/gptq/base.py
def compress_modules(self):
    """
    Quantize modules which have been calibrated
    """
    ### Not Distributed
    if not is_distributed():
        self.compress_module_list(list(self._num_samples.keys()))
        return

    ### Distributed
    rank = dist.get_rank()
    world_size = dist.get_world_size()

    # Assign modules to ranks
    module_list, rank_to_modules, module_to_rank = greedy_bin_packing(
        list(self._hessians.keys()),
        world_size,
        item_weight_fn=lambda mod: self._hessians[mod].shape[0],
    )

    # send hessians to assigned ranks
    self._reduce_hessian_to_target_rank(module_list, module_to_rank)

    self.compress_module_list(rank_to_modules[rank])

    # broadcast compressed modules to each rank
    broadcast_qparams_and_cleanup(module_list, module_to_rank, _GPTQ_Q_PARAMS)

estimate_sequential_memory

estimate_sequential_memory(
    module: Module, num_tokens: int
) -> tuple[int, int]

Estimate the Hessian held for a module until its subgraph is compressed, and the temporary memory needed to compress it

Parameters:

  • module (Module) –

    module belonging to the subgraph being calibrated

  • num_tokens (int) –

    number of calibration tokens, unused

Returns:

  • tuple[int, int] –

    tuple of (held, scratch) bytes

Source code in src/llmcompressor/modifiers/gptq/base.py
def estimate_sequential_memory(
    self, module: torch.nn.Module, num_tokens: int
) -> tuple[int, int]:
    """
    Estimate the Hessian held for a module until its subgraph is compressed, and
    the temporary memory needed to compress it

    :param module: module belonging to the subgraph being calibrated
    :param num_tokens: number of calibration tokens, unused
    :return: tuple of (held, scratch) bytes
    """
    if not self._is_calibrated(module):
        return 0, 0

    # the Hessian is allocated from the weight's second dimension whatever its
    # shape, see `make_empty_hessian`
    num_columns = module.weight.shape[1]
    held = num_columns * num_columns * GPTQ_PRECISION.itemsize

    # batched_quantization="auto" sizes batches to the memory left over at
    # compression time, so only a batch of one module has to fit
    batch_size = (
        self.batched_quantization
        if isinstance(self.batched_quantization, int)
        else 1
    )
    scratch = batch_size * quantization_peak_bytes(module, self.block_size)

    return held, scratch

on_calibration_end

on_calibration_end(state: State, event: Event, **kwargs)

Finish calibrating by removing observers and calibration hooks

Source code in src/llmcompressor/modifiers/gptq/base.py
def on_calibration_end(self, state: State, event: Event, **kwargs):
    """
    Finish calibrating by removing observers and calibration hooks
    """
    QuantizationMixin.end_calibration(self, state.model)
    self.remove_hooks()  # remove gptq hooks

on_finalize

on_finalize(state: State, **kwargs) -> bool

disable the quantization observers used by the OBCQ algorithm

Parameters:

  • state (State) –

    session state storing input model and calibration data

Source code in src/llmcompressor/modifiers/gptq/base.py
def on_finalize(self, state: State, **kwargs) -> bool:
    """
    disable the quantization observers used by the OBCQ algorithm

    :param state: session state storing input model and calibration data
    """
    if not self.ended_:
        self.on_end(state, None)

    self._log_rtn_fallback_summary()

    if len(self._num_samples) > 0:
        raise ValueError(f"Failed to compress {len(self._num_samples)} modules")

    self._hessians = dict()
    self._num_samples = dict()
    self._num_compressed_modules = 0
    self._rtn_fallback_module_names = []

    return True

on_initialize

on_initialize(state: State, **kwargs) -> bool

Initialize and run the GPTQ algorithm on the current state

Parameters:

  • state (State) –

    session state storing input model and calibration data

Source code in src/llmcompressor/modifiers/gptq/base.py
def on_initialize(self, state: State, **kwargs) -> bool:
    """
    Initialize and run the GPTQ algorithm on the current state

    :param state: session state storing input model and calibration data
    """
    # apply config to model and prepare calibration hooks
    if QuantizationMixin.has_config(self):
        QuantizationMixin.initialize_quantization(self, state)

    # prepare module names
    self._module_names = {
        m: name
        for name, m in match_named_modules(
            state.model, self.resolved_targets, self.ignore
        )
    }

    return True

quantize_weight

quantize_weight(
    weights: Tensor,
    hessians: Tensor,
    scale: Tensor,
    zero_point: Tensor,
    global_scale: Tensor | None,
    quant_args: QuantizationArgs,
    blocksize: int = 128,
    percdamp: float = 0.01,
) -> tuple[torch.Tensor, torch.Tensor]

Quantize a batch of weights according to the GPTQ algorithm.

Parameters:

  • weights (Tensor) –

    weights with shape [batch, rows, columns]

  • hessians (Tensor) –

    normalized Hessians with shape [batch, columns, columns]

  • scale (Tensor) –

    stacked observer scales

  • zero_point (Tensor) –

    stacked observer zero points

  • global_scale (Tensor | None) –

    optional stacked observer global scales

  • quant_args (QuantizationArgs) –

    quantization arguments used to find quantization parameters

  • blocksize (int, default: 128 ) –

    chunk size of quantization updates

  • percdamp (float, default: 0.01 ) –

    dampening factor on hessian diagonal

Returns:

  • tuple[Tensor, Tensor] –

    quantized weights, per-batch losses, and RTN fallback flags

Source code in src/llmcompressor/modifiers/gptq/gptq_quantize.py
def quantize_weight(
    weights: torch.Tensor,
    hessians: torch.Tensor,
    scale: torch.Tensor,
    zero_point: torch.Tensor,
    global_scale: torch.Tensor | None,
    quant_args: QuantizationArgs,
    blocksize: int = 128,
    percdamp: float = 0.01,
) -> tuple[torch.Tensor, torch.Tensor]:
    """
    Quantize a batch of weights according to the GPTQ algorithm.

    :param weights: weights with shape [batch, rows, columns]
    :param hessians: normalized Hessians with shape [batch, columns, columns]
    :param scale: stacked observer scales
    :param zero_point: stacked observer zero points
    :param global_scale: optional stacked observer global scales
    :param quant_args: quantization arguments used to find quantization parameters
    :param blocksize: chunk size of quantization updates
    :param percdamp: dampening factor on hessian diagonal
    :return: quantized weights, per-batch losses, and RTN fallback flags
    """
    if weights.ndim != 3 or hessians.ndim != 3:
        raise ValueError("weights and hessians must have shape [batch, ...]")
    if weights.shape[0] != hessians.shape[0]:
        raise ValueError("weights and hessians must have matching batch sizes")

    batch_size, num_rows, num_columns = weights.shape
    strategy = quant_args.strategy
    actorder = quant_args.actorder
    final_dtype = weights.dtype
    device = weights.device
    # The caller provides a disposable stacked weight tensor, so use it as the
    # working buffer when it is already FP32 instead of allocating another copy.
    W = weights.to(device=device, dtype=GPTQ_PRECISION)
    # The stacked Hessian is the disposable working buffer.
    H = hessians.to(device=device, dtype=GPTQ_PRECISION)
    del weights, hessians
    scale = scale.to(device=device)
    zero_point = zero_point.to(device=device)
    if global_scale is not None:
        global_scale = global_scale.to(device=device)

    W, H, perm = apply_activation_ordering(W, H, quant_args.actorder)
    # handle g_idx
    g_idx = None
    if strategy in (
        QuantizationStrategy.GROUP,
        QuantizationStrategy.TENSOR_GROUP,
        QuantizationStrategy.BLOCK,
    ):
        # mapping from column index to group index
        divisor = (
            quant_args.group_size
            if strategy != QuantizationStrategy.BLOCK
            else quant_args.block_structure[1]
        )
        g_idx = torch.arange(num_columns, device=device, dtype=torch.int) // divisor

        if actorder == ActivationOrdering.WEIGHT:
            g_idx = torch.gather(g_idx.unsqueeze(0).expand(batch_size, -1), 1, perm)

    losses = torch.zeros(batch_size, num_rows, device=device)
    used_rtn_fallback = torch.zeros(batch_size, dtype=torch.bool, device=device)
    Hinv = factorize_hessian(W, H, percdamp, used_rtn_fallback)
    # See section 3.4 of https://arxiv.org/abs/2203.07259
    for i1 in range(0, num_columns, blocksize):
        i2 = min(i1 + blocksize, num_columns)

        W1 = W[:, :, i1:i2].clone()
        Q1 = torch.zeros_like(W1)
        Err1 = torch.zeros_like(W1)
        losses1 = torch.zeros_like(W1)
        Hinv1 = Hinv[:, i1:i2, i1:i2]

        gptq_block_update(
            W1,
            Hinv1,
            Q1,
            Err1,
            losses1,
            scale=scale,
            zero_point=zero_point,
            global_scale=global_scale,
            g_idx=g_idx,
            quant_args=quant_args,
            i1=i1,
        )

        # propagate block error
        W[:, :, i1:i2] = Q1
        losses += torch.sum(losses1, 2) / 2

        w_err = torch.bmm(Err1, Hinv[:, i1:i2, i2:])
        W[:, :, i2:] -= w_err

    if perm is not None:
        # restore original permutation
        # Release block-local tensors before allocating the restored output.
        del W1, Q1, Err1, losses1, Hinv1, w_err
        invperm = torch.argsort(perm, dim=-1)
        W = torch.gather(W, -1, invperm.unsqueeze(-2).expand(-1, num_rows, -1))

    return W.to(final_dtype), losses.sum(dim=1), used_rtn_fallback