llmcompressor.modifiers.quantization.gptq
Classes:
-
GPTQModifier–Implements the GPTQ algorithm from https://arxiv.org/abs/2210.17323. This modifier
Functions:
-
quantize_weight–Quantize a batch of weights according to the GPTQ algorithm.
GPTQModifier
Bases: Modifier, QuantizationMixin
Implements the GPTQ algorithm from https://arxiv.org/abs/2210.17323. This modifier uses activations to calibrate a hessian matrix, which is then used to determine optimal quantization values and orderings for the model weights.
Sample yaml:
test_stage:
obcq_modifiers:
GPTQModifier:
block_size: 128
dampening_frac: 0.001
actorder: static
config_groups:
group_0:
targets:
- "Linear"
input_activations: null
output_activations: null
weights:
num_bits: 8
type: "int"
symmetric: true
strategy: group
group_size: 128
Lifecycle:
- on_initialize
- apply config to model
- on_calibration_start
- add activation calibration hooks
- add gptq weight calibration hooks
- on_sequential_epoch_end
- quantize_weight
- on_calibration_end
- remove_hooks()
- model.apply(freeze_module_quantization)
Parameters:
-
block_size–Used to determine number of columns to compress in one pass
-
dampening_frac–Amount of dampening to apply to H, as a fraction of the diagonal norm
-
actorder–order in which weight columns are quantized. Defaults to "static" activation ordering, which achieves best accuracy recovery with no runtime cost. For more information, see https://github.com/vllm-project/vllm/pull/8135.
-
batched_quantization–Controls batching of same-shape modules (e.g. linearized MoE experts).
"auto"(the default) limits batches to 75% of available CUDA memory. A positive integer sets a maximum batch size without consulting available memory.Nonedisables batching. -
config_groups–dictionary specifying quantization schemes to apply to target modules. Modules not matching a scheme target will NOT be quantized.
-
targets–list of layer names to quantize if a scheme is provided. Defaults to Linear layers
-
ignore–optional list of module class names or submodule names to not quantize even if they match a target in config_groups. Defaults to empty list.
-
scheme–a single quantization scheme to apply to the model. This is a dictionary that supports all keys from QuantizationScheme except targets, which will be set to the targets parameter set at the modifier level. Can also be set to a dictionary of the format
preset_scheme_name: targetsfor example:W8A8: ['Linear']for weight and activation 8-bit. -
kv_cache_scheme–optional QuantizationArgs, that specify the quantization of the kv cache. If None, kv cache is not quantized. When applying kv cache quantization to transformer AutoModelForCausalLM, the kv_cache_scheme gets converted into a QuantizationScheme that: - targets the
q_projandk_projmodules of the model. The outputs of those modules are the keys and values that might be cached - quantizes the outputs of the aforementioned layers, so that keys and values are compressed before storing them in the cache There is an explicit assumption that the model contains modules withk_projandv_projin their names. If this is not the case and kv_cache_scheme != None, the quantization of kv cache will fail
Methods:
-
calibrate_module–Calibration hook used to accumulate the hessian of the input to the module
-
compress_modules–Quantize modules which have been calibrated
-
estimate_sequential_memory–Estimate the Hessian held for a module until its subgraph is compressed, and
-
on_calibration_end–Finish calibrating by removing observers and calibration hooks
-
on_finalize–disable the quantization observers used by the OBCQ algorithm
-
on_initialize–Initialize and run the GPTQ algorithm on the current state
calibrate_module
Calibration hook used to accumulate the hessian of the input to the module
Parameters:
-
module(Module) –module being calibrated
-
args(tuple[Tensor, ...]) –inputs to the module, the first element of which is the canonical input
-
_output(Tensor) –uncompressed module output, unused
Source code in src/llmcompressor/modifiers/gptq/base.py
compress_modules
Quantize modules which have been calibrated
Source code in src/llmcompressor/modifiers/gptq/base.py
estimate_sequential_memory
Estimate the Hessian held for a module until its subgraph is compressed, and the temporary memory needed to compress it
Parameters:
-
module(Module) –module belonging to the subgraph being calibrated
-
num_tokens(int) –number of calibration tokens, unused
Returns:
-
tuple[int, int]–tuple of (held, scratch) bytes
Source code in src/llmcompressor/modifiers/gptq/base.py
on_calibration_end
Finish calibrating by removing observers and calibration hooks
Source code in src/llmcompressor/modifiers/gptq/base.py
on_finalize
disable the quantization observers used by the OBCQ algorithm
Parameters:
-
state(State) –session state storing input model and calibration data
Source code in src/llmcompressor/modifiers/gptq/base.py
on_initialize
Initialize and run the GPTQ algorithm on the current state
Parameters:
-
state(State) –session state storing input model and calibration data
Source code in src/llmcompressor/modifiers/gptq/base.py
quantize_weight
quantize_weight(
weights: Tensor,
hessians: Tensor,
scale: Tensor,
zero_point: Tensor,
global_scale: Tensor | None,
quant_args: QuantizationArgs,
blocksize: int = 128,
percdamp: float = 0.01,
) -> tuple[torch.Tensor, torch.Tensor]
Quantize a batch of weights according to the GPTQ algorithm.
Parameters:
-
weights(Tensor) –weights with shape [batch, rows, columns]
-
hessians(Tensor) –normalized Hessians with shape [batch, columns, columns]
-
scale(Tensor) –stacked observer scales
-
zero_point(Tensor) –stacked observer zero points
-
global_scale(Tensor | None) –optional stacked observer global scales
-
quant_args(QuantizationArgs) –quantization arguments used to find quantization parameters
-
blocksize(int, default:128) –chunk size of quantization updates
-
percdamp(float, default:0.01) –dampening factor on hessian diagonal
Returns:
-
tuple[Tensor, Tensor]–quantized weights, per-batch losses, and RTN fallback flags
Source code in src/llmcompressor/modifiers/gptq/gptq_quantize.py
34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 | |