Observers Overview
An Observer in llm-compressor is a utility class responsible for analyzing weight and activation tensors during calibration. Observers work in two phases:
- Observe: accumulate statistics from tensors via
forward()/update_statistics_from_observed(). Most built-in observers accumulatemin_vals/max_vals, but observers can track any statistics they need (e.g.,imatrix_msetracks_imatrix_sum/_imatrix_count). - Compute: derive quantization parameters (
scale,zero_point, and optionallyglobal_scale) from accumulated statistics viaget_qparams().
The default get_qparams passes min_vals/max_vals to compressed-tensors — specifically calculate_qparams for standard quantization and generate_gparam for FP4 global scales — to produce the final quantization parameters. Observers with custom statistics can override this method.
This two-phase design enables correct handling of fused layers (e.g., Q/K/V projections) that must share a global_scale: all observers accumulate statistics first, then global_scale is computed from the combined statistics of all fused observers.
Observers are designed to be flexible and support a variety of quantization strategies, including per-tensor, per-group, per-channel, and per-token quantization.
Base Class
Observer
Base class for all observers. Subclasses must implement update_statistics_from_observed to define how statistics are accumulated. Subclasses that use statistics other than min_vals/max_vals must also override get_qparams and has_statistics.
The base class handles:
- Reshaping and slicing tensors according to the quantization strategy (group, channel, token, etc.) via flatten_for_calibration
- Computing scale and zero_point from accumulated statistics via get_qparams (default implementation uses min_vals/max_vals with calculate_qparams in compressed-tensors)
- Computing global_scale for FP4 schemes (e.g., NVFP4, MXFP4) from the combined statistics of all fused observers via generate_gparam in compressed-tensors
- Fusing observers across layer groups (Q/K/V, gate/up) via Observer.fuse() for shared global_scale
- DDP synchronization of accumulated statistics via sync_activation_stats() using the declarative _act_sync_dict
This class is not used directly but provides the scaffolding for all custom observers.
Key Methods
| Method | Description |
|---|---|
forward(observed) |
Accumulates statistics from the observed tensor. Returns self (for chaining). Does not compute qparams. |
update_statistics_from_observed(observed) |
Abstract. Subclasses implement this to accumulate statistics from a pre-shaped tensor. |
get_qparams() |
Converts accumulated statistics into a QParamsDict with keys scale, zero_point, and global_scale. For TENSOR_GROUP, auto-observes any fused observers without statistics before computing shared global_scale. Default implementation uses min_vals/max_vals. Override for custom statistics. |
Observer.fuse(observers_and_modules) |
Static method. Takes an iterable of (observer, module) tuples. Links observers with weak module references so they share global_scale computed from combined statistics. |
sync_activation_stats() |
All-reduces accumulated statistics across DDP ranks using _act_sync_dict. |
has_statistics |
Property. Returns True if the observer has been called at least once. Default checks for min_vals; override for custom statistics. |
QParamsDict
Observers return a QParamsDict (a TypedDict) from get_qparams():
class QParamsDict(TypedDict, total=False):
scale: torch.Tensor
zero_point: torch.Tensor
global_scale: Optional[torch.Tensor] # only set for TENSOR_GROUP
Implemented Observers
MinMax Observers
All MinMax observers compute min/max values by tracking the minimum and maximum of the observed tensor. They differ in how they handle state across multiple observations.
memoryless_minmax
Computes min/max from each observed tensor independently, with no memory of past observations. Each observation is handled in isolation.
This is the default weight observer when no calibration data is available (e.g. data-free quantization).
Best used when: - Only a single observation e.g. for weight quantization - You want the most recent observation to fully determine the range
static_minmax (default)
Tracks the running global min/max across all observations. The final range is the union of all observed ranges — the smallest min and largest max seen across all batches.
Best used for: - Scenarios where the range must encompass all possible observed values - Most standard activation quantization scenarios with static/local quantization (FP8, NVFP4)
minmax
Computes min/max using an exponential moving average across observations, controlled by averaging_constant. This smooths out batch-to-batch variance while still adapting to new observations.
Best used for: - Scenarios with static/local activation quantization where there are infrequent outliers we'd like to average out but MSE observers are too slow. - Scenarios where the activations change over time and we want to gradually update the statistics over time.
MSE Observers
MSE observers find the min/max range that minimizes the mean quantization error, using a grid search over candidate scaled min/max factors. They are more expensive than MinMax observers but can yield better accuracy, particularly for integer and low-bit floating point quantization.
memoryless_mse
Performs an MSE grid search on each observed tensor independently, with no memory of past observations.
Best used when: - A single calibration batch is used - Minimizing quantization error is more important than calibration speed
mse
Performs an MSE grid search and maintains a moving average of the resulting min/max values across observations, controlled by averaging_constant.
Best used when: - Calibration accuracy is critical across multiple batches - Quantization error needs to be tightly controlled (e.g., 4-bit weight quantization) - You are doing NVFP4 (see expanded observer below) or integer quantization
NVFP4 expanded MSE (nvfp4_expanded_mse)
nvfp4_expanded_mse is a memoryless MSE observer with defaults tuned for NVFP4
weight quantization. Its defaults search from 1.8x down to approximately 0.8x of the observed
range. This lets the selected range be wider than the strict observed range, which can be beneficial as observed in the Four Over Six paper.
FourOverSix comparison
FourOverSix, introduced in the Four Over Six paper, is an NVFP4 quantization approach that attempts to mitigate for the fact that FP4 has a relatively large gap between representable values near 4 and 6. In some cases quantizing the max to 4 (a 1.5x decrease in range) rather than 6 (1x decrease) can be beneficial. However, rather than checking the 1x and 1.5x values, we found that searching a range of values (which includes 1x and 1.5x) yielded significantly better results. For this reason, among others, we chose to implement the expanded observer rather than adding explicit four over six support.
The following results are from PR #2950.
The table preserves the PR's reported +delta values i.e. increase in PPL above the baseline bf16 eval;
| Configuration | Llama-8B | Qwen3-8B | Qwen3-14B | Qwen3-32B | Llama-70B | MoE-30B | Avg Delta |
|---|---|---|---|---|---|---|---|
| minimax (RTN) | +0.333 | +0.284 | +0.138 | +0.099 | +0.201 | +0.208 | +0.211 |
| FourOverSix | +0.230 | +0.098 | +0.128 | +0.183 | +0.261 | +0.300 | +0.200 |
nvfp4_expanded_mse |
+0.242 | +0.028 | +0.093 | +0.171 | +0.149 | +0.149 | +0.139 |
IMatrix Observer
The IMatrix observer weights quantization error by per-input-channel activation importance (E[x²]), so channels that carry more signal get more careful range optimization.
Supports CHANNEL, GROUP, and TENSOR_GROUP strategies for weight-only Linear modules. Falls back silently to uniform MSE (i.e., standard memoryless_mse behavior) whenever importance data is unavailable.
This observer (imatrix_mse) is the default weight observer whenever calibration data is available, (except for both AWQ which uses minmax and nvfp4 which uses nvfp4_expanded_imatrix below) since activation importance can only be accumulated by running calibration data through the model. When there is no calibration data (e.g. data-free flows), weight observers default to memoryless_minmax instead.
Best used when: - 4-bit weight quantization accuracy is critical - You want to combine with GPTQ for further improvement
NVFP4 expanded IMatrix (nvfp4_expanded_imatrix)
nvfp4_expanded_imatrix uses the same expanded candidate range and grid as
nvfp4_expanded_mse, while weighting quantization error by per-input-channel
importance and using imatrix's default magnitude normalization.
Results (W4A16, Llama-3.1-8B, group_size=128, WikiText-2 PPL):
| Config | PPL |
|---|---|
| FP16 baseline | 6.24 |
RTN memoryless_minmax |
6.96 |
| GPTQ | 6.92 |
| AWQ | 6.89 |
RTN imatrix_mse |
6.85 |
GPTQ + imatrix_mse |
6.83 |
Observer Fusion (global_scale)
For TENSOR_GROUP quantization schemes (e.g., NVFP4), layers that are fused at inference time (Q/K/V projections, gate/up MLP projections) must share the same global_scale. This is handled automatically by observer fusion:
fuse_weight_observers(model)scans the model for known fused layer groups and callsObserver.fuse(observers_and_modules)with(observer, module)tuples- Each observer stores weak references to its fused partners' modules in
_fusions: dict[Observer, ref[Module]] - When
get_qparams()is called on any fused observer: - It automatically observes any fused observers that don't have statistics yet by calling them with their stored module references
- It computes
global_scalefrom the absmax across its own statistics and all fused observers' statistics
This design eliminates the need to explicitly observe all fused modules before calling get_qparams() - unobserved fused modules are handled on-demand. The weak references prevent circular reference cycles between observers and modules. It also handles situations where the subgraphs being quantized at the same time don't include all fused modules.
The fused layer groups are defined in FUSED_MODULE_MAPPINGS (llmcompressor.observers.fused_mappings), which model_free_ptq uses as well. A group applies to a parent module when all of its required layers are present, and its optional layers join when present:
- gate_proj / up_proj, w1 / w3 (MLP and MoE expert layers)
- c_fc_0 / c_fc_1 (EXAONE 3 and 3.5 MLP)
- q_proj / k_proj, optional v_proj (attention; Gemma 4 attention_k_eq_v layers have no v_proj)
- q_proj / k_proj / v_proj / b_proj / f_a_proj, optional g_proj (Kimi delta attention)
- in_proj_qkv / in_proj_z and in_proj_b / in_proj_a (Qwen3.5 gated delta net)
- q_proj / k_proj / v_proj / g_proj, when unpacked a_proj and b_proj are also present (OLMo-Hybrid gated delta net)
- q_a_proj / kv_a_proj_with_mqa, optional g_proj (multi-latent attention; Kimi MLA output gate)
- wq_a / wkv_a_with_mqa (multi-latent attention, Mistral checkpoint names)
- q_a_proj / kv_proj and kv_proj / gate_proj (DeepSeek V4 attention and compressors; checkpoint names wq_a / wkv and wkv / wgate)
- wk / weights_proj (DeepSeek sparse attention indexer)
DDP Synchronization
Each observer subclass declares an _act_sync_dict mapping attribute names to DDP reduce operations. The base class sync_activation_stats() iterates this dict to all-reduce accumulated statistics across ranks:
| Observer | Synced Attributes | Reduce Op |
|---|---|---|
static_minmax |
min_vals, max_vals |
MIN, MAX |
minmax / mse |
min_vals, max_vals |
AVG |
memoryless_minmax / memoryless_mse / nvfp4_expanded_mse |
(none) | — |
imatrix_mse / nvfp4_expanded_imatrix |
_imatrix_sum, _imatrix_count |
SUM |
Quantization Strategies
Observers support multiple quantization strategies via the QuantizationArgs.strategy field:
TENSOR: Statistics computed across the entire tensor.GROUP,TENSOR_GROUP: Tensor sliced into equal-sized groups along columns.TENSOR_GROUPadditionally computes aglobal_scale.CHANNEL: Per-channel statistics (e.g., across output dimensions).TOKEN: Per-token statistics along token or sequence dimensions.BLOCK: Block-wise quantization with configurable block structure.
GPTQ Activation Ordering (actorder)
Column reordering for GPTQ actorder is handled at compression time, after
observer statistics have been collected; it does not change observer ranges or
the statistics accumulated during calibration. When enabled, GPTQ sorts the
working weight columns by descending Hessian diagonal (activation importance),
reorders both axes of the working Hessian, performs the solve, and restores the
original column order before saving. The resulting checkpoint keeps the normal
column order and runtime quantization format; any g_idx used to select group
scales is an internal GPTQ work tensor, not a runtime activation-ordering
requirement.
The currently supported GPTQ values are:
static: the default forGPTQModifier; an alias forweight.weight: activation ordering during the GPTQ solve only. It keeps the normal runtime quantization format and typically gives a small accuracy improvement over no activation ordering without adding runtime latency.None: disables activation ordering.
The older group and dynamic activation-ordering modes have been removed,
and actorder=true is rejected. Set actorder on GPTQModifier or on the
weight QuantizationArgs; conflicting values at both levels raise an error.
Observer Configuration Parameters
Observers can be configured with optional keyword arguments via QuantizationArgs.observer_kwargs.
MinMax observers (minmax, static_minmax, memoryless_minmax)
| Argument | Default | Description |
|---|---|---|
averaging_constant |
0.01 |
EMA weight for moving average observers. Only applies to minmax. Higher values weight recent observations more heavily. |
MSE observers (mse, memoryless_mse)
Both observers use the same search options described in the
MemorylessMSEObserver docstring.
mse additionally smooths successive ranges with averaging_constant;
memoryless_mse processes each observed tensor independently. The expanded
observer inherits these options but overrides some defaults as described above.
| Argument | Default | Description |
|---|---|---|
maxshrink |
0.20 |
Maximum shrink amount in grid steps. Number of search steps is int(maxshrink * grid). |
patience |
5 |
Number of consecutive steps without strict improvement before early stopping in the eager implementation. Triton always evaluates the full grid. |
grid |
100.0 |
Resolution of the shrink search. Higher values give finer granularity. |
norm |
2.4 |
Exponent used when computing the error. norm=2 approximates MSE. |
expand |
1.0 |
Multiplier for the initial min/max range. Must be at least 1.0; the NVFP4 expanded MSE observer defaults to 1.8. |
averaging_constant |
0.01 |
EMA weight for moving average. Only applies to mse. |
grid_type |
None |
"linear", "exponential". default spacing is exponential when MX scales are generated (E8M0), linear otherwise. Exponential candidates are 1, 1/2, 1/4, .... |
magnitude_normalization |
False |
Multiply each powered error by sqrt(w² + group_absmax²), where group_absmax is the maximum absolute observed weight in the quantization group. |
MSE and iMatrix grid searches run with global_scale=None — the shrink search optimizes using FP32 scales, since global_scale cancels out when comparing quantization error across shrink candidates. The actual global_scale is computed later in get_qparams() from the final min/max values.
IMatrix observer (imatrix_mse)
The search options and objective are documented once in the
IMatrixMSEObserver docstring.
The expanded imatrix observer inherits them and changes the default search
range, grid, norm, and patience.
| Argument | Default | Description |
|---|---|---|
maxshrink |
0.95 |
Maximum shrink factor for the grid search. The search evaluates int(maxshrink * grid) shrink steps. |
patience |
5 |
Number of consecutive steps without strict improvement before early stopping in the eager implementation. Triton always evaluates the full grid. |
grid |
100 |
Number of grid steps. Higher values give finer granularity at the cost of speed. |
norm |
2.8 |
Exponent used when computing the importance-weighted error. |
expand |
1.0 |
Multiplier for the initial min/max range. Must be at least 1.0; nvfp4_expanded_imatrix defaults to 1.8. |
strict |
False |
If True, raise an error instead of falling back to uniform MSE when importance data is unavailable. |
grid_type |
None |
"linear", "exponential", default spacing is exponential when MX scales are generated (E8M0), linear otherwise. |
magnitude_normalization |
True |
Multiply each importance-weighted error by sqrt(w² + group_absmax²). |
DDP (Distributed) Support
Observers support distributed data-parallel calibration. When activation quantization runs across multiple ranks (each rank processing a disjoint partition of the calibration dataset), observer statistics must be synchronized before quantization parameters are computed so all ranks produce identical results.
How it works
Each observer class declares a _act_sync_dict class attribute mapping statistic attribute names to their torch.distributed.ReduceOp. The base class sync_activation_stats() method iterates this dict and issues async dist.all_reduce calls:
# Called by QuantizationMixin.sync_obs_act_stats() at each layer boundary
pending_comms = observer.sync_activation_stats()
# ... wait_for_comms(pending_comms) once all observers in the subgraph are queued
Only activation statistics are synchronized. Weight statistics are never synced because weights are identical across all ranks (broadcast during model load).
Per-observer sync behavior
| Observer | Synced statistics | Reduce op | Notes |
|---|---|---|---|
static_minmax |
min_vals, max_vals |
MIN, MAX | Global min/max across all ranks |
minmax (EMA) |
min_vals, max_vals |
AVG | Global min/max across all ranks |
memoryless_minmax |
(none) | — | Stateless; each rank's data is independent |
mse (EMA) |
min_vals, max_vals |
AVG | Averages the per-rank MSE-optimal ranges |
memoryless_mse / nvfp4_expanded_mse |
(none) | — | Stateless; each rank's data is independent |
imatrix_mse / nvfp4_expanded_imatrix |
_imatrix_sum, _imatrix_count |
SUM | Accumulates importance scores across ranks before normalization |
For more information on the distributed oneshot workflow, see Distributed Oneshot.
Example Usage
import torch
from llmcompressor.observers import Observer
from compressed_tensors.quantization import QuantizationArgs
args = QuantizationArgs(num_bits=4, strategy="group", group_size=128)
observer = Observer.load_from_registry(
"minmax",
base_name="weight",
args=args,
)
# Phase 1: accumulate statistics
x = torch.randn(64, 512)
observer(x)
# Phase 2: compute quantization parameters
qparams = observer.get_qparams()
scale = qparams["scale"]
zero_point = qparams["zero_point"]