Skip to content

llmcompressor.observers.imatrix

Classes:

  • IMatrixMSEObserver –

    Search candidate min/max ranges using per-input-channel activation

IMatrixMSEObserver

IMatrixMSEObserver(*args, **kwargs)

Bases: Observer

Search candidate min/max ranges using per-input-channel activation importance (E[x²]) to weight QDQ error.

Supports CHANNEL, GROUP, and TENSOR_GROUP for weight-only Linear modules. Falls back to uniform MSE when importance data is unavailable.

Importance is accumulated on the observer as raw _imatrix_sum / _imatrix_count and synced across DDP ranks via _act_sync_dict before observation.

The base objective is importance * abs(QDQ(w) - w) ** norm, summed over each quantization group. If magnitude_normalization is enabled, it also multiplies each element by sqrt(w**2 + group_absmax**2). Importance is accumulated from observer inputs as E[x²].

Observer kwargs are read from QuantizationArgs.observer_kwargs: maxshrink (float): Maximum fraction of range to shrink. Defaults to 0.95. patience (int): Number of consecutive candidates without a strict improvement before early stopping in the eager implementation. Triton always evaluates the full grid. 0 runs the full eager search. Defaults to 5. grid (float): Linear grid resolution or exponential candidate budget. Defaults to 100. norm (float): Power applied to absolute QDQ error before importance weighting. Defaults to 2.8. expand (float): Multiplier applied to observed min/max before the shrink search; must be at least 1. Defaults to 1.0. strict (bool): Raise an error if importance is unavailable or has an unsupported shape. If false, warn and fall back to uniform MSE. Defaults to False. grid_type (str or None): "linear" or "exponential". None selects exponential spacing when MX scales are generated (E8M0), and linear spacing otherwise. Exponential candidates are powers of two starting at 1. Defaults to None. magnitude_normalization (bool): Multiply each error by sqrt(w**2 + group_absmax**2). Defaults to True.

Methods:

  • attach –

    Attach a forward-pre hook to accumulate E[x²] per input channel.

  • detach –

    Remove the activation collection hook.

Source code in src/llmcompressor/observers/imatrix.py
def __init__(self, *args, **kwargs):
    super().__init__(*args, **kwargs)
    kw = self.args.observer_kwargs
    self.maxshrink = kw.get("maxshrink", 0.95)
    self.patience = kw.get("patience", 5)
    self.grid = kw.get("grid", 100)
    self.norm = kw.get("norm", 2.8)
    self.strict = kw.get("strict", False)
    self.expand = kw.get("expand", 1.0)
    self.magnitude_normalization = kw.get("magnitude_normalization", True)
    self.grid_type = _resolve_grid_type(kw.get("grid_type"), self.args)

    self._imatrix_sum: torch.Tensor | None = None
    self._imatrix_count: torch.Tensor = torch.tensor(0, dtype=torch.int64)
    self._imatrix_hook: RemovableHandle | None = None

    if self.grid <= 0:
        raise ValueError(f"grid must be > 0, got {self.grid}")
    if self.patience < 0:
        raise ValueError(f"patience must be >= 0, got {self.patience}")
    if not (0 <= self.maxshrink <= 1):
        raise ValueError(f"maxshrink must be in [0, 1], got {self.maxshrink}")
    if not isinstance(self.magnitude_normalization, bool):
        raise ValueError(
            "magnitude_normalization must be a bool, got "
            f"{self.magnitude_normalization!r}"
        )
    if (
        not isinstance(self.norm, (int, float))
        or not math.isfinite(self.norm)
        or self.norm <= 0
    ):
        raise ValueError(f"norm must be a finite positive number, got {self.norm}")

attach

attach(module: Module) -> None

Attach a forward-pre hook to accumulate E[x²] per input channel.

Source code in src/llmcompressor/observers/imatrix.py
def attach(self, module: torch.nn.Module) -> None:
    """Attach a forward-pre hook to accumulate E[x²] per input channel."""
    if self._imatrix_hook is not None:
        self._imatrix_hook.remove()
        self._imatrix_hook = None

    if not hasattr(module, "in_features"):
        return

    in_features = module.in_features
    param = next(module.parameters(), None)
    device = param.device if param is not None else None
    self._imatrix_sum = torch.zeros(
        in_features, dtype=IMATRIX_PRECISION, device=device
    )
    self._imatrix_count = torch.tensor(0, dtype=torch.int64, device=device)

    def _hook(mod, args):
        if (
            HooksMixin._HOOKS_DISABLED
            and getattr(self, "_imatrix_hook", None)
            not in HooksMixin._HOOKS_KEEP_ENABLED
        ):
            return
        x = args[0] if isinstance(args, tuple) else args
        if isinstance(x, tuple):
            x = x[0]
        if x is None or not isinstance(x, torch.Tensor):
            return

        x_f = x.detach().to(IMATRIX_PRECISION)
        device = x_f.device
        n_tokens = math.prod(x_f.shape[:-1])
        token_sum = x_f.pow(2).sum(dim=list(range(x_f.dim() - 1)))

        self._imatrix_sum = self._imatrix_sum.to(device)
        self._imatrix_count = self._imatrix_count.to(device)

        self._imatrix_sum.add_(token_sum)
        self._imatrix_count += n_tokens

    self._imatrix_hook = module.register_forward_pre_hook(_hook)

detach

detach(module: Module) -> None

Remove the activation collection hook.

Source code in src/llmcompressor/observers/imatrix.py
def detach(self, module: torch.nn.Module) -> None:
    """Remove the activation collection hook."""
    if self._imatrix_hook is not None:
        self._imatrix_hook.remove()
        self._imatrix_hook = None

NVFP4ExpandedIMatrixObserver

NVFP4ExpandedIMatrixObserver(*args, **kwargs)

Bases: IMatrixMSEObserver

IMatrix observer with candidate range and grid defaults tuned for NVFP4.

Covers 1.8x down to ~0.8x of the per-group range, matching the candidate range of :class:NVFP4ExpandedMSEObserver. It retains imatrix's default magnitude normalization.

Usage::

QuantizationArgs(
    ...
    observer="nvfp4_expanded_imatrix",
)
Source code in src/llmcompressor/observers/imatrix.py
def __init__(self, *args, **kwargs):
    super().__init__(*args, **kwargs)
    kw = self.args.observer_kwargs
    if should_generate_mx_scales(self.args):
        self.expand = kw.get("expand", 1.0)
        self.maxshrink = kw.get("maxshrink", 1.0 - 1.0 / 64.0)
    else:
        self.expand = kw.get("expand", 1.8)
        self.maxshrink = kw.get("maxshrink", 1 - 0.8 / 1.8)
    self.grid = kw.get("grid", 200)
    self.norm = kw.get("norm", 2.4)
    self.patience = kw.get("patience", 1000)