Skip to content

llmcompressor.observers.mse

Classes:

MemorylessMSEObserver 

MemorylessMSEObserver(*args, **kwargs)

Bases: Observer

Compute quantization parameters by finding the optimal min/max values which minimize the mean of quantization error squared.

Source code in src/llmcompressor/observers/mse.py 
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
def __init__(self, *args, **kwargs):
    super().__init__(*args, **kwargs)
    observer_kwargs = self.args.observer_kwargs
    self.maxshrink = observer_kwargs.get("maxshrink", 0.20)
    self.patience = observer_kwargs.get("patience", 5)
    self.grid = observer_kwargs.get("grid", 100.0)
    self.norm = observer_kwargs.get("norm", 2.4)
    self.triton_error_buffer = observer_kwargs.get(
        "triton_error_buffer", _default_triton_error_buffer(self.args)
    )
    self.expand = observer_kwargs.get("expand", 1.0)
    if self.expand < 1.0:
        raise ValueError(f"expand value must be at least 1.0, got {self.expand}")

    # Pre-create token_args to avoid patch_attr context manager
    # which causes torch.compile graph breaks
    self._token_args = self.args.model_copy(
        update={"strategy": QuantizationStrategy.TOKEN}
    )

MovingAverageMSEObserver 

MovingAverageMSEObserver(*args, **kwargs)

Bases: Observer

Compute quantization parameters by finding the optimal min/max values which minimize the mean of quantization error squared, with moving average smoothing.

Source code in src/llmcompressor/observers/mse.py 
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
def __init__(self, *args, **kwargs):
    super().__init__(*args, **kwargs)
    self.avg_constant = self.args.observer_kwargs.get("averaging_constant", 0.01)
    observer_kwargs = self.args.observer_kwargs
    self.maxshrink = observer_kwargs.get("maxshrink", 0.20)
    self.patience = observer_kwargs.get("patience", 5)
    self.grid = observer_kwargs.get("grid", 100.0)
    self.norm = observer_kwargs.get("norm", 2.4)
    self.triton_error_buffer = observer_kwargs.get(
        "triton_error_buffer", _default_triton_error_buffer(self.args)
    )
    self.expand = observer_kwargs.get("expand", 1.0)
    if self.expand < 1.0:
        raise ValueError(f"expand value must be at least 1.0, got {self.expand}")

    # Pre-create token_args to avoid patch_attr context manager
    # which causes torch.compile graph breaks
    self._token_args = self.args.model_copy(
        update={"strategy": QuantizationStrategy.TOKEN}
    )

NVFP4ExpandedMSEObserver 

NVFP4ExpandedMSEObserver(*args, **kwargs)

Bases: MemorylessMSEObserver

MSE observer with defaults tuned for NVFP4 range expansion.

Searches from expand times the observed range down to (1 - maxshrink) * expand times the observed range. With the defaults below, this covers 1.8x down to ~0.8x of the original per-group range in 112 search steps.

Usage::

QuantizationArgs(
    ...
    observer="nvfp4_expanded_mse",
)
Source code in src/llmcompressor/observers/mse.py 
138
139
140
141
142
143
144
145
def __init__(self, *args, **kwargs):
    super().__init__(*args, **kwargs)
    observer_kwargs = self.args.observer_kwargs
    self.expand = observer_kwargs.get("expand", 1.8)
    self.maxshrink = observer_kwargs.get("maxshrink", 1 - 0.8 / 1.8)
    self.grid = observer_kwargs.get("grid", 200.0)
    self.patience = observer_kwargs.get("patience", 1000)
    self.triton_error_buffer = observer_kwargs.get("triton_error_buffer", 1.00)