llmcompressor.modifiers.qad

Bases: Modifier

Quantization-aware distillation (QAD) trains the weights of each sequential subgraph's targets, such as decoder layers, so that their fake-quantized outputs match their unquantized outputs. A subgraph's targets train jointly as a chain: each target takes the previous target's output, and the loss compares the last target's outputs. sequential_targets_per_subgraph sets how many targets train together; training memory grows with it. For each subgraph, this modifier:

1. Caches the targets' inputs and the last target's unquantized outputs for every calibration batch. The unquantized outputs are the teacher; no teacher model is loaded
2. Splits the cached batches into training and validation batches
3. After the preceding modifier calibrates the targets' weight qparams, trains their weights through compressed-tensors fake quantization, re-observing weight qparams before training and after every epoch
4. Restores the weights with the lowest validation loss, which may be the untrained weights
5. Re-observes weight qparams and writes the fake-quantized weights back

With propagate_error=True (default, recommended), each subgraph trains on the preceding subgraphs' quantized outputs, matching its inputs at inference. With propagate_error=False, it trains on the original model's activations. Validation batches are withheld from QAD updates but still calibrate the quantizer.

Place QADModifier after a weight quantization modifier in the same recipe and use the sequential pipeline. RTN and GPTQ are tested. Other preceding methods must keep weights unquantized during calibration, initialize qparams before QAD's sequential_epoch_end, and use floating weights compatible with compressed-tensors fake quantization.

Sample usage::

from llmcompressor import oneshot
from llmcompressor.modifiers.qad import QADModifier
from llmcompressor.modifiers.quantization import QuantizationModifier

recipe = [
    QuantizationModifier(
        targets="Linear", scheme="NVFP4A16", ignore=["lm_head"]
    ),
    QADModifier(num_epochs=1, lr=2e-6),
]
oneshot(model=model, dataset=ds, recipe=recipe, pipeline="sequential")

Parameters:

• reobserve_weights –

  re-observe weight qparams before training, after every epoch, and before writing the final weights. Requires the preceding modifier to keep live weight observers. Set to False to keep its qparams fixed. Defaults to True

• num_epochs –

  training epochs per subgraph. Defaults to 1

• lr –

  AdamW learning rate. Defaults to 2e-6

• weight_decay –

  AdamW weight decay. Defaults to 0

• gradient_accumulation_steps –

  calibration batches per optimizer update. Defaults to 1

• max_grad_norm –

  gradient norm clipping threshold, or None to disable clipping. Defaults to 1.0

• seed –

  seed for the validation split and per-epoch shuffling. Defaults to 42

• offload_device –

  device that holds cached batches and best-weight snapshots between uses. Defaults to "cpu"

• validation_fraction –

  fraction of calibration batches held out from training to select the best epoch; at least one batch is held out. Defaults to 0.1