llmcompressor.entrypoints.model_free
Modules:
-
converter– -
helpers– -
microscale– -
reindex_fused_weights– -
spinquant–
Classes:
-
SpinQuantConverter–Model-free implementation of the offline SpinQuant rotations
-
SpinQuantConverterMapping–Name-based description of where SpinQuant rotations are fused in a checkpoint.
Functions:
-
model_free_ptq–Quantize a model without the need for a model definition. This function
SpinQuantConverter
SpinQuantConverter(
hidden_size: int,
head_dim: int,
norms: dict[str, Tensor],
tie_word_embeddings: bool = False,
emit_lm_head: bool = False,
rotations: tuple[Literal["R1", "R2"], ...] = (
"R1",
"R2",
),
transform_type: Literal[
"hadamard", "random-hadamard"
] = "hadamard",
transform_block_size: int | None = None,
seed: int = 0,
precision: dtype = torch.float64,
mapping: SpinQuantConverterMapping | None = None,
ignore: list[str] = (),
num_attention_heads: int | None = None,
num_key_value_heads: int | None = None,
)
Bases: Converter
Model-free implementation of the offline SpinQuant rotations (https://arxiv.org/abs/2405.16406). Norm scales are fused into the linears which follow them, then R1 is fused into every layer which reads or writes the residual stream and R2 is fused into each attention head's value/output projections. The resulting checkpoint is numerically equivalent to the source model, but has weights which are easier to quantize. No online transforms are added, so the checkpoint loads like any dense (or subsequently quantized) model.
Construct with :meth:from_pretrained and chain before quantization, e.g.
model_free_ptq(
MODEL_ID,
SAVE_DIR,
scheme="MXFP4",
converter=[
FP8BlockDequantizer(),
SpinQuantConverter.from_pretrained(MODEL_ID),
],
)
Parameters:
-
hidden_size(int) –size of the residual stream
-
head_dim(int) –size of each attention head
-
norms(dict[str, Tensor]) –module name -> norm weight, for every norm being fused
-
tie_word_embeddings(bool, default:False) –whether the source model ties
lm_headto the embedding. If so, the saved config is untied, since the rotatedlm_headand embedding differ -
emit_lm_head(bool, default:False) –whether to create
lm_head.weightfrom the embedding, for tied checkpoints which do not savelm_head.weight -
rotations(tuple[Literal['R1', 'R2'], ...], default:('R1', 'R2')) –offline rotations to apply.
from_pretraineddefaults an MLA model (one withkv_lora_rank) to R1 only: R2 needs an uncompressednum_heads * head_dimpath betweenv_projando_proj, which MLA replaces with a compressed latent -
transform_type(Literal['hadamard', 'random-hadamard'], default:'hadamard') –"hadamard"supports power-of-two sizes,"random-hadamard"also supports sizes with a known hadamard divisor -
transform_block_size(int | None, default:None) –R1 block size. Defaults to
hidden_size. The rotation is block-diagonal when smaller thanhidden_size -
seed(int, default:0) –seed for
"random-hadamard" -
precision(dtype, default:float64) –dtype used to fuse rotations into weights
-
mapping(SpinQuantConverterMapping | None, default:None) –names of the layers to rotate
-
ignore(list[str], default:()) –regexes of tensor names which are left untouched, for modules outside of the rotated residual stream such as a vision tower
Methods:
-
from_pretrained–Build the converter from a checkpoint's config.json and norm weights. Norm
Source code in src/llmcompressor/entrypoints/model_free/spinquant.py
from_pretrained
classmethod
Build the converter from a checkpoint's config.json and norm weights. Norm weights are small, so they are loaded up front rather than declared as dependencies of every linear which reads them.
Parameters:
-
model_name_or_path(str | PathLike) –HuggingFace stub or local checkpoint path
-
kwargs–additional arguments passed to the constructor
Source code in src/llmcompressor/entrypoints/model_free/spinquant.py
SpinQuantConverterMapping
Bases: BaseModel
Name-based description of where SpinQuant rotations are fused in a checkpoint.
Patterns are regexes searched against module names (tensor names without the
trailing .weight / .bias). Defaults target pre-norm decoders with
Llama-style names, including MoE routers and experts.
Parameters:
-
embedding–embedding whose rows are rotated by R1
-
attn_in–attention layers reading the residual stream
-
attn_out–attention layers writing the residual stream
-
attn_v–layer whose output heads are rotated by R2
-
attn_o–layer whose input heads are rotated by R2
-
mlp_in–mlp layers reading the residual stream
-
mlp_out–mlp layers writing the residual stream
-
lm_head–language model head
-
residual–bare parameters, without a
.weightsuffix, whose last dimension is one or more residual streams. Each stream is rotated by R1, i.e. a last dimension ofk * hidden_sizeis rotated by a block-diagonal R1. Defaults include DeepSeek-V4 mHC mixers (hc_attn_fn,hc_ffn_fn,hc_head_fn) -
attn_norm–norm fused into
attn_in, relative to the decoder layer -
mlp_norm–norm fused into
mlp_in, relative to the decoder layer -
final_norm–norm fused into
lm_head, relative to the decoder prefix
model_free_ptq
model_free_ptq(
model_stub: str | PathLike,
save_directory: str | PathLike,
scheme: QuantizationScheme | str | None = None,
config: QuantizationConfig | None = None,
ignore: Iterable[str] = tuple(),
max_workers: int | Literal["auto"] = "auto",
device: Optional[
str | device | list[str | device]
] = None,
converter: Converter
| list[Converter]
| tuple[Converter, ...]
| None = None,
)
Quantize a model without the need for a model definition. This function operates on a model stub or folder containing weights saved in safetensors files.
For microscale schemes (NVFP4, MXFP4), fused weight sets (q/k/v, gate/up) are handled correctly even when split across shards. Each shard job receives a precomputed inverse_weight_map specifying exactly which tensors to load from which files — enabling true partial reads with no runtime discovery and no redundant tensor reads.
Parameters:
-
model_stub(str | PathLike) –huggingface model hub or path to local weights files
-
save_directory(str | PathLike) –directory to save quantized weights to
-
scheme(QuantizationScheme | str | None, default:None) –weight quantization scheme or preset scheme name. Mutually exclusive with config.
-
config(QuantizationConfig | None, default:None) –quantization config containing one or more schemes and optional kv cache quantization. Mutually exclusive with scheme.
-
ignore(Iterable[str], default:tuple()) –modules to ignore. Modules ending with "norm" are automatically ignored
-
max_workers(int | Literal['auto'], default:'auto') –maximum number of concurrent worker threads. Effective concurrency may be lower when GPU memory is tight. If "auto" (default), the number of workers is chosen from the number of safetensors files, the estimated memory of each job, the free memory of each device, and the number of CPUs available. Host memory is not taken into account, so pass a smaller number of workers if quantization runs out of host memory.
-
device(Optional[str | device | list[str | device]], default:None) –device(s) for quantization. Accepts a single device string/object or a list. When multiple devices are given, shards are dynamically assigned based on real-time GPU memory.
-
converter(Converter | list[Converter] | tuple[Converter, ...] | None, default:None) –optional converter to apply to the checkpoint before running model-free PTQ. Accepts a single converter, a list, or a tuple; they run in the given order and model-free PTQ is applied last. A single converter behaves as before. For example, pass an fp8 dequantizer followed by a SpinQuantConverter