Skip to content

llmcompressor.entrypoints.model_free

Modules:

Classes:

Functions:

  • model_free_ptq –

    Quantize a model without the need for a model definition. This function

SpinQuantConverter

SpinQuantConverter(
    hidden_size: int,
    head_dim: int,
    norms: dict[str, Tensor],
    tie_word_embeddings: bool = False,
    emit_lm_head: bool = False,
    rotations: tuple[Literal["R1", "R2"], ...] = (
        "R1",
        "R2",
    ),
    transform_type: Literal[
        "hadamard", "random-hadamard"
    ] = "hadamard",
    transform_block_size: int | None = None,
    seed: int = 0,
    precision: dtype = torch.float64,
    mapping: SpinQuantConverterMapping | None = None,
    ignore: list[str] = (),
    num_attention_heads: int | None = None,
    num_key_value_heads: int | None = None,
)

Bases: Converter

Model-free implementation of the offline SpinQuant rotations (https://arxiv.org/abs/2405.16406). Norm scales are fused into the linears which follow them, then R1 is fused into every layer which reads or writes the residual stream and R2 is fused into each attention head's value/output projections. The resulting checkpoint is numerically equivalent to the source model, but has weights which are easier to quantize. No online transforms are added, so the checkpoint loads like any dense (or subsequently quantized) model.

Construct with :meth:from_pretrained and chain before quantization, e.g.

model_free_ptq(
    MODEL_ID,
    SAVE_DIR,
    scheme="MXFP4",
    converter=[
        FP8BlockDequantizer(),
        SpinQuantConverter.from_pretrained(MODEL_ID),
    ],
)

Parameters:

  • hidden_size (int) –

    size of the residual stream

  • head_dim (int) –

    size of each attention head

  • norms (dict[str, Tensor]) –

    module name -> norm weight, for every norm being fused

  • tie_word_embeddings (bool, default: False ) –

    whether the source model ties lm_head to the embedding. If so, the saved config is untied, since the rotated lm_head and embedding differ

  • emit_lm_head (bool, default: False ) –

    whether to create lm_head.weight from the embedding, for tied checkpoints which do not save lm_head.weight

  • rotations (tuple[Literal['R1', 'R2'], ...], default: ('R1', 'R2') ) –

    offline rotations to apply. from_pretrained defaults an MLA model (one with kv_lora_rank) to R1 only: R2 needs an uncompressed num_heads * head_dim path between v_proj and o_proj, which MLA replaces with a compressed latent

  • transform_type (Literal['hadamard', 'random-hadamard'], default: 'hadamard' ) –

    "hadamard" supports power-of-two sizes, "random-hadamard" also supports sizes with a known hadamard divisor

  • transform_block_size (int | None, default: None ) –

    R1 block size. Defaults to hidden_size. The rotation is block-diagonal when smaller than hidden_size

  • seed (int, default: 0 ) –

    seed for "random-hadamard"

  • precision (dtype, default: float64 ) –

    dtype used to fuse rotations into weights

  • mapping (SpinQuantConverterMapping | None, default: None ) –

    names of the layers to rotate

  • ignore (list[str], default: () ) –

    regexes of tensor names which are left untouched, for modules outside of the rotated residual stream such as a vision tower

Methods:

  • from_pretrained –

    Build the converter from a checkpoint's config.json and norm weights. Norm

Source code in src/llmcompressor/entrypoints/model_free/spinquant.py
def __init__(
    self,
    hidden_size: int,
    head_dim: int,
    norms: dict[str, torch.Tensor],
    tie_word_embeddings: bool = False,
    emit_lm_head: bool = False,
    rotations: tuple[Literal["R1", "R2"], ...] = ("R1", "R2"),
    transform_type: Literal["hadamard", "random-hadamard"] = "hadamard",
    transform_block_size: int | None = None,
    seed: int = 0,
    precision: torch.dtype = torch.float64,
    mapping: SpinQuantConverterMapping | None = None,
    ignore: list[str] = (),
    num_attention_heads: int | None = None,
    num_key_value_heads: int | None = None,
):
    rotations = tuple(r.upper() for r in rotations)
    if not rotations or not set(rotations) <= {"R1", "R2"}:
        raise ValueError(
            f"SpinQuantConverter supports R1 and R2 only, got {rotations}"
        )

    self.hidden_size = hidden_size
    self.head_dim = head_dim
    self.norms = norms
    self.tie_word_embeddings = tie_word_embeddings
    self.emit_lm_head = emit_lm_head
    self.rotations = rotations
    self.precision = precision
    self.mapping = mapping or SpinQuantConverterMapping()

    block_size = transform_block_size or hidden_size
    if hidden_size % block_size != 0:
        raise ValueError(
            f"transform_block_size {block_size} must divide "
            f"hidden_size {hidden_size}"
        )
    generator = torch.Generator().manual_seed(seed)
    self.r1 = _hadamard(block_size, transform_type, precision, generator)
    # R2's head-sized matrix is only needed, and only constructible for every
    # head_dim, when R2 is actually applied.
    self.r2 = (
        _hadamard(head_dim, transform_type, precision, generator)
        if "R2" in rotations
        else None
    )
    self.r2_v_dim = (
        None if num_key_value_heads is None else num_key_value_heads * head_dim
    )
    self.r2_o_dim = (
        None if num_attention_heads is None else num_attention_heads * head_dim
    )

    m = self.mapping
    self._embedding = re.compile(m.embedding)
    self._lm_head = re.compile(m.lm_head)
    self._attn_in = [re.compile(p) for p in m.attn_in]
    self._attn_out = [re.compile(p) for p in m.attn_out]
    self._mlp_in = [re.compile(p) for p in m.mlp_in]
    self._mlp_out = [re.compile(p) for p in m.mlp_out]
    self._attn_v = re.compile(m.attn_v)
    self._attn_o = re.compile(m.attn_o)
    self._residual = [re.compile(p) for p in m.residual]
    self._ignore = [re.compile(p) for p in ignore]

from_pretrained classmethod

from_pretrained(
    model_name_or_path: str | PathLike, **kwargs
) -> SpinQuantConverter

Build the converter from a checkpoint's config.json and norm weights. Norm weights are small, so they are loaded up front rather than declared as dependencies of every linear which reads them.

Parameters:

  • model_name_or_path (str | PathLike) –

    HuggingFace stub or local checkpoint path

  • kwargs –

    additional arguments passed to the constructor

Source code in src/llmcompressor/entrypoints/model_free/spinquant.py
@classmethod
def from_pretrained(
    cls, model_name_or_path: str | os.PathLike, **kwargs
) -> "SpinQuantConverter":
    """
    Build the converter from a checkpoint's config.json and norm weights. Norm
    weights are small, so they are loaded up front rather than declared as
    dependencies of every linear which reads them.

    :param model_name_or_path: HuggingFace stub or local checkpoint path
    :param kwargs: additional arguments passed to the constructor
    """
    model_files = get_checkpoint_files(model_name_or_path)
    weight_map = get_weight_map(model_files)
    config = load_model_config(model_files)
    text_config = config.get("text_config", config)

    hidden_size = text_config["hidden_size"]
    head_dim = text_config.get("head_dim") or (
        hidden_size // text_config["num_attention_heads"]
    )
    tie_word_embeddings = config.get(
        "tie_word_embeddings", text_config.get("tie_word_embeddings", False)
    )

    mapping = kwargs.get("mapping") or SpinQuantConverterMapping()
    ignore = [re.compile(p) for p in kwargs.get("ignore", ())]
    names = [name for name in weight_map if not _search_any(ignore, name)]
    lm_head = re.compile(mapping.lm_head)
    has_lm_head = any(
        lm_head.search(name.removesuffix(".weight"))
        for name in names
        if name.endswith(".weight")
    )

    norm_names = _find_norm_names(names, mapping)
    if not norm_names:
        raise ValueError("No norm weights found to fuse into linear layers")

    names_by_file: dict[str, list[str]] = defaultdict(list)
    for name in norm_names:
        names_by_file[model_files[weight_map[name]]].append(name)

    norms = {}
    for path, names in names_by_file.items():
        with safe_open(path, framework="pt") as file:
            for name in names:
                norms[name.removesuffix(".weight")] = file.get_tensor(name)

    if "rotations" not in kwargs and _is_mla(text_config):
        kwargs["rotations"] = ("R1",)
        logger.warning(
            "MLA model detected (kv_lora_rank is set). SpinQuant R2 needs an "
            "uncompressed head dimension between v_proj and o_proj, so "
            "rotations default to R1 only. Pass rotations explicitly to override"
        )
    kwargs.setdefault("num_attention_heads", text_config.get("num_attention_heads"))
    kwargs.setdefault("num_key_value_heads", text_config.get("num_key_value_heads"))

    return cls(
        hidden_size=hidden_size,
        head_dim=head_dim,
        norms=norms,
        tie_word_embeddings=tie_word_embeddings,
        emit_lm_head=tie_word_embeddings and not has_lm_head,
        **kwargs,
    )

SpinQuantConverterMapping

Bases: BaseModel

Name-based description of where SpinQuant rotations are fused in a checkpoint. Patterns are regexes searched against module names (tensor names without the trailing .weight / .bias). Defaults target pre-norm decoders with Llama-style names, including MoE routers and experts.

Parameters:

  • embedding –

    embedding whose rows are rotated by R1

  • attn_in –

    attention layers reading the residual stream

  • attn_out –

    attention layers writing the residual stream

  • attn_v –

    layer whose output heads are rotated by R2

  • attn_o –

    layer whose input heads are rotated by R2

  • mlp_in –

    mlp layers reading the residual stream

  • mlp_out –

    mlp layers writing the residual stream

  • lm_head –

    language model head

  • residual –

    bare parameters, without a .weight suffix, whose last dimension is one or more residual streams. Each stream is rotated by R1, i.e. a last dimension of k * hidden_size is rotated by a block-diagonal R1. Defaults include DeepSeek-V4 mHC mixers (hc_attn_fn, hc_ffn_fn, hc_head_fn)

  • attn_norm –

    norm fused into attn_in, relative to the decoder layer

  • mlp_norm –

    norm fused into mlp_in, relative to the decoder layer

  • final_norm –

    norm fused into lm_head, relative to the decoder prefix

model_free_ptq

model_free_ptq(
    model_stub: str | PathLike,
    save_directory: str | PathLike,
    scheme: QuantizationScheme | str | None = None,
    config: QuantizationConfig | None = None,
    ignore: Iterable[str] = tuple(),
    max_workers: int | Literal["auto"] = "auto",
    device: Optional[
        str | device | list[str | device]
    ] = None,
    converter: Converter
    | list[Converter]
    | tuple[Converter, ...]
    | None = None,
)

Quantize a model without the need for a model definition. This function operates on a model stub or folder containing weights saved in safetensors files.

For microscale schemes (NVFP4, MXFP4), fused weight sets (q/k/v, gate/up) are handled correctly even when split across shards. Each shard job receives a precomputed inverse_weight_map specifying exactly which tensors to load from which files — enabling true partial reads with no runtime discovery and no redundant tensor reads.

Parameters:

  • model_stub (str | PathLike) –

    huggingface model hub or path to local weights files

  • save_directory (str | PathLike) –

    directory to save quantized weights to

  • scheme (QuantizationScheme | str | None, default: None ) –

    weight quantization scheme or preset scheme name. Mutually exclusive with config.

  • config (QuantizationConfig | None, default: None ) –

    quantization config containing one or more schemes and optional kv cache quantization. Mutually exclusive with scheme.

  • ignore (Iterable[str], default: tuple() ) –

    modules to ignore. Modules ending with "norm" are automatically ignored

  • max_workers (int | Literal['auto'], default: 'auto' ) –

    maximum number of concurrent worker threads. Effective concurrency may be lower when GPU memory is tight. If "auto" (default), the number of workers is chosen from the number of safetensors files, the estimated memory of each job, the free memory of each device, and the number of CPUs available. Host memory is not taken into account, so pass a smaller number of workers if quantization runs out of host memory.

  • device (Optional[str | device | list[str | device]], default: None ) –

    device(s) for quantization. Accepts a single device string/object or a list. When multiple devices are given, shards are dynamically assigned based on real-time GPU memory.

  • converter (Converter | list[Converter] | tuple[Converter, ...] | None, default: None ) –

    optional converter to apply to the checkpoint before running model-free PTQ. Accepts a single converter, a list, or a tuple; they run in the given order and model-free PTQ is applied last. A single converter behaves as before. For example, pass an fp8 dequantizer followed by a SpinQuantConverter

Source code in src/llmcompressor/entrypoints/model_free/__init__.py
def model_free_ptq(
    model_stub: str | os.PathLike,
    save_directory: str | os.PathLike,
    scheme: QuantizationScheme | str | None = None,
    config: QuantizationConfig | None = None,
    ignore: Iterable[str] = tuple(),
    max_workers: int | Literal["auto"] = "auto",
    device: Optional[str | torch.device | list[str | torch.device]] = None,
    converter: Converter | list[Converter] | tuple[Converter, ...] | None = None,
):
    """
    Quantize a model without the need for a model definition. This function
    operates on a model stub or folder containing weights saved in safetensors
    files.

    For microscale schemes (NVFP4, MXFP4), fused weight sets (q/k/v, gate/up)
    are handled correctly even when split across shards. Each shard job receives
    a precomputed inverse_weight_map specifying exactly which tensors to load
    from which files — enabling true partial reads with no runtime discovery
    and no redundant tensor reads.

    :param model_stub: huggingface model hub or path to local weights files
    :param save_directory: directory to save quantized weights to
    :param scheme: weight quantization scheme or preset scheme name.
        Mutually exclusive with config.
    :param config: quantization config containing one or more schemes and
        optional kv cache quantization. Mutually exclusive with scheme.
    :param ignore: modules to ignore. Modules ending with "norm" are
        automatically ignored
    :param max_workers: maximum number of concurrent worker threads.
        Effective concurrency may be lower when GPU memory is tight. If "auto"
        (default), the number of workers is chosen from the number of
        safetensors files, the estimated memory of each job, the free memory of
        each device, and the number of CPUs available. Host memory is not taken
        into account, so pass a smaller number of workers if quantization runs
        out of host memory.
    :param device: device(s) for quantization. Accepts a single device
        string/object or a list. When multiple devices are given, shards
        are dynamically assigned based on real-time GPU memory.
    :param converter: optional converter to apply to the checkpoint before
        running model-free PTQ. Accepts a single converter, a list, or a tuple;
        they run in the given order and model-free PTQ is applied last. A
        single converter behaves as before. For example, pass an fp8
        dequantizer followed by a SpinQuantConverter
    """
    model_files = get_checkpoint_files(model_stub)
    config = validate_config(config, scheme, ignore)
    validate_safetensors_index(model_files, config)

    weight_map = get_weight_map(model_files)
    mfptq = ModelFreePtqConverter(config, weight_names=weight_map.keys())
    if converter is None:
        converters = []
    elif isinstance(converter, (list, tuple)):
        converters = list(converter)
    else:
        converters = [converter]
    converters.append(mfptq)

    convert_checkpoint(
        model_stub=model_stub,
        save_directory=save_directory,
        converter=converters,
        max_workers=max_workers,
        device=device,
    )