Skip to content

speculators.config

Configuration classes for Speculators library.

This module contains the configuration classes for speculative decoding implementations in the Speculators library. These includes configurations for token proposal methods, verifier models, speculative decoding algorithms, and speculator models.

The configurations use Pydantic for validation, serialization, and deserialization, and extend Hugging Face's PretrainedConfig where appropriate to maintain compatibility with the transformers ecosystem.

Classes: TokenProposalConfig: Base configuration for token proposal methods VerifierConfig: Configuration for verifier models with compatibility validation SpeculatorsConfig: Configuration for speculative decoding implementations SpeculatorModelConfig: Configuration for speculator models with transformers compatibility

Classes:

  • SpeculatorModelConfig

    The base config for a speculator model and implementation which defines the

  • SpeculatorsConfig

    The base config for a spec decode implementation which defines the parameters

  • VerifierConfig

    The base config for a verifier model which defines the parameters that are required

Functions:

  • reload_schemas

    Automatically populates the registry for all PydanticClassRegistryMixin subclasses

SpeculatorModelConfig

SpeculatorModelConfig(**kwargs)

Bases: PydanticClassRegistryMixin, PretrainedConfig

The base config for a speculator model and implementation which defines the hyperparameters and settings required to implement a speculator model. It includes details on the speculator model architecture along with the speculators config describing the algorithm, token proposals, and verifier model.

It inherits from the Transformers PretrainedConfig class to ensure full compatibility with standard Transformers model pathways while building on the standard methods for PretrainedConfigs to load, save, and push to the HF hub.

This is the main config which maps to the config.json file for saved speculators.

Methods:

  • from_dict

    Create a SpeculatorModelConfig from a dictionary, automatically instantiating

  • from_pretrained

    Load a SpeculatorModelConfig from the name/id of a model on the Hugging Face Hub

  • to_dict

    :return: A dictionary representation of the full config, including the

  • to_diff_dict

    :return: A dictionary representation of a simplified config,

Source code in speculators/config.py
def __init__(self, **kwargs):
    # initialize the Pydantic arguments first to set all valid fields
    PydanticClassRegistryMixin.__init__(self, **kwargs)

    # reset kwargs handled by Pydantic so PretrainedConfig doesn't override
    for field in self.__class__.model_fields:
        kwargs[field] = getattr(self, field)

    # strip ClassVars so PretrainedConfig.__post_init__ doesn't try to
    # setattr them (pydantic blocks setattr on ClassVar names)
    class_vars = self.__class__.__class_vars__
    for cv in class_vars:
        kwargs.pop(cv, None)

    # initialize the Hugging Face PretrainedConfig arguments for the model
    PretrainedConfig.__init__(self, **kwargs)

    # ensure we always update the transformers version
    self.transformers_version = version("transformers")

from_dict classmethod

from_dict(
    config_dict: dict[str, Any], **kwargs
) -> SpeculatorModelConfig

Create a SpeculatorModelConfig from a dictionary, automatically instantiating the correct subclass based on the speculators_model_type field.

Parameters:

  • config_dict

    (dict[str, Any]) –

    Dictionary containing the configuration

  • kwargs

    Additional keyword arguments that override config values

Returns:

Source code in speculators/config.py
@classmethod
def from_dict(
    cls, config_dict: dict[str, Any], **kwargs
) -> "SpeculatorModelConfig":
    """
    Create a SpeculatorModelConfig from a dictionary, automatically instantiating
    the correct subclass based on the speculators_model_type field.

    :param config_dict: Dictionary containing the configuration
    :param kwargs: Additional keyword arguments that override config values
    :return: A SpeculatorModelConfig instance of the appropriate subclass
    """
    dict_obj = {**config_dict, **kwargs}

    if "speculators_model_type" not in dict_obj:
        raise ValueError(
            "The config dictionary must contain the 'speculators_model_type' field "
            "for loading a SpeculatorModelConfig in the Speculators library."
        )

    return cls.model_validate(dict_obj)

from_pretrained classmethod

from_pretrained(
    pretrained_model_name_or_path: str | PathLike,
    cache_dir: str | PathLike | None = None,
    force_download: bool = False,
    local_files_only: bool = False,
    token: str | bool | None = None,
    revision: str = "main",
    **kwargs,
) -> SpeculatorModelConfig

Load a SpeculatorModelConfig from the name/id of a model on the Hugging Face Hub or from a local directory. Will automatically instantiate the correct config from speculators.models package.

Parameters:

  • pretrained_model_name_or_path

    (str | PathLike) –

    The name or path to the pretrained model.

  • cache_dir

    (str | PathLike | None, default: None ) –

    The directory to cache the config in.

  • force_download

    (bool, default: False ) –

    Whether to force download the config from the Hub.

  • local_files_only

    (bool, default: False ) –

    Whether to use local files, not download from the Hub.

  • token

    (str | bool | None, default: None ) –

    The token to use for authentication with the Hub.

  • revision

    (str, default: 'main' ) –

    The revision of the config to load from the Hub.

  • kwargs

    Additional keyword arguments to pass to the config.

Returns:

Source code in speculators/config.py
@classmethod
def from_pretrained(
    cls,
    pretrained_model_name_or_path: str | os.PathLike,
    cache_dir: str | os.PathLike | None = None,
    force_download: bool = False,
    local_files_only: bool = False,
    token: str | bool | None = None,
    revision: str = "main",
    **kwargs,
) -> "SpeculatorModelConfig":
    """
    Load a SpeculatorModelConfig from the name/id of a model on the Hugging Face Hub
    or from a local directory. Will automatically instantiate the correct config
    from speculators.models package.

    :param pretrained_model_name_or_path: The name or path to the pretrained model.
    :param cache_dir: The directory to cache the config in.
    :param force_download: Whether to force download the config from the Hub.
    :param local_files_only: Whether to use local files, not download from the Hub.
    :param token: The token to use for authentication with the Hub.
    :param revision: The revision of the config to load from the Hub.
    :param kwargs: Additional keyword arguments to pass to the config.
    :return: A SpeculatorModelConfig object with the loaded parameters.
    """
    # Transformers config loading
    config_dict, kwargs = cls.get_config_dict(
        pretrained_model_name_or_path,
        cache_dir=cache_dir,
        force_download=force_download,
        local_files_only=local_files_only,
        token=token,
        revision=revision,
        **kwargs,
    )

    if "speculators_model_type" not in config_dict:
        # Conversion pathway
        raise NotImplementedError(
            "Loading a non-speculator model config is not supported yet."
        )

    return cls.from_dict(config_dict, **kwargs)

to_dict

to_dict() -> dict[str, Any]

Returns:

  • dict[str, Any]

    A dictionary representation of the full config, including the PretrainedConfig variables and Pydantic model fields.

Source code in speculators/config.py
def to_dict(self) -> dict[str, Any]:
    """
    :return: A dictionary representation of the full config, including the
        PretrainedConfig variables and Pydantic model fields.
    """
    pretrained_dict = super().to_dict()
    model_dict = self.model_dump()
    config_dict = {**pretrained_dict, **model_dict}

    # strip all class variables and metadata that are not needed in the output
    for key in (
        "model_config",
        "auto_package",
        "registry_auto_discovery",
        "schema_discriminator",
        "model_type",
        "base_config_key",
        "sub_configs",
        "is_composition",
        "attribute_map",
        "base_model_tp_plan",
        "base_model_pp_plan",
        "base_model_ep_plan",
        "has_no_defaults_at_init",
        "keys_to_ignore_at_inference",
        "_auto_class",
    ):
        config_dict.pop(key, None)

    return config_dict

to_diff_dict

to_diff_dict() -> dict[str, Any]

Returns:

  • dict[str, Any]

    A dictionary representation of a simplified config, including only the PretrainedConfig fields that have been modified or set, along with all Pydantic fields.

Source code in speculators/config.py
def to_diff_dict(self) -> dict[str, Any]:
    """
    :return: A dictionary representation of a simplified config,
        including only the PretrainedConfig fields that have been modified
        or set, along with all Pydantic fields.
    """
    return super().to_diff_dict()

SpeculatorsConfig

Bases: ReloadableBaseModel

The base config for a spec decode implementation which defines the parameters required to implement a speculators algorithm for the parent, speculator model. It includes details on the algorithm, token proposals, and the verifier model.

Methods:

check_default_proposal_method

check_default_proposal_method() -> SpeculatorsConfig

Validate default_proposal_method is one of the proposal_methods.

Source code in speculators/config.py
@model_validator(mode="after")
def check_default_proposal_method(self) -> "SpeculatorsConfig":
    """Validate default_proposal_method is one of the proposal_methods."""
    available = [method.proposal_type for method in self.proposal_methods]
    if self.default_proposal_method not in available:
        raise ValueError(
            "default_proposal_method "
            f"'{self.default_proposal_method}' must match the proposal_type of "
            f"one of the configured proposal_methods. Available proposal types: "
            f"{available}."
        )
    return self

VerifierConfig

Bases: BaseModel

The base config for a verifier model which defines the parameters that are required to either load the original verifier model or validate compatibility with a new verifier based on the the architecture and tokenizers/processor properties. It provides convenience methods to extract the required parameters from a PretrainedConfig object.

Methods:

  • from_config

    Create a VerifierConfig from a PretrainedConfig object.

from_config classmethod

from_config(
    config: PretrainedConfig,
    name_or_path: str | None = "UNSET",
) -> VerifierConfig

Create a VerifierConfig from a PretrainedConfig object. Used to extract the required parameters from the original verifier config and create a VerifierConfig object.

Parameters:

  • config

    (PretrainedConfig) –

    The PretrainedConfig object to extract the parameters from.

  • name_or_path

    (str | None, default: 'UNSET' ) –

    The name or path for the verifier model. Set to None to not add a specific name_or_path. If not provided, the name_or_path from the config will be used.

Returns:

  • VerifierConfig

    A VerifierConfig object with the extracted parameters.

Source code in speculators/config.py
@classmethod
def from_config(
    cls, config: PretrainedConfig, name_or_path: str | None = "UNSET"
) -> "VerifierConfig":
    """
    Create a VerifierConfig from a PretrainedConfig object.
    Used to extract the required parameters from the original verifier
    config and create a VerifierConfig object.

    :param config: The PretrainedConfig object to extract the parameters from.
    :param name_or_path: The name or path for the verifier model.
        Set to None to not add a specific name_or_path.
        If not provided, the name_or_path from the config will be used.
    :return: A VerifierConfig object with the extracted parameters.
    """
    config_dict = config.to_dict()

    if name_or_path == "UNSET":
        name_or_path = (
            getattr(config, "name_or_path", None)
            or config_dict.get("_name_or_path", None)
            or config_dict.get("name_or_path", None)
        )

    return cls(
        name_or_path=name_or_path,
        architectures=config_dict.get("architectures") or [],
    )

reload_schemas

reload_schemas()

Automatically populates the registry for all PydanticClassRegistryMixin subclasses and reloads schemas for all Config classes to ensure their schemas are up-to-date with the current registry state.

Source code in speculators/config.py
def reload_schemas():
    """
    Automatically populates the registry for all PydanticClassRegistryMixin subclasses
    and reloads schemas for all Config classes to ensure their schemas are up-to-date
    with the current registry state.
    """
    TokenProposalConfig.reload_schema()
    SpeculatorsConfig.reload_schema()
    SpeculatorModelConfig.reload_schema()