llmcompressor.modeling.kimi_k3.configuration_kimi_k3

Classes:

KimiK3Config –

Kimi-K3 model configuration.

KimiK3Config

KimiK3Config(
    text_config: dict | KimiLinearConfig = None,
    vision_config: dict | KimiK3VisionConfig = None,
    ignore_index: int = -100,
    media_placeholder_token_id: int = 163605,
    pad_token_id: int = 0,
    **kwargs,
)

Bases: PretrainedConfig

Kimi-K3 model configuration.

Args: text_config (dict | KimiLinearConfig): Configuration for the text model.

Vision Tower Parameters (from MoonViT3dConfig):
    patch_size (int): Patch size for vision tower.
    init_pos_emb_height (int): Initial position embedding height.
    init_pos_emb_width (int): Initial position embedding width.
    init_pos_emb_time (int): Initial position embedding time dimension.
    pos_emb_type (str): Type of position embedding.
    vt_num_attention_heads (int): Number of attention heads in vision tower.
    vt_num_hidden_layers (int): Number of hidden layers in vision tower.
    vt_hidden_size (int): Hidden size of vision tower.
    vt_intermediate_size (int): Intermediate size in vision tower FFN.
    merge_kernel_size (tuple): Kernel size for patch merging.
    merge_type (str): Type of merge operation.
    _attn_implementation (str): Attention implementation type.

MM Projector Parameters (from MultiModalProjectorConfig):
    mm_projector_type (str): Type of multimodal projector.
    mm_hidden_size (int): Hidden size from vision tower (should match vt_hidden_size).
    projector_hidden_act (str): Activation function for projector.
    projector_ln_eps (float): Layer norm epsilon for projector.

Other Parameters:
    ignore_index (int): The ignore index for the loss function.
    media_placeholder_token_id (int): The token ID to use for media placeholders.
    pad_token_id (int): The token ID to use for padding.

Source code in src/llmcompressor/modeling/kimi_k3/configuration_kimi_k3.py

def __init__(
    self,
    text_config: dict | KimiLinearConfig = None,
    vision_config: dict | KimiK3VisionConfig = None,
    # Other parameters
    ignore_index: int = -100,
    media_placeholder_token_id: int = 163605,
    pad_token_id: int = 0,
    **kwargs,
):
    if isinstance(text_config, dict):
        text_config = KimiLinearConfig(**text_config)
    if isinstance(vision_config, dict):
        vision_config = KimiK3VisionConfig(**vision_config)
    self.text_config = text_config
    self.vision_config = vision_config
    # Other config
    self.ignore_index = ignore_index
    self.media_placeholder_token_id = media_placeholder_token_id
    if getattr(self.text_config, "quantization_config", None) is not None:
        self.quantization_config = self.text_config.quantization_config

    super().__init__(pad_token_id=pad_token_id, **kwargs)