llmcompressor

LLM Compressor is a library for compressing large language models utilizing the latest techniques and research in the field for both training aware and post-training techniques.

The library is designed to be flexible and easy to use on top of PyTorch and HuggingFace Transformers, allowing for quick experimentation.

Modules:

• args –

  Arguments package for LLM Compressor.

• core –

  Provides the core compression framework for LLM Compressor.

• datasets –

  Provides dataset utilities for model calibration and processing.

• entrypoints –

  Provides entry points for model compression workflows.

• logger –

  Provides a flexible logging configuration for LLM Compressor.

• modeling –

  Model preparation and fusion utilities for compression workflows.

• modifiers –

  Compression modifiers for applying various optimization techniques.

• observers –

  Framework for monitoring and analyzing model behavior during compression.

• pipelines –

  Compression pipelines for orchestrating different compression strategies.

• pytorch –

  PyTorch-specific utilities and tools for model compression workflows.

• recipe –

  Recipe system for defining and managing compression workflows.

• sentinel –

  Sentinel value implementation for LLM compression workflows.

• transformers –

  Tools for integrating LLM Compressor with transformers training flows.

• typing –

  Defines type aliases for the llm-compressor library.

• utils –

  General utility functions used throughout LLM Compressor.

Functions:

• configure_logger –

  Configure the logger for LLM Compressor.

configure_logger(
    logger_config: LoggerConfig = LOGGER_CONFIG,
)

 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129

def configure_logger(logger_config: LoggerConfig = LOGGER_CONFIG):
    """
    Configure the logger for LLM Compressor.

    This function sets up the console and file logging
    as per the specified or default parameters.

    **Note**: Environment variables take precedence over function parameters.

    :param config: The configuration for the logger to use.
    :type config: LoggerConfig
    """

    # env vars get priority
    if (disabled := os.getenv("LLM_COMPRESSOR_LOG_DISABLED")) is not None:
        logger_config.disabled = disabled.lower() == "true"
    if (console_log_level := os.getenv("LLM_COMPRESSOR_LOG_LEVEL")) is not None:
        logger_config.console_log_level = console_log_level.upper()
    if (log_file := os.getenv("LLM_COMPRESSOR_LOG_FILE")) is not None:
        logger_config.log_file = log_file
    if (log_file_level := os.getenv("LLM_COMPRESSOR_LOG_FILE_LEVEL")) is not None:
        logger_config.log_file_level = log_file_level.upper()

    if logger_config.disabled:
        logger.disable("llmcompressor")
        return

    logger.enable("llmcompressor")

    # reset logger configuration
    logger.remove()

    # initialize metric logger on loguru
    logger_levels = logger._core.levels.keys()
    if not logger_config.metrics_disabled and "METRIC" not in logger_levels:
        logger.level("METRIC", no=38, color="<yellow>", icon="📈")

    # set format (optionally adding rank)
    format = "{time:YYYY-MM-DDTHH:mm:ss.SSSS} | {function} | {level} - {message}"
    if logger_config.rank is not None:
        logger.configure(extra={"rank": dist.get_rank()})
        format = "[Rank {extra[rank]}] " + format

    if logger_config.console_log_level:
        logger.add(
            sys.stdout,
            level=logger_config.console_log_level.upper(),
            format=format,
            filter=support_log_once,
        )

    if logger_config.log_file or logger_config.log_file_level:
        log_file = logger_config.log_file or "llmcompressor.log"
        log_file_level = logger_config.log_file_level or "INFO"
        # log as json to the file for easier parsing
        logger.add(
            log_file,
            level=log_file_level.upper(),
            serialize=True,
            filter=support_log_once,
        )

    # set global value for later calls
    global LOGGER_CONFIG
    LOGGER_CONFIG = logger_config