Skip to content
Navigation
LLM Compressor Docs
model_load
Search
Search
vllm-project/llm-compressor
LLM Compressor Docs
vllm-project/llm-compressor
Home
Why use LLM Compressor?
Compressing your model, step-by-step
Compressing your model, step-by-step
Choosing your model
Choosing the right compression scheme
Choosing the right compression algorithm
Choosing a dataset
Compressing your model
Deploying with vLLM
Getting started
Getting started
Installing LLM Compressor
Key Models
Key Models
DeepSeek V4 Flash
DeepSeek V4 Flash
NVFP4 + FP8 Example
Kimi-K3
Kimi-K3
NVFP4 Example
FP8 Block Example
Qwen3.8-27B
Qwen3.8-27B
4-bit Example
Qwen3.8-2.4T-A95B
Qwen3.8-2.4T-A95B
NVFP4 + FP8 Example
REAP + NVFP4 Example
Qwen3.8-Flash-Next
Qwen3.8-Flash-Next
NVFP4 Example
Gemma 4
Gemma 4
NVFP4 Example
FP8 Block Example
NVFP4 MoE Example
Llama 4
Llama 4
FP8 Example
Mistral Large 3
Mistral Large 3
FP8 Example
HY-V3
HY-V3
NVFP4 + FP8 Example
GLM-5.3
GLM-5.3
MXFP4 Example
NVFP4 Layerwise Example
GLM-5.3-Flash
GLM-5.3-Flash
NVFP4 Example
User Guides
User Guides
Entrypoints
Entrypoints
oneshot
model-free-ptq
convert_checkpoint
Compression Schemes
Observers
Big Models and Distributed Support
Big Models and Distributed Support
Model Loading
Sequential Onloading
Distributed Oneshot
Saving a Compressed Model
Memory Requirements
Runtime Performance
Developer Guides
Developer Guides
Adding a New Modifier
Adding a New Observer
Adding MoE Calibration Support for a New Model
Open Research Hub
Open Research Hub
NVFP4 Sweeps
NVFP4 Sweeps
NVFP4 sweep on Gemma-4 26B-A4B and 31B for agentic reasoning
NVFP4 sweep on Llama-3.1-8B-Instruct, including long-context MRCR
NVFP4 W4A4 sweep on Qwen3-30B-A3B: 20 variants across six reasoning tasks
FP8 KV cache on Phi-3's fused QKV projection, calibrated and served
Native MXFP4 + FP8-block after DeepSeek-V4-Flash-0731 fine-tuning: RTN versus SpinQuant R1 + ceil
REAP expert pruning across MoE sizes and architectures
Arabic, English or mixed calibration for W4A16 GPTQ, NVFP4 and FP8 on Qwen3-8B
Examples
Examples
`AutoRound` Quantization
AWQ Quantization
Big Model Quantization with Sequential Onloading
Disk Offloading
iMatrix Importance-Weighted Quantization
Model-free Quantization
Multimodal Audio Model Quantization
Multimodal Vision-Language Quantization
Block-wise quantization-aware distillation
Attention Quantization in LLM Compressor
Quantizing the Input Embedding
KV Cache Quantization
Non-uniform Quantization
`int4` Weight Quantization
`fp4` Quantization with NVFP4
`fp8` Weight and Activation Quantization
`int8` Weight and Activation Quantization
Quantizing Mixture of Experts (MoE) models
Mixture of Experts (MoE) Compression with REAP Expert Pruning
Applying Transforms to Improve Quantization Accuracy
Experimental
Experimental
Attention Quantization in LLM Compressor
Mistral-format model compression (experimental)
Developer
Developer
Governance
API Reference
API Reference
llmcompressor
llmcompressor
args
args
dataset_arguments
model_arguments
recipe_arguments
utils
core
core
events
events
event
lifecycle
session
session_functions
state
datasets
datasets
utils
entrypoints
entrypoints
converter
converter
expert_pruner
higgs
higgs
base
ilp_solver
utils
model_free
model_free
converter
helpers
lifecycle
microscale
reindex_fused_weights
spinquant
validate
oneshot
utils
logger
modeling
modeling
deepseekv32
deepseekv32
config
kernel
model
fuse
kimi_k3
kimi_k3
configuration_kimi_k3
encoding_k3
kimi_k3_processor
kimi_k3_vision_processing
media_utils
modeling_kimi_k3
modeling_kimi_linear
tokenization_kimi
offset_norm
modifiers
modifiers
autoround
autoround
base
utils
awq
experimental
factory
gptq
gptq
base
gptq_quantize
helpers
interface
logarithmic_equalization
logarithmic_equalization
base
modifier
pruning
pruning
mone
mone
base
utils
reap
reap
base
qad
qad
base
quantization
quantization
calibration
gptq
group_size_validation
quantization
quantization
base
mixin
smoothquant
smoothquant
base
utils
transform
transform
awq
awq
base
dynamic_mappings
mappings
quip
quip
base
smoothquant
smoothquant
base
dynamic_mappings
utils
spinquant
spinquant
base
mappings
norm_mappings
utils
utils
hybrid_attention
utils
utils
constants
helpers
hooks
pytorch_helpers
observers
observers
base
fused_mappings
fusion
grid_search
helpers
imatrix
min_max
mse
pipelines
pipelines
basic
basic
pipeline
cache
data_free
data_free
pipeline
independent
independent
pipeline
registry
sequential
sequential
ast_helpers
ast_utils
ast_utils
auto_wrapper
control_flow_analyzer
name_analyzer
error_logging
grouping
helpers
pipeline
transformers_helpers
pytorch
pytorch
model_load
model_load
helpers
utils
utils
helpers
recipe
recipe
metadata
recipe
utils
sentinel
transformers
transformers
compression
compression
compressed_tensors_utils
mtp
mtp_compat
mtp_compat
glm_moe_dsa_for_causal_lm
qwen3_5_for_conditional_generation
data
data
base
c4
cnn_dailymail
custom
data_helpers
evolcodealpaca
flickr_30k
gsm8k
open_platypus
peoples_speech
perfectblend
razorcal
ultrachat_200k
wikitext
tracing
tracing
debug
utils
utils
helpers
utils
utils
dev
dist
helpers
metric_logging
pytorch
pytorch
module
utils
transformers
triton_utils
FAQ
FAQ
Frequently Asked Questions
Home
API Reference
llmcompressor
pytorch
model_load
llmcompressor.pytorch.model_load
Modules:
helpers
–
Back to top