Skip to content
LLM Compressor Docs
Disk Offloading
Search
vllm-project/llm-compressor
LLM Compressor Docs
vllm-project/llm-compressor
Home
Why use LLM Compressor?
Compressing your model, step-by-step
Compressing your model, step-by-step
Choosing your model
Choosing the right compression scheme
Choosing the right compression algorithm
Choosing a dataset
Compressing your model
Deploying with vLLM
Getting started
Getting started
Installing LLM Compressor
Key Models
Key Models
DeepSeek V4
DeepSeek V4
NVFP4 + FP8 Example
Kimi-K2.6
Kimi-K2.6
NVFP4 Example
FP8 Block Example
Kimi-K3
Kimi-K3
NVFP4 Example
FP8 Block Example
Qwen3.5
Qwen3.5
NVFP4A16 VL Example
NVFP4 MoE Example
Qwen3.6
Qwen3.6
NVFP4 MoE Example
Gemma 4
Gemma 4
NVFP4 Example
FP8 Block Example
NVFP4 MoE Example
Llama 4
Llama 4
FP8 Example
Mistral Large 3
Mistral Large 3
FP8 Example
HY-V3
HY-V3
NVFP4 + FP8 Example
GLM-5.2
GLM-5.2
NVFP4 + FP8 Example
GLM-5.3-Flash
GLM-5.3-Flash
NVFP4 Example
User Guides
User Guides
Entrypoints
Entrypoints
oneshot
model-free-ptq
convert_checkpoint
Compression Schemes
Observers
Big Models and Distributed Support
Big Models and Distributed Support
Model Loading
Sequential Onloading
Distributed Oneshot
Saving a Compressed Model
Memory Requirements
Runtime Performance
Developer Guides
Developer Guides
Adding a New Modifier
Adding a New Observer
Adding MoE Calibration Support for a New Model
Examples
Examples
`AutoRound` Quantization
AWQ Quantization
Big Model Quantization with Sequential Onloading
Disk Offloading
Disk Offloading
On this page
Disk Offloading
iMatrix Importance-Weighted Quantization
Model-free Quantization
Multimodal Audio Model Quantization
Multimodal Vision-Language Quantization
Attention Quantization in LLM Compressor
Quantizing the Input Embedding
KV Cache Quantization
Non-uniform Quantization
`int4` Weight Quantization
`fp4` Quantization with NVFP4
`fp8` Weight and Activation Quantization
`int8` Weight and Activation Quantization
Quantizing Mixture of Experts (MoE) models
Mixture of Experts (MoE) Compression with REAP Expert Pruning
Applying Transforms to Improve Quantization Accuracy
Experimental
Experimental
Attention Quantization in LLM Compressor
Mistral-format model compression (experimental)
Developer
Developer
Governance
API Reference
API Reference
llmcompressor
llmcompressor
args
args
dataset_arguments
model_arguments
recipe_arguments
utils
core
core
events
events
event
lifecycle
model_layer
session
session_functions
state
datasets
datasets
utils
entrypoints
entrypoints
model_free
model_free
converter
helpers
lifecycle
microscale
reindex_fused_weights
save_utils
validate
oneshot
utils
logger
modeling
modeling
deepseekv32
deepseekv32
config
kernel
model
fuse
kimi_k3
kimi_k3
configuration_kimi_k3
encoding_k3
kimi_k3_processor
kimi_k3_vision_processing
media_utils
modeling_kimi_k3
modeling_kimi_linear
tokenization_kimi
offset_norm
modifiers
modifiers
autoround
autoround
base
utils
awq
experimental
factory
gptq
gptq
base
gptq_quantize
helpers
interface
logarithmic_equalization
logarithmic_equalization
base
modifier
obcq
obcq
sgpt_base
pruning
pruning
constant
constant
base
helpers
magnitude
magnitude
base
reap
reap
base
utils
sparsegpt
sparsegpt
base
sgpt_base
sgpt_sparsify
utils
utils
pytorch
pytorch
layer_mask
mask_factory
wanda
wanda
base
wanda_sparsify
quantization
quantization
calibration
gptq
group_size_validation
quantization
quantization
base
mixin
smoothquant
smoothquant
base
utils
transform
transform
awq
awq
base
dynamic_mappings
mappings
quip
quip
base
smoothquant
smoothquant
base
dynamic_mappings
utils
spinquant
spinquant
base
mappings
norm_mappings
utils
utils
hybrid_attention
utils
utils
constants
helpers
hooks
pytorch_helpers
observers
observers
base
fusion
helpers
imatrix
min_max
mse
mse_quant
pipelines
pipelines
basic
basic
pipeline
cache
data_free
data_free
pipeline
independent
independent
pipeline
registry
sequential
sequential
ast_helpers
ast_utils
ast_utils
auto_wrapper
control_flow_analyzer
name_analyzer
helpers
pipeline
transformers_helpers
pytorch
pytorch
model_load
model_load
helpers
utils
utils
helpers
sparsification
sparsification_info
sparsification_info
configs
helpers
module_sparsification_info
recipe
recipe
metadata
recipe
utils
sentinel
transformers
transformers
compression
compression
compressed_tensors_utils
sparsity_helpers
data
data
base
c4
cnn_dailymail
custom
data_helpers
evolcodealpaca
flickr_30k
gsm8k
open_platypus
peoples_speech
perfectblend
ultrachat_200k
wikitext
tracing
tracing
debug
utils
utils
helpers
utils
utils
dev
dist
helpers
metric_logging
pytorch
pytorch
module
utils
transformers
triton_utils
FAQ
FAQ
Frequently Asked Questions
On this page
Disk Offloading
Home
Examples
Disk Offloading
Disk Offloading
For more information on disk offloading, see
Model Loading
.
Back to top