Quantization

Quantization trades off model precision for smaller memory footprint, allowing large models to be run on a wider range of devices.

Contents

• Supported Hardware
• AutoAWQ
• BitsAndBytes
• GGUF
• INT8 W8A8
• FP8 W8A8
• FP8 E5M2 KV Cache
• FP8 E4M3 KV Cache