Quantization and Inference

Quantization trades off model precision for smaller memory footprint, allowing large models to be run on a wider range of devices. The Intel® Gaudi® Backend supports the following quantization backends:

• Intel® Neural Compressor
• Auto_Awq
• Gptqmodel