AI Glossary / Quantization

🏗️ Models and training

Quantization is a model slimming technique: its weights are stored with lower numerical precision (e.g., 4-bit instead of 16-bit), making the file take up several times less memory and speeding up computation. The cost is a small, often unnoticeable loss of quality. Thanks to quantization, large open models can run on a home computer with a graphics card, or even on a laptop. Popular quantized model formats circulate in the self-hosting community and are the foundation of running AI locally.