AI Glossary / Distillation

🏗️ Models and training

Knowledge distillation is a technique where a small model ("student") learns to mimic a large model's ("teacher") responses. Instead of training the small one from scratch on raw data, you feed it the giant's outputs - so it inherits much of its capability at a fraction of the size. It's the standard way to create fast and cheap models for mass applications. It also raises controversy when someone distills another's closed model through its API - most providers' terms explicitly forbid this.