AI Glossary / Overfitting

🏗️ Models and training

Overfitting is a situation where the model, instead of learning general patterns, "memorizes" training data. On material it has seen, it performs great; on new cases - poorly. It's a classic problem across all machine learning. In large language models, overfitting manifests as reproducing memorized training fragments word for word. It's prevented through proper data diversity, regularization techniques and testing the model on material it hasn't seen during training.