AI Glossary / Transformer
🧠 Basics
Transformer
The architecture underlying modern language models. The "T" in GPT.
Transformer is a neural network architecture described by Google researchers in the 2017 paper "Attention Is All You Need." Its key innovation is the attention mechanism, which allows the model to weigh how much individual words in text relate to each other-regardless of the distance between them. The transformer enabled the quality leap we've observed in recent years: it scales excellently with data volume and computational power. The "T" in GPT comes from transformer; virtually all leading language models are based on this architecture.
Related concepts