Startups betting against the Transformer as core of future LLMs
A wave of AI startups, including Subquadratic, is working to replace or overhaul the transformer architecture that has underpinned every major large language model since Google researchers introduced it in 2017.
👁 114 reads
A wave of AI startups, including Subquadratic, is actively working to replace or fundamentally redesign the transformer architecture that has powered every major large language model since Google researchers described it in the 2017 paper “Attention Is All You Need.”
- OpenAI alone is set to spend $50 billion on computing in 2025, illustrating the cost burden that transformer-based dense attention creates
- A 10,000-word document can require roughly 50 million multiplications under dense attention - a figure that scales rapidly as context windows grow
- The International Energy Agency predicts data-center electricity consumption will double by 2030, adding urgency to the search for more efficient architectures
The case for change centers on cost and scale. Transformers rely on a mechanism called dense attention, which compares every token in a piece of text with every other token through repeated multiplication. A document of 10,000 words can require roughly 50 million multiplications, and that number grows fast as inputs get longer. OpenAI is set to spend $50 billion on computing this year, according to company president Greg Brockman, and the International Energy Agency predicts total data-center electricity use will double by 2030.
The architecture also struggles with the workloads modern LLMs are asked to handle. Reasoning models write chains of thought to a scratch-pad and then re-read them, continuously expanding the amount of data a model must track. Agents must ingest output from other LLMs, entire code bases, or large document sets. Both use cases demand large context windows - exactly where transformers slow down and become expensive.
“The entire AI industry is built on transformers. They are one of the most important innovations in the history of computer science, and they’ve changed the world.”
MIT Technology Review, which published the analysis as part of its What’s Next series, labeled this coming generation of models “LLMs+” in its 2025 list of the ten things that matter in AI. Researchers and engineers are now exploring at least four distinct architectural directions as alternatives or supplements to dense attention.
Practitioner takeaway: For developers and businesses building on today’s LLM APIs, the transformer bottleneck is not a distant theoretical problem - it is already the reason large-context calls are expensive and slow. If even one of these architectural bets pays off, inference costs and context limits could shift materially. Keeping an eye on which startups attract serious funding or land model partnerships is a practical early signal that the economics of building with LLMs are about to change.
Google · Your sourcesQuantslant higher in your Google - one clickAdd to preferred sources →
More on this from QUANTSLANT
magazine·Situational Awareness puts $400M into chip startup Source Foundry