Transformers hitting a wall - four startups building what comes next
Nine years after Google researchers introduced the transformer architecture, startups are developing four distinct alternatives to overcome its speed and memory limitations in large language models.
👁 119 reads
Nine years after Google researchers introduced the transformer, the neural network architecture powering every major large language model is running into hard limits - and a new wave of startups is building alternatives designed to be faster, more efficient, and potentially smarter.
- Transformers are increasingly expensive to run as context grows, due to their dense attention mechanism
- MIT Technology Review has identified four emerging architectural ideas that could replace or supplement transformers in LLMs
- A separate story from the same edition covers academic AI researchers navigating a shifting landscape, profiled at a Schmidt Sciences AI2050 convening in Mountain View
That is the central finding of a new piece by Will Douglas Heaven, part of MIT Technology Review’s What’s Next series, published on 11 August. The report identifies four distinct architectural innovations targeting what Heaven calls “the transformer problem.”
The core issue is structural. Transformers rely on a dense attention mechanism - a method of weighing every word against every other word in a piece of text. That computation becomes increasingly expensive as context length grows, scaling poorly with the volume of text a model must process. Transformers also struggle to reliably track large amounts of information simultaneously, a limitation that has become more visible as models are pushed toward longer documents and more complex reasoning tasks.
Google · Your sourcesQuantslant higher in your Google - one clickAdd to preferred sources →Four ideas for what comes after the transformer
The What’s Next piece outlines four proposed solutions, though the source material does not name the specific startups or architectures involved in detail. What the reporting makes clear is that interest in post-transformer design has moved well beyond academic curiosity - startups are now actively commercialising alternatives with the explicit goal of making LLMs faster and far more efficient at scale.
The same edition of The Download also carries a piece by Grace Huckins on the state of academic AI research. Huckins reported from a convening of the Schmidt Sciences AI2050 program - an initiative funded by Eric and Wendy Schmidt that supports academics working in AI - held at a hotel in Mountain View, California. The fellows roster is described as a concentration of accomplished and promising researchers, many of whom Huckins had previously interviewed or whose work she knew well.
University researchers make up most of the AI2050 group, and Huckins describes the current moment as a “weird time” for them, citing shifting pressures between academia and the private sector without specifying the outcomes. The full analysis is available through MIT Technology Review’s The Algorithm newsletter.
Elsewhere in the same edition, Nvidia was reported to have secured 500 billion dollars from Wall Street for AI infrastructure through deals with BlackRock, Goldman Sachs, and four other financial institutions - a signal of how deeply capital markets are now tied to the buildout of AI compute. The Chinese humanoid robotics firm Unitree also raised 900 million dollars ahead of its Shanghai IPO, with its listing pricing the company at 9 billion dollars.
More on this from QUANTSLANT
news·IBM Research Method Cuts Agent Memory Tokens Without Dropping Lessons