In the summer of 2017, Google's AI researchers released a paper titled "Attention Is All You Need", which described a new type of neural network called the Transformer.
It later turned out that the Transformer network performed extremely well in handling long sequence data, especially text.
Today, in 2026, the Transformer can be said to be the core engine of all mainstream large language models (LLMs) on the market. Justin Dangel, co-founder and CEO of AI startup Subquadratic, said: "The entire AI industry is built on top of Transformers. They are one of the most important innovations in the history of computer science and have changed the world."
However, the Transformer technology is also showing signs of fatigue. Many recent advancements in large language models, such as the development of reasoning models and the ability to process massive inputs at one time, are not a natural extension of the core technology, but rather seem like makeshift fixes for some of its fundamental flaws.
More and more scientists and engineers are pondering how technology will evolve, what form the next generation of AI will take, and what the future holds for Large Language Models (LLMs). We believe that LLMs themselves will not disappear, but it remains to be seen how they will be constructed in the future.
A batch of startups are vying to get in on the action, seeking to break new ground at the forefront of existing technology. While some competitors are bound to fail, these up-and-coming players have enormous potential and may even have a competitive edge over current industry leaders.
The Twilight of the Web
First, we need to clarify the existing technical difficulties. The core advantage of the Transformer comes from the so-called "dense attention" mechanism, which encodes the semantics of a piece of text through a series of numbers. During the processing, each word, or part of a word, i.e., token, in the text is compared with all other words or tokens through multiplication operations.
Dense attention mechanisms can capture the meaning of text with extremely high precision, but as the length of the text increases, the amount of computation required surges. A document of 10,000 English words may require the Transformer to perform 50 million multiplication operations, which is the main reason why large language models consume a huge amount of energy.
Computing power costs are extremely high. According to OpenAI President Greg Brockman, OpenAI's computing power expenditure will reach $50 billion this year. The International Energy Agency predicts that the total power consumption of data centers will double by 2030.
What's worse, when it comes to handling the tasks required to deal with numerous new models, Transformer networks often fall short. Limited by their word-by-word processing of text, Transformers are not good at maintaining a large amount of information at the same time, in other words, their context window cannot be expanded too large.
However, for LLMs to undertake more challenging tasks, they must receive larger-scale data, such as an entire document library, a complete codebase, or output from other large language models.
The reasoning model works by first using the thought process chain as a draft to take notes for itself, and then continuing to reason by reading the notes. This approach also increases the amount of data that the model needs to maintain.
As LLMs continue to grow in size and strength, the bottlenecks of Transformers are becoming increasingly prominent, with their former core advantages now turning into limitations.
The following text introduces four new ideas aimed at addressing the limitations of Transformer networks, which are expected to radically reshape large language models, making them faster and more efficient, and potentially even more intelligent.
Redesigning Attention Mechanism
To make large language models run faster and at a lower cost, it may be effective to directly address the pain points and change the attention mechanism, replacing dense attention with sparse attention.
The sparse attention mechanism only performs calculations on a portion of the word pairs in the text block, rather than all of them, which can greatly reduce the computational requirements of the LLM.
For years, researchers have proposed numerous sparse attention mechanisms, but they have always fallen short of dense attention mechanisms in terms of capturing semantics. Of course, times have changed.
The aforementioned startup Subquadratic claims that its developed sparse attention mechanism has, for the first time, demonstrated performance comparable to that of mainstream top-tier LLMs in multiple tasks such as search and code generation. This assertion is a bombshell, but some industry insiders remain skeptical.
In terms of Subquadratic, its in-house model SubQ can judge in real time which words are important or not for each paragraph of the input text. It is said that tens of thousands of users have signed up for the waiting list to try SubQ, and the day when the model is open to the general public will arrive soon.
Another startup, Manifest AI, has taken a different approach, choosing not to reform the working mechanism of attention, but instead replacing it with a completely new solution.
The company has developed a mechanism called "power retention", which only retains information most relevant to the current task, ensuring that the amount of data that large models need to maintain does not grow exponentially.
Traditional attention mechanisms require LLMs to remember all content within the context window. In contrast, sparse attention models, such as SubQ, discard a large number of individual words but still retain a general outline of the content they have seen. By comparison, the power retention mechanism provides the model with a rolling summary of the context window content, which is all the text the model can currently see, and discards old information with lower relevance when new information is added.
The fundamental principle of order preservation has existed for about ten years. Manifest AI claims that they have completed a technological iteration, building a product that can rival large models based on Transformer for the first time. Additionally, they can modify Transformer models into order preservation models with only a small amount of retraining.
To demonstrate the aforementioned transformation capabilities, the technology team has modified the existing open-source large model StarCoder to create PowerCoder, which is equipped with a power retention mechanism, and has also released the Brumby model, claiming its performance can rival certain versions of Alibaba's Turing Question Answering.
Manifest AI's vision is that when large language models need to handle massive data tasks, their power-preserving technology will be the go-to solution. The company's co-founder and Chief Technology Officer, Carles Gelada, has stated that their technology has a wide range of applications, from analyzing videos that are several hours long to building AI entities that can perform tasks continuously for several weeks.
Building Smaller, More Flexible Models
A startup incubated by the Massachusetts Institute of Technology, Liquid AI, has not completely overhauled or abandoned the Transformer, but instead combined its self-developed liquid neural networks with the Transformer to build what it calls a "liquid foundation model," abbreviated as LFM.
Liquid AI's models have a much smaller size than most LLMs and consume less power. The company has developed models for automakers such as Mercedes that can be adapted to small chips for in-vehicle use; its latest version can even run on a Raspberry Pi development board that costs $50 and has low power consumption.
Companies with annual revenues of less than $10 million can allegedly use Liquid AI's products for free, with model downloads having reached tens of millions and receiving widespread acclaim.
Liquid neural networks are an extension of convolutional neural networks, inspired by the brains of worms; convolutional neural networks are another type of neural network that existed before the emergence of Transformers. The key to liquid neural networks' prominence lies in their adaptive mechanisms, which enable the model to dynamically adjust its behavior based on new information, achieving learning while running. In contrast, models based on Transformers lack this ability, with their behavior patterns fixed and unchanging once training is complete.
Liquid AI's early models, although relatively basic, were already capable of controlling drones and driving vehicles. Now, with the help of its liquid foundation model, the company is attempting to scale up its technology to compete with mainstream large language models. The new model's performance can match that of competitors that are four times its size, including certain versions of Alibaba's Tongyi Qianwen and Google's open-source model Gemma.
A typical LLM is built by stacking multiple Transformers. Liquid AI's latest LFM features a hybrid architecture, consisting of 20% Transformers and 80% liquid neural networks.
The calculation of the aforementioned ratio is not done by humans, but by another AI system developed by the company, which is specifically used for auxiliary model design. It is said that this AI system, which is capable of designing AI, combines various different neural networks, such as liquid networks, convolutional networks, and Transformers, and screens combination schemes to find the best balance between performance and efficiency.
Ramin Hasani, co-founder and CEO of Liquid AI, believes the future of neural networks is full of endless possibilities, with the Transformer being just the beginning.
The human brain is a universal artificial intelligence system, yet its power consumption is only 20 watts, so how does it achieve this? We are fully capable of making even more innovations.
——Ramin Hassani
Doubao and Nubia Partner with ByteDance to Revolutionize AI-Powered Content Creation
Almost all large language models output content word by word, which is in line with human speaking and writing habits, but is extremely inefficient for computers. If LLMs can generate text in one go - spitting out entire sentences or even paragraphs at once - speed and cost will be optimized.
One of the practitioners of the above idea is Inception, which is using diffusion technology to build large models.
Diffusion technology is perhaps better known for its role as the driving core of the vast majority of image and video generation models. The training method for large diffusion models is as follows: first, it receives a random pixel grid, similar to the snowflake noise on old TVs, and then simultaneously processes all pixels, determining which ones need to be modified, and turning the snowflake noise into a high-definition image.
It turns out that the above process also applies to text. The Inception team, through specific training, enables the diffusion large model to receive a string of random word sequences and convert them into semantically coherent sentences. This diffusion LLM still uses Transformer to encode semantics, but outputs a whole block of text at once, completing more tasks at a lower cost.
The company's co-founder and CEO, Stefano Ermon, said that their innovation still belongs to large Transformer models, but it can predict multiple tokens simultaneously, which is why this model generates text much faster and has a higher cost-effectiveness compared to most solutions on the market.
The challenge lies in how to migrate the technology originally designed for image generation to the text domain.
When processing images, a blue pixel can be changed to a red pixel by gradually adjusting through intermediate colors. However, this does not apply to text, as there are no intermediate words between "cat" and "dog".
——Stefano Elmoni
Elon has another identity as a Stanford researcher. In 2024, he and two Stanford colleagues discovered a mathematical method to bridge the text gap and trained a diffusion model based on this method that matches the performance of OpenAI's 2019 GPT-2, but generates text 10 times faster.
Based on this achievement, Elrom founded Inception, and his ambition is now aimed at the industry leaders. Inception claims its latest model, Mercury 2, has performance comparable to some versions of GPT-4 released by OpenAI in 2023, and is 10 times faster.
We are very optimistic about this route, which has the potential for large-scale expansion. Ultimately, speed and cost are the core standards for measuring the value of large models - how much intelligence can you get for every dollar you put in.
Companies betting on diffusion technology are not limited to Inception. Google is also actively exploring the field and has built a prototype LLM called Diffusion Gemma.
Breaking Free from the Text World
Among startups exploring innovative solutions to address the limitations of Transformers, Pathway's concept is the most disruptive. They aim to free large language models from the constraints of language, and have developed the Dragon Hatchling large model with this goal in mind.
To date, its most impressive achievement comes from a benchmark test where it competed against more than 250,000 extremely difficult Sudoku puzzles. Mainstream large models from several top laboratories were unable to crack even a single puzzle, while Dragon Hatchling overcame more than 97% of the challenges.
Pathway's point is that while mainstream large language models excel in many tasks, they are still powerless in dealing with many key types of problems, with Sudoku being just one example.
The company's co-founder and CEO, Zuzanna Stamirowska, said that in order for large models to produce truly original solutions to real-world problems, they must move beyond the Transformer framework, as it forces large models to rely entirely on text for reasoning, but language is not the optimal vehicle for certain forms of reasoning.
Pathway's solution modifies the underlying mathematical logic of the Transformer, replacing the attention mechanism with a mathematical structure known as "state space".
The state space no longer encodes information word by word, but instead compresses it into more abstract expressions. Based on this, Dragon Hatchling can still read and write text, while also simulating reasoning patterns that are not dependent on word sequences. This not only improves model efficiency, but also theoretically enables tasks that other LLMs are incapable of.
Chess and math problems do not exist in the human brain in the form of long sentences. The brain's flashes of insight do not necessarily appear in linguistic form. We believe that if reasoning must be completed relying on language, this is a limitation in itself. Mainstream large language models have the ability to read Sudoku solution tutorials and then write solution code, but this is not what we are looking for. The model we idealize is not limited to book knowledge.
AI's vast universe is not about solving Sudoku, but about conquering cancer - however, there is no ready-made manual for curing cancer in this world.
Zuzanna Stamirowska
Source:
Several startups are currently focusing on the development of Large Language Models (LLMs), with the goal of achieving breakthroughs in this field. Companies such as ByteDance, Tencent, and Huawei are investing heavily in LLM research, while startups like Mobvoi and AISpeech are also actively exploring this area. In recent years, the application of LLMs has become increasingly widespread, with tech giants like Google, Microsoft, and Baidu all making significant advancements. However, there is still room for innovation and growth, particularly in terms of improving model accuracy, reducing computational costs, and expanding the scope of applications. Some startups are working on developing more specialized LLMs, such as those tailored for specific industries or languages. For example, the Chinese startup, Iflytek, has developed an LLM designed specifically for the Chinese language, which has shown promising results in terms of accuracy and efficiency. Meanwhile, other startups are exploring the potential of LLMs in emerging areas like multimodal interaction and human-computer dialogue. Companies like iQIYI and Kuaishou are using LLMs to enhance their video and live streaming services, enabling more intuitive and interactive user experiences. As the LLM landscape continues to evolve, it is likely that we will see