On March 16, 2026, Kimi's team posted a paper called 'Attention Residuals' on arXiv, and things quickly spiraled. Elon Musk retweeted it. Andrej Karpathy commented, 'We haven't really taken the title Attention Is All You Need seriously.' Jerry Tworek, a former OpenAI co-founder, called it 'deep learning 2.0.' A Chinese team's architecture paper hasn't sparked this level of Silicon Valley discussion since DeepSeek-V3.
But for all the buzz, most of the discussion stayed at the level of 'Kimi made something new and the big names are excited.' What got overlooked: on the same day, ByteDance's Seed team and Huazhong University of Science and Technology jointly released another paper, 'Mixture-of-Depths Attention (MoDA),' addressing the exact same problem with a completely different approach. Within the same week, a third paper by Nanjing University's Dilxat Muhtar, MPI's Shiwei Liu, and others, 'When Does Sparsity Mitigate the Curse of Depth in LLMs,' provided the most precise diagnosis from the theoretical side.
Three papers appeared in rapid succession, all aimed at the same target. That is no coincidence. A structural problem ignored for nearly a decade has finally reached a tipping point where it must be solved.
The problem is not in the sequence dimension of attention. Attention has evolved through many generations over the past few years, from multi-head attention to grouped-query attention, to DeepSeek's MLA, to various sparse variants. Each generation has optimized how tokens attend to one another. That arms race has been impressive, but it has obscured a fact: the way information passes between layers has had the same answer since the Transformer paper was published in 2017. Residual connection: h = h + f(h), an addition operation with no learnable parameters.
The outputs of all previous layers are summed with equal weight. No selection. No forgetting. No learning. Each layer's contribution is piled into the residual stream indiscriminately, whether it learned a key feature or noise.
The residual connection is the most successful 'temporary fix' in deep learning history.
The most successful temporary fix.
Residual connections were introduced by Kaiming He in ResNet in 2015. The idea was extremely simple: when networks were stacked to about twenty layers, they stopped training; vanishing gradients meant deep parameters barely updated. So he added a 'highway' to each layer, letting the input skip the layer and connect directly to the output. Even if the layer learned nothing, information and gradients could still flow through the shortcut. The effect was immediate: ResNet pushed networks from about twenty layers to more than a hundred. Two years later, the Transformer arrived and adopted residual connections unchanged. Since then, no one has touched the design.
It's not that no one tried. ReZero, FixUp, and Highway Network all proposed variants with learnable residual weights. None became a mainstream architecture choice for large models, because residual connections work too well. They are simple, stable, and add almost no computational overhead. At the model scales of the time, the side effects hadn't surfaced.
44% of layers are idle.
What are the side effects? In early 2025, Shiwei Liu's team at Westlake University, Emory, and MPI published 'The Curse of Depth.' In March, Dilxat Muhtar of Nanjing University and colleagues published 'When Does Sparsity Mitigate the Curse of Depth in LLMs,' offering a quantitative diagnosis: in current mainstream LLM architectures, transformations in deep layers increasingly approach the identity mapping. Whatever goes in comes out unchanged—the layer is effectively nonexistent.
The numbers are ugly. Researchers used a 'usefulness score' to measure whether each layer performs a meaningful transformation. In a 12-layer model, all layers work. At 16 layers, three are useless. At 24, nine are useless. At 32, 14 are useless—44% of layers learn almost nothing. Increasing parameters from 900 million to 2.3 billion costs 156% more budget, yet effective layers only rise from 12 to 18.
Figure 2: Quantitative diagnosis of the curse of depth—effective layers increase with model size but with diminishing returns. AI-generated image.
The cause is directly related to how residual connections work. Each layer's output is added via a residual connection to a 'main trunk.' As layers increase, the accumulated signal on the trunk grows—think of background volume rising—while the magnitude of each layer's new signal is limited. In deep layers, the new signal is drowned out by background noise. Input and output are nearly identical, making the layer useless.
Residual connections solved the problem of letting gradients flow, but created the problem of making deep layers meaningful.
In the era of large models, this cost is real money. One layer is billions of floating-point operations. If a 128-layer model has 44% of layers idle, nearly 60 layers' compute is wasted. The community has spent years optimizing inference efficiency—quantization, distillation, pruning, sparse attention, KV cache compression—all optimizing 'useful computation.'
The biggest efficiency bottleneck is not attention's quadratic complexity, but an addition operation that has remained unchanged since 2015.
Instead of repairing the old road, they opened a new one.
ByteDance's Seed team and Huazhong University of Science and Technology didn't start from the premise that residual connections are broken and need replacing. Their question was more direct: since attention already lets tokens look at each other, why not let it also see information along the depth dimension?
Traditional attention has only one dimension—the sequence dimension. When a token at layer 20 computes attention, it can only see information from other tokens in the same layer. It cannot see its own state from layers 3 or 10, even if those shallow-layer features are highly useful for the current computation. Those shallow features do remain in the residual stream, but they have been repeatedly overlaid and gradually diluted by more than a dozen layers of residual updates. When deeper layers want to use shallow features, they can only use this 'diluted fruit juice' that has been watered down a dozen times.
MoDA adds a second dimension to attention—the depth dimension. Each attention head performs normal sequence attention (token looking at token) while also performing depth attention (directly fetching pristine KV pairs from all previous layers). The two streams of information are jointly normalized under the same Softmax, letting the model decide whether to focus more on the current layer's context or to look back at features learned in shallow layers. Residual connections remain and are not replaced, but deeper layers no longer have to rely solely on them to access shallow-layer information.
The idea is easy to understand; the hard part is implementing it without sacrificing speed.
Figure 3 | MoDA's dual-dimensional attention mechanism—sequence and depth dimensions are jointly normalized under the same Softmax.
Moving scattered archives to the workstation.
The problem lies in GPU memory access patterns. In normal attention computation, all KV pairs come from the same layer and are stored contiguously in memory, so GPU reads are highly efficient. MoDA, however, needs to fetch KV pairs from all previous layers, and these data are scattered across different memory locations. GPUs are particularly poor at such scattered random reads, causing a sharp drop in speed. If one naively concatenates KV from all historical layers, a 48-layer model would require each layer's attention computation to search through the archives of the previous 47 layers, making memory access almost entirely random and slowing training to an unusable level.
MoDA's solution is called Grouped Rearrangement. The core idea: since random access is slow, rearrange the data into contiguous blocks before computation.
The approach has two steps. First, the current layer's queries are divided into fixed-size groups (e.g., 64 tokens per group). Second, for each group, the deep KV it needs—KV pairs from all previous layers—are moved from scattered memory locations into a contiguous memory region, rearranged, and then attention is computed in one pass. Think of it this way: instead of having a worker run across the entire assembly line to dig through files, an assistant first brings all needed files to a desk next to the worker's station, neatly arranged, so the worker can browse them while seated. The cost of moving files is far less than the cost of repeated trips.
The key to this design is the granularity of the groups. If groups are too large, each group must move too much deep KV, making the data movement itself a bottleneck. If groups are too small, the GPU's parallel compute capacity is underutilized. MoDA uses the same block size as FlashAttention—the industry-standard high-speed attention engine—so the deep attention computation can directly reuse FlashAttention's underlying implementation, without writing a new set of GPU kernels.
At a 64K sequence length, MoDA's kernel efficiency reaches 97.3% of FlashAttention-2. With the entire deep attention mechanism added, speed is only less than 3% slower.
Figure 4 | Grouped rearrangement strategy—moving historical-layer KV scattered across memory into a contiguous memory region.
This number means that deep attention is not a lightweight add-on; it requires every layer to read the KV caches of all preceding layers. If engineering is sloppy, this cross-layer data dependency could slow training by several times. MoDA keeps the extra overhead to a 3.7% FLOPs increase, showing that the grouped rearrangement strategy cleanly solves the random-access problem.
A 3.7% cost for a 2.11% gain.
On a 1.5B-parameter model trained with the OLMo2 recipe, MoDA lifts average performance across 10 downstream tasks by 2.11 percentage points at only 3.7% additional compute. That looks modest at first, but it is an architecture-level gain, not one produced by more data or longer training, and it should persist as scale increases. Task-level variation is large: commonsense reasoning (WinoGrande) is up 2.37 percentage points, scientific reasoning (ARC-Challenge) up 4.35 percentage points, and tasks that require cross-layer feature integration benefit far more.
Figure 5 | MoDA performance comparison across 10 downstream tasks.
The debt incurred by Pre-Norm.
The most valuable part of the MoDA paper may not be MoDA itself, but rather an experiment on normalization strategies.
Some background is necessary. After each Transformer layer finishes its computation, a normalization step keeps numerical ranges stable and prevents values from exploding or vanishing during training. Where that step is placed follows two dominant conventions: before the layer's computation, called Pre-Norm (Pre-LN), or after it, called Post-Norm (Post-LN). Since 2020, nearly all large models use Pre-Norm because it makes training more stable and less likely to collapse. But the 'deep-layer idling' problem is precisely a side effect of Pre-Norm. To stabilize training, Pre-Norm steadily dilutes signal strength in deep layers.
MoDA's experiments ran two comparisons. On a 48-layer model, they used Pre-Norm and Post-Norm, respectively, and added MoDA's deep attention to each. In the Post-Norm configuration, adding deep KV lowered validation loss by 0.0409; in the Pre-Norm configuration, it lowered it by only 0.0041 — a nearly tenfold difference.
This result says something larger than MoDA itself: Pre-Norm is not merely stabilizing training; it is also systematically suppressing deep layers' learning capacity. Post-Norm was long avoided because training was unstable and gradients tended to blow up. MoDA's deep attention creates a new gradient pathway, so gradients no longer depend entirely on residual connections. With that pathway, Post-Norm's original instability is no longer a fatal flaw.
The MoDA + Post-Norm combination opens the possibility that the compromise once made for training stability — using Pre-Norm — can be reversed.
Chart 6: Pre-Norm vs. Post-Norm validation loss after adding deep KV.
Don't build a new road; renovate the old one.
MoDA left the residual connection untouched, choosing to open a separate path outside it. On the same day, Kimi's team released Attention Residuals (AttnRes), which took a more direct route by modifying the residual connection itself.
A standard residual connection simply sums the outputs of all previous layers with equal weight and feeds them into the main trunk. No selection, no forgetting. AttnRes replaces this fixed equal-weight addition with an attention operation: each layer uses its own state as the query, takes all previous layer outputs as candidates, and uses attention to decide which earlier features are useful for the current layer and with what weights.
The residual connection changes from a fixed formula into learnable dynamic routing.
Chart 7: AttnRes's core idea — replacing equal-weight residual addition with attention.
The cost is that each layer must run an additional deep attention computation, which is not cheap. Kimi's team controls the cost with a block strategy (Block AttnRes), dividing layers into blocks, performing full deep attention within blocks, and between blocks only attending to block-level aggregated representations.
AttnRes has been integrated into Kimi Linear (48 billion total parameters / 3 billion activated parameters), pretrained on 1.4 trillion tokens, and its effectiveness is confirmed to be consistent across model scales. This paper has been widely covered, so we won't expand on technical details. It's worth mentioning here because of its route comparison with MoDA.
Figure 8 | AttnRes training curves and ablation experiments.
Both approaches diagnose the same root cause: shallow-layer information reaching deep layers is repeatedly diluted by residual updates. But they differ in where to intervene. MoDA leaves residual connections untouched, adding a depth dimension to attention so deep layers can bypass the residual stream and directly access raw shallow features. AttnRes directly modifies the residual connection, replacing equal-weight addition with attention weighting. One builds a new path; the other renovates the existing one.
The two papers appeared on the same day, taking different routes but aiming at the same target. That is no coincidence. The depth problem in attention is already a consensus in the research community; the only difference is the angle of attack.
Figure 9 | AttnRes performance consistency across model scales.
The Scaffolding That Was Never Removed.
Back to the original question: why did the problem of deep-layer idling only get serious attention in 2026?
Because residual connections worked too well. They solved the most pressing problem of the time—vanishing gradients—at a manageable cost (deep-layer degradation was not obvious in small models), and alternatives were immature (neither ReZero nor Highway Networks had been validated at scale). No one had an incentive to change them. They were not a deliberate design choice but a forgotten stopgap. The scaffolding erected during construction was never dismantled after the building was finished; over time, everyone mistook it for a load-bearing wall.
Figure 1 | Signal dilution effect of residual connections—the deeper the layers, the harder it is for new signals to be heard. This image was AI-generated.
But what truly made this problem hard to spot wasn't the residual connection itself—it was that attention mechanisms have long operated in only one dimension. Over the past eight years, every evolution of attention—multi-head, grouped-query, sparse, linear—has worked along the sequence dimension. How tokens look at each other has been optimized countless times. But how layers look at each other? No one has asked that question. The depth dimension is attention's blind spot.
MoDA and AttnRes have opened this blind spot from different directions. MoDA adds a second dimension to attention, allowing it to operate along both sequence and depth. AttnRes turns inter-layer information transfer itself into an attention operation. The routes differ, but they point to the same conclusion: attention shouldn't only look horizontally—it should also look vertically.
The implications of this conclusion extend far beyond the two papers. Transformers still contain many fixed mechanisms that operate in only a single dimension. Each layer must execute in order and cannot be skipped. Each attention head computes independently and is simply concatenated, with no dynamic coordination between heads. Every token follows the exact same computational path, regardless of difficulty. These designs were originally engineering compromises to make models trainable and convergent.
The evolution of deep learning over the past decade, abstracted to its highest level, is one thing: handing more and more structural decisions from human designers back to the model itself. Hand-crafted convolution kernels were replaced by learnable attention. Fixed positional encodings were replaced by learnable rotary encodings. Fixed expert assignment was replaced by learnable routing. Now, the way information flows along the depth dimension is also beginning to be decided by attention itself.
Karpathy says we haven't taken 'Attention Is All You Need' literally. He may be right. But not in the sense that attention is sufficient—rather, that attention hasn't been used enough. It has evolved for many generations along the sequence dimension, but along the depth dimension, it's only just beginning.