Attention Didn't Die, It Just Got Outnumbered

Every few months someone declares the Transformer dead.

I have started keeping a mental list of the headlines. Attention is obsolete. The post-Transformer era has arrived. State space models change everything. And then I go read the actual model cards for the systems making those headlines, and I find the same thing every single time. The attention layers are still there. Diluted, rationed, interleaved with cheaper machinery, but there. Nobody wants to say it plainly, so I will. In 2026 the frontier did not move past attention. It surrounded it.

This matters because the story we tell about architecture shapes where the money and the research effort go. If you believe attention is finished, you make one set of bets. If you understand that the winning designs are hybrids that keep a small amount of attention on purpose, you make much smarter ones. In twenty years of building systems, I have learned that the most expensive mistakes come from believing a narrative that the engineering has already quietly abandoned.

So let me walk through what actually happened, why three different ideas got fused into one architecture, and where I think the honest limits are. The efficiency wins are real. The replacement story is not.

Three Ideas That Solve Three Different Problems

The thing that finally made this click for me was realizing that the three headline techniques of the moment are not competitors. They are answers to three unrelated questions, which is exactly why they compose so well.

Start with state space models, the Mamba lineage. The Transformer has one economic flaw that has haunted it since 2017. Self-attention costs grow with the square of the sequence length, so doubling the context roughly quadruples the work. State space models replace that with a recurrence that scales linearly, offering O(N) computation where attention demands O(N squared). Mamba's original result is the one that lit the fuse, roughly five times higher inference throughput than a comparable Transformer and clean scaling to million-length sequences.

Then there is Mixture of Experts, or MoE, which answers a completely different question. Attention versus recurrence is about how you mix information across a sequence. MoE is about how many parameters you can afford to own. It decouples the total parameter count from the compute you spend on any single token, because only a handful of expert subnetworks activate per token. You can hold an enormous amount of knowledge in the full model while paying, per token, for only the slice you actually route to.

And attention itself is the third idea, the one that supposedly lost. Its job is precise recall. When you need to reach back into the context and retrieve an exact fact, a name, a number, a line of code, attention is still the best mechanism we have. This is the part the replacement narrative keeps skipping over.

Put those three together and the logic of the modern hybrid is obvious. Use state space layers to carry the sequence cheaply, sprinkle in enough attention to preserve recall, and wrap the whole thing in MoE to scale capacity without scaling the bill. Three levers, three problems, one architecture.

Primitive	Problem it solves	What it costs you
State Space Model (Mamba)	Sequence-length cost: O(N) instead of O(N²)	Weaker precise recall from a compressed state
Attention (Transformer)	Precise recall and in-context retrieval	Quadratic cost as context grows
Mixture of Experts (MoE)	Capacity: more parameters at fixed per-token compute	Fragile routing, load imbalance, training pathology

What the Production Models Actually Show

Theory is cheap. What convinced me this was a genuine shift and not a research curiosity was watching the recipe move into shipping systems.

The archetype is Jamba from AI21, the first model to combine all three primitives at scale by interleaving Transformer blocks, Mamba blocks, and MoE. The base Jamba runs 12 billion active parameters out of 52 billion total, supports a 256K context window, and fits up to 140K tokens on a single 80GB GPU. That sparsity ratio, roughly one active parameter for every four it owns, is the MoE lever doing its job. Jamba-1.5 later scaled the same idea into two sizes, a Large at 94 billion active parameters and a Mini at 12 billion.

By 2026 the recipe had reached the frontier through NVIDIA's Nemotron line, and the framing there is worth quoting because it is unusually honest about the division of labor. In Nemotron 3 Ultra, MoE improves accuracy per active parameter while the hybrid Mamba-Attention structure improves inference throughput by cutting attention cost and shrinking the key-value cache footprint. Two orthogonal wins, stated plainly, from two different parts of the design.

The numbers are genuinely impressive. Nemotron 3 Ultra reports up to roughly six times higher inference throughput than state-of-the-art comparable models at on-par accuracy, with a one-million-token context aimed squarely at long-running agents. Its distilled sibling, Nemotron 3.5 Lightning, is a 30-billion-total, 3-billion-active hybrid MoE that interleaves Mamba-2, MoE, and attention layers. If you want a single snapshot of where the frontier sits, that layer cake is it.

Notice what did not happen in any of these systems. None of them threw attention away. Jamba keeps it. Nemotron-H keeps it. Nemotron 3 Ultra keeps it. A well-studied research reference point, an 8-billion-parameter hybrid, lands at about 43 percent Mamba-2, 7 percent self-attention, and 50 percent MLP. Seven percent attention is not zero attention. It is a carefully rationed dose of the one thing the cheaper layers cannot do.

The Part the Headlines Skip

Here is where I part ways with the triumphalist version of this story.

The reason every frontier hybrid keeps attention is not nostalgia or inertia. It is that state space models still lose on the abilities that matter most for reasoning, and they lose in a way that is structural, not incidental. Compressing an entire history into a fixed-size state creates what the literature calls a selective recall bottleneck. The model simply cannot reliably reach back and retrieve a precise detail from deep in the context, because that detail has been squeezed into a representation that had no room for all of it.

This is not a temporary gap that the next paper will close. As late as September 2026, the research still reports state space models trailing Transformers on in-context learning and precise retrieval, the exact capabilities that make a model feel intelligent rather than merely fluent. And the hybrid win is conditional. Hybrids match or beat Transformers on some retrieval tasks while Transformers still lead on others, particularly tasks that hinge on the position of information in the context. Pure attention is not obsolete. It is specialized.

Then there is the MoE half of the story, which gets described as a free lunch and is nothing of the kind. Routing tokens to experts is fragile. Left alone, it collapses into a rich-get-richer dynamic where a few experts get all the traffic and the rest wither, a failure mode the field openly calls the rich-get-richer problem. The standard fix, an auxiliary load-balancing loss, has its own cost. It pushes routing toward uniformity, which smears out the very expert specialization that made MoE attractive in the first place. You are constantly trading between balanced load and meaningful specialization, and there is no setting of the dial that gives you both for free.

I will admit I find the "on-par accuracy" phrase quietly revealing every time I see it in a model card. If the new architecture truly beat attention on quality, that is the number they would lead with. Instead they lead with throughput and cost, which are real and valuable wins, and they match on quality. That is an efficiency story wearing the costume of a revolution.

None of this is a knock on the work. It is some of the most useful engineering in the field. I just think we should call it what it is.

What This Means If You Are Choosing a Model

If you build with these systems, the replacement narrative is not just wrong, it is actively misleading for your decisions. Here is how I would think about it.

Match the architecture to the context length you actually have

The state space advantage is not universal. It inverts with sequence length. Transformers are up to roughly 1.9 times faster at short sequences under about 8K tokens, while state space models flip to as much as 4 times faster past roughly 57K tokens, with about 64 percent less memory. So the honest question is not whether hybrids are faster. It is whether your workload actually lives in the long-context regime where they win. If your median request is a few thousand tokens, the hybrid may buy you very little.

Treat long-running agents as the real use case

The place these architectures shine is exactly where Nemotron aimed them, long-horizon agentic work with enormous context windows that would make a pure Transformer's key-value cache explode. If you are running agents that accumulate a million tokens of history, the throughput and memory story is transformative. If you are doing short question answering, you are paying for machinery you will not use.

Do not trust the "post-Transformer" label on the tin

When a vendor markets a model as post-Transformer, read the architecture card and count the attention layers. You will almost always find them. What you are really buying is a hybrid with a tuned ratio, and the ratio is the interesting engineering decision, not the marketing category. Ask how much attention, placed where, and why. That question will teach you more than any benchmark table.

None of this requires you to be a skeptic about the direction of travel. Hybrids are the frontier for good reasons. It just requires you to buy the real thing rather than the headline.

Where This Leaves Us

The most interesting fact about the post-Transformer era is that it kept the Transformer. We spent two years watching Mamba, then MoE, then hybrids get framed as successive replacements for attention, and the actual frontier systems answered that framing by quietly using all of them at once. Jamba, Nemotron-H, Nemotron 3 Ultra. Every one of them a truce, not a conquest.

I think that is a healthier way to see the last few years of architecture research. Not a series of coronations where each new idea deposes the last, but an accumulation of specialized parts that we are learning to assemble. State space layers to move through long sequences cheaply. Attention to remember precisely. Experts to hold more than any single dense network could afford. The art is in the ratio, and the ratio is still being worked out.

So the next time you see a headline announcing that attention is finally dead, do what I now do reflexively. Go find the model card. Count the attention layers. I would bet you a good dinner they are still in there, outnumbered and indispensable, doing the one job nothing cheaper has learned to do.

Key Takeaways