The Architecture Atlas

There is no single Transformer anymore

Ask someone what a modern language model looks like and the answer will usually be some variation of “a Transformer.” That answer is still useful, but it is no longer very precise. Since the original Transformer, researchers have spent years changing almost every part of the design: how tokens interact, how information is remembered, how much computation each token receives, where parameters live, and what gets recomputed at inference time.

Some of those changes are replacements. Many are not. Grouped-query attention does not replace the Transformer; it changes how its key and value heads are organized. Mixture-of-Experts does not require abandoning attention; it changes how parameters are activated. Sliding-window attention changes the connectivity pattern. Gated DeltaNet changes the memory mechanism itself. N-gram systems can even sit partly outside the neural network and still change the effective system we deploy.

The next architecture is often not a replacement for the previous one. It is the part that the previous one was bad at.

This makes the field easier to understand. Instead of memorizing dozens of names, we can follow the engineering pressures that produced them.

The Transformer: arbitrary retrieval as a superpower

The 2017 Transformer replaced recurrence and convolution in the core sequence model with attention. Its defining trick is simple to describe: for every token, construct queries, keys, and values, then use similarity between queries and keys to decide which values matter. This gives the model direct, content-dependent access to earlier positions. Attention Is All You Need ↗

This is the property that keeps attention so hard to beat: the model does not have to compress the entire past into one fixed state before consulting it. The past remains explicitly addressable. That flexibility is powerful, but it comes with a price. During training, full attention scales quadratically with sequence length, and during autoregressive inference the key-value cache grows with the number of stored positions.

First pressure: make attention cheaper

The first response was not to throw attention away. It was to make attention less wasteful. Grouped-Query Attention (GQA) is the natural generalization of the standard design: several query heads share one key-value head per group instead of each having its own. The mechanism stays familiar, but the inference footprint becomes substantially smaller while quality stays close to conventional multi-head attention. GQA — Ainslie et al., 2023 ↗ Multi-Query Attention (MQA) is the extreme case of the same idea: every query head shares a single key-value head, which sharply reduces inference memory and bandwidth. One Write-Head is All You Need — Shazeer, 2019 ↗

This is an important pattern because it appears again and again: an architectural innovation does not need to change the conceptual role of a component. Sometimes it simply changes its cost.

There is an even more aggressive answer in the same direction. Multi-head Latent Attention (MLA), introduced with DeepSeek-V2, compresses the keys and values into a single low-rank latent vector that is cached at inference instead of the full per-head K/V tensors. Attention still happens — it is decompressed on the fly — but the cache shrinks well beyond what head grouping achieves. DeepSeek-V2 ↗

Second pressure: make the connectivity local

If every token does not need to inspect every other token, why pay for a global interaction pattern? Sliding Window Attention (SWA) limits each token's direct attention to a local neighborhood. Longformer is an early well-known example of sparse attention patterns that combine local windows with selected global positions. The trade is immediate: less computation and a smaller effective attention footprint, but less direct access to arbitrary distant tokens. Longformer ↗

The interesting insight is that “local” does not necessarily mean “short-context.” A model can propagate information across many layers. The architecture is changing how information travels, not simply deleting the past. Gemma 3 takes the local-global split literally: five sliding-window layers (window 1024, short-rotation positional encoding) followed by one global layer (long-rotation encoding) — a pure alternating hybrid with no recurrent state at all. Gemma 3 ↗

Learned sparsity: NSA and DSA

A sliding window is a hand-drawn connectivity pattern: the architect decides, in advance, that every token will see the same fixed neighborhood. Native Sparse Attention (NSA, DeepSeek) makes the pattern itself learnable. Three branches run in parallel over the key-value cache — a compression branch that pools distant KV into coarse blocks, a selection branch that keeps only the top-scoring blocks, and a local window branch — and a learned gate mixes their outputs per head. Trained natively from scratch rather than bolted on afterward, it won a best paper award at ACL 2025. NSA — best paper, ACL 2025 ↗

The idea shipped fast. DeepSeek-V3.2 (September 2025) replaced its attention with DSA, a descendant tuned for serving — one of the levers behind API prices dropping by up to three quarters — and GLM-5 (February 2026) combines MLA with DSA in a 744B-parameter model. Fixed windows tell every token the same story; learned sparsity lets the model decide, per head and per input, which connections are worth paying for. DeepSeek-V3.2 ↗ GLM-5 ↗

Third pressure: increase capacity without proportional compute

Mixture-of-Experts attacks a different bottleneck. Instead of forcing every token through the same feed-forward parameters, an MoE layer contains multiple experts and a router selects which ones participate for each token. A model can therefore contain a very large number of parameters while activating only a subset for any given token. Switch Transformer is one of the canonical demonstrations of this sparse-activation idea at scale. Switch Transformers ↗

The same logic can be pushed one level higher. Mixture-of-Depths routes individual tokens past entire layers, turning depth itself into a per-token compute budget rather than a fixed property of the architecture. We return to it in the closing section.

Once you see MoE as a parameter-allocation mechanism rather than a complete model architecture, many “hybrid” designs stop looking hybrid.

Fourth pressure: escape quadratic attention

The more ambitious family of approaches changes the memory mechanism itself. Linear attention, recurrent architectures, and state-space models try to represent the past in a compact state rather than retaining a full pairwise attention matrix. Mamba is a prominent example: a selective state-space model whose parameters depend on the input, allowing the state to selectively propagate or forget information while maintaining linear scaling with sequence length. Mamba ↗

This changes the fundamental question. Full attention asks the model to keep a rich, addressable record of the sequence. A recurrent or linear mechanism asks it to maintain a compact state that is updated as the sequence advances. RWKV showed earlier that attention-like time-mixing can itself be reformulated as a cheap recurrence — a useful bridge in the lineage toward delta-rule models. RWKV ↗

Gated DeltaNet: memory that can be rewritten

Delta-rule approaches take the recurrent idea further. Rather than treating the state as something that merely evolves according to a fixed transition, the delta rule provides a targeted update mechanism: change the stored state in response to the current key, value, and prediction of what the state should contain. Gated Delta Networks combine two complementary ideas—gating for rapid forgetting and the delta rule for targeted state modification. Gated Delta Networks ↗

This distinction is subtle but important. A useful recurrent memory cannot merely “remember more.” It needs a way to decide what survives, what is overwritten, and what deserves a precise update. That is why gating and delta-style writes fit together so naturally.

Kimi Delta Attention: pushing the hybrid further

Kimi Linear takes this family of ideas into a larger hybrid architecture. Its Kimi Delta Attention (KDA) extends Gated DeltaNet with finer-grained gating, while the full model combines KDA with Multi-Head Latent Attention (MLA) in a 3:1 ratio rather than attempting to eliminate attention altogether. The reported design targets both memory efficiency and long-context decoding, illustrating a broader direction: a model can assign different layers to different kinds of memory rather than insisting on one universal mechanism. Kimi Linear ↗

Qwen3-Next: the other hybrid lineage

Kimi Linear is not the only bet on gated delta memory at scale — and not even the only one from this family. Qwen3-Next (September 2025) built its hybrid from the other side of it: Gated DeltaNet layers handle the cheap, continuous sequence processing, and full Gated Attention layers are interleaved in a 3:1 ratio — twelve blocks of three delta layers plus one attention layer, each followed by a routed feed-forward. Qwen3-Next — Qwen Team, 2025 ↗

Around that trunk, nearly every mechanism in this article shows up at once: an ultra-sparse MoE — 512 routed experts plus one shared, ten activated per token — keeps active compute near 3B parameters out of 80B, and a multi-token prediction module is trained natively: the same MTP trick described in section 12, now part of a shipped architecture. In February 2026, Qwen3.5 promoted this exact layout to flagship status — 397B total, 17B active, 256K tokens of native context extensible to 1M. Qwen3.5-397B-A17B ↗

The pattern is robust enough to survive ingredient swaps. MiniMax-01 replaced the delta layers with a linear Lightning Attention — seven linear layers for every softmax one — and Ling 2.5 keeps one heavy MLA layer for every seven light linear-attention layers. Same hybrid contract, different internal machinery. MiniMax-01 ↗ Ling 2.5 / 2.6 ↗

N-grams: the strange return of explicit local memory

N-gram methods look almost prehistoric next to modern neural architectures, and that is precisely why they are interesting. An n-gram is simply a sequence of n tokens. Instead of learning every local continuation only through neural weights, a system can exploit repeated token patterns directly. Modern inference stacks already use n-gram caches and matching strategies for speculative decoding, where an inexpensive mechanism proposes likely continuations and the main model verifies them. llama.cpp speculative decoding docs ↗

But n-grams are no longer confined to the edge of the system. In 2026 they moved inside the model, as a first-class memory primitive. Engram (DeepSeek, 2026) is the clearest statement of the idea — and a direct answer to the capacity question of section 06. Repeated token patterns are stored in explicit lookup tables: hashed n-gram keys map to learned vectors, and the retrieved vectors are injected into the residual stream at fixed layers, gated by the current hidden state. Engram — DeepSeek, 2026 ↗ The table grows the model's total parameters, but a token pays only for a few constant-time lookups — capacity without proportional compute. The production proof has already landed: DeepSeek-V4.1-Flash (September 2026) ships 196 billion Engram parameters — n-gram orders {2,3,4}, eight hash heads, host-side prefetch over RDMA — inside a 552B-parameter model with a one-million-token context. DeepSeek-V4.1-Flash ↗

A wave of 2026 work explores the corners of the same space: LongCat-Flash-Lite folds hashed n-gram embeddings into the input, LongCat ↗ NGM adds untrained n-gram memory on top of an existing checkpoint, NGM ↗ TN-gram tensorizes the tables to remove hash collisions, TN-gram ↗ and SDM sparsifies a gated delta state itself. SDM ↗ The structural point for this atlas is that the third kind of memory — explicit, cheap, pattern-level — is no longer a proposer standing outside the model. It is a component inside the trunk, spending capacity the way MoE does: more total parameters, almost no extra compute per token.

This is not the same job as DeltaNet or attention. An n-gram mechanism is excellent when the relevant information is a literal or near-literal local pattern. It does not understand the pattern in the way a neural representation does. But it can be extremely cheap, and in workloads with repetition that can be enough to matter.

Seen this way, n-grams stop being an awkward historical footnote. They become another point in the architectural design space: explicit, cheap, pattern-level memory.

Multi-Token Prediction: more signal per position

Every pressure so far lives inside the model: how tokens interact, what is remembered, which parameters fire. One more axis sits at the output — how many future tokens the model is asked to predict from a single position. The classic setup asks for exactly one. Multi-token prediction (MTP) adds auxiliary heads that predict the token after next, and sometimes the one after that, all sharing the same trunk. Better & Faster LLMs via Multi-token Prediction ↗

This buys two things at once. During training, each position supervises several future tokens, so the trunk receives a denser learning signal — representations that must anticipate more than one step tend to capture structure earlier. During inference, the auxiliary heads can act as a learned speculative proposer: they guess upcoming tokens, and the main model verifies them. DeepSeek-V3 explicitly keeps its MTP module in the released model for exactly this purpose. DeepSeek-V3 Technical Report ↗

Recurrent depth: compute that loops

Every mechanism so far spends each budget once: a layer runs, a memory is written, an expert fires. One budget remains almost untouched — compute itself. The Universal Transformer proposed weight-shared depth recurrence: one shared block, applied repeatedly, refines every token in place, with an adaptive halting budget per position. Universal Transformer ↗ Follow-up theory showed the point is not parameter saving but computation: constant-size looped blocks can execute iterative algorithms and in-context learning procedures that a fixed stack of the same size provably cannot. Looped Transformers as Programmable Computers ↗ In 2025 the idea was scaled into a real language model: Huginn, a 3.5B-parameter looped-depth model trained on 800B tokens, where the number of loop iterations becomes a dial for test-time compute. Huginn ↗

The Recurrent Looped Transformer (RLT), a September 2026 tech report, pushes the loop in a new direction: across tokens. The decoder keeps a hidden state that crosses the prompt–response boundary. For each token, the previous final decoder output is merged — through a learned gate — with the token's fresh encoder representation, and the merged result passes through the decoder blocks. Three memories coexist: an encoder-derived global KV read by cross-attention, a per-layer sliding-window cache, and the recurrent state itself. RLT report ↗

The striking consequence is depth: after t tokens, the recurrent path has traversed t × L blocks while the number of blocks evaluated per token stays fixed. Recurrence buys effective depth with time instead of parameters. The evidence is early but instructive: at 26–29M parameters on algorithmic tasks, RLT generalizes far beyond training lengths where matched transformers collapse — 100% parity at 256 bits, state tracking at four times the training length — yet no language-modeling results, throughput numbers, or large-scale runs exist yet. On the map of this article it is a hybrid of explicit local memory (SWA), explicit global memory (cross-attention KV), and implicit recurrent memory — on the compute axis that mixture-of-depths approaches from an entirely different direction by routing tokens past layers. Mixture-of-Depths ↗

Most innovations are compositions

This is where the taxonomy gets much more interesting. Modern architectures are increasingly composites of mechanisms that solve different bottlenecks.

An architecture is a composition of choices — not a product name.

A hypothetical modern model could therefore be “a Transformer” and “an MoE” and “GQA” and “SWA” all at once. Another could combine Gated DeltaNet with occasional attention, a routed feed-forward block, and an auxiliary n-gram system. There is no contradiction because these mechanisms answer different questions.

The architecture matrix

One useful way to compare these families is to stop asking which one is “best” and instead ask where each one spends its budget.

Why attention still matters

It is tempting to read the history of efficient architectures as a story in which attention keeps getting replaced. The actual story is less dramatic. Attention remains extremely attractive precisely because its memory is explicit and its retrieval is content-dependent.

Even recent linear-attention work frequently reintroduces attention somewhere in the stack. Griffin combines gated linear recurrence with local attention. Griffin ↗ The Gated DeltaNet paper evaluates its own hybrids along the same lines — Gated DeltaNet layers with sliding-window attention (H1), or with Mamba2 stacked on top (H2). Kimi Linear combines KDA with MLA, and Qwen3-Next interleaves Gated Attention in a 3:1 ratio. These designs do not look like a failed attempt to escape attention. They look like attempts to reserve attention for the places where it earns its cost. Jamba pushed the same philosophy into a released large-scale hybrid, interleaving Mamba layers with attention and using MoE feed-forward blocks throughout the stack. Jamba ↗ NVIDIA's Nemotron 3 Super folds the vocabulary of this article into one open model: Mamba-2 layers, attention, latent MoE, and a shared-weight MTP head for speculative decoding. Nemotron 3 Super ↗ YOCO attacks the cache from yet another angle: a stack of local layers writes a single shared global KV cache that every later layer reads — the cache is written once, not once per layer. YOCO ↗ The contract is no longer hypothetical: DeepSeek-V4.1-Flash (September 2026) builds its causal encoder-decoder on it — twenty causal encoder layers, with the upper layers' key-value state projected from the encoder's final state, so the global cache is written once instead of once per layer. DeepSeek-V4.1-Flash ↗

The future may not be “attention versus recurrence.” It may be learning where recurrence is enough, where attention is necessary, and how the two should communicate.

A timeline of the design pressure

That is the lens I find most useful when looking at a new model. Instead of asking “what architecture is this?”, ask what the designers decided to preserve, what they decided to compress, what they decided to make sparse, and what they decided was worth paying for. Once you ask those questions, names like Transformer, MoE, SWA, Gated DeltaNet, KDA, n-grams, multi-token prediction, and learned sparse attention stop being isolated inventions. They become pieces of the same design space.

And that design space is still moving. The most interesting models are increasingly not trying to invent one perfect memory mechanism. They are trying to build a stack in which different memories handle different timescales, different retrieval problems, and different compute budgets. Even the compute axis itself is starting to move: Mixture-of-Depths routes individual tokens past entire layers, turning depth into a per-token budget rather than a fixed property of the architecture. Mixture-of-Depths ↗