The Transformer Isn't Alone Anymore

The Transformer won the modern AI race by doing one thing exceptionally well: looking everywhere.

Every token can attend to every other token. Nothing is compressed away. The model keeps a growing key-value cache and repeatedly asks the same question:

What part of the context matters right now?

It is an extraordinarily powerful mechanism.

It is also expensive.

The longer the context becomes, the more memory the model needs. During decoding, the KV cache grows with every token. And when a model runs for hundreds of thousands of tokens, the cost of repeatedly operating on that history becomes impossible to ignore.

So the obvious question is:

What if most of the model didn't need full attention at all?

Qwen's answer is surprisingly simple.

Don't remove attention.

Use much less of it.

The architecture hidden inside Qwen

Qwen3.6-27B has 64 layers. Its layout is not a conventional Transformer stack.

It is:

In other words:

Three quarters of the network use Gated DeltaNet.

Only one quarter uses full attention.

The same architecture is used throughout the Qwen3.5/Qwen3.6 family, including the 0.8B class model that became the reference for my own experiment. Qwen3.6-27B has 48 value heads and 16 query/key heads in its Gated DeltaNet path, while its full-attention blocks use 24 query heads and 4 KV heads.

This isn't a cosmetic optimization.

The model is fundamentally no longer a Transformer in the traditional sense.

Most of its sequence processing is performed by something else.

Why not just use linear attention everywhere?

Because the two mechanisms solve different problems.

Standard attention has a superpower:

exact retrieval.

A token can directly look at a specific position hundreds or thousands of tokens away. If the model needs to copy something, retrieve a precise fact, match two distant pieces of context, or perform certain kinds of in-context learning, attention is extremely good at it.

Linear attention changes the problem.

Instead of storing the entire history and repeatedly attending over it, it maintains a fixed-size recurrent state.

The sequence becomes a state update.

Something conceptually like:

The state does not grow with the sequence length.

This is the fundamental attraction of Gated DeltaNet: the model can carry information forward without maintaining a quadratic attention operation over the entire history.

But compression has a price.

You are no longer storing the past exactly.

You are learning a state that represents whatever the network believes is worth preserving.

DeltaNet is not just “Mamba with another name”

The important part of Gated DeltaNet is the delta rule.

A simplified recurrent view looks like this:

The state S is not a simple vector.

It is a matrix.

The key produces a location in the associative memory. The current state predicts what value should be there. The difference between that prediction and the incoming value becomes the update.

That is the “delta” in DeltaNet.

The model isn't simply saying:

remember this token.

It is saying something closer to:

this is what the memory currently predicts; here is the correction.

That makes the recurrent state an associative memory rather than a generic hidden state.

The gates are the important part

Qwen's implementation adds several gates around this process.

There is a decay gate controlling how much of the existing state survives.

There is a write gate controlling how strongly new information modifies the state.

And there is output gating after the recurrent computation.

In my reconstruction, the core looks like this:

This is not a Transformer attention score matrix.

There is no Q @ Kᵀ.

There is no growing sequence dimension in the recurrent state.

The model is maintaining a learned memory.

So why keep attention?

Because memory and retrieval are not the same thing.

A useful way to think about the architecture is:

DeltaNet handles most of the sequence.

Attention periodically gets to look back at the actual context.

This is the architectural compromise.

You don't ask attention to do everything.

You don't ask the recurrent mechanism to remember everything.

You give each mechanism the job it is good at.

And Qwen did not choose 3:1 arbitrarily

This is the part I find most interesting.

Qwen3-Next introduced the hybrid design and explicitly describes experiments comparing linear-attention approaches and standard attention.

Their conclusion was that linear attention is significantly more efficient, but weaker at recall, while standard attention is strong at recall but expensive.

They found Gated DeltaNet to have stronger in-context-learning behaviour than approaches such as Sliding Window Attention and Mamba2.

Then they tested hybrids.

The result they report is a 3:1 ratio:

According to Qwen, this configuration consistently outperformed monolithic architectures in their experiments while also improving efficiency.

That ratio subsequently became the backbone of the Qwen3.5 and Qwen3.6 family.

So the architecture isn't:

“We replaced attention because attention is expensive.”

It is:

“We found that most of the network does not need full attention, but some of it absolutely does.”

That's a much more interesting claim.

The 3:1 pattern is not a coincidence

The same architecture appears across dramatically different model sizes.

At the large end, Qwen3.6-27B uses 64 layers organized as sixteen four-layer groups. At the small end, Qwen3.5-0.8B follows the same underlying hybrid pattern.

That matters.

A design that survives from sub-billion parameters all the way to a 27B dense model is probably not an incidental implementation detail.

It suggests that Qwen considers the ratio itself to be part of the architecture.

And that is exactly why I chose the 0.8B model as the reference for my own experiment.

Other labs found the same ratio — with different ideas

Two other models on my own disk make the 3:1 pattern harder to dismiss as a Qwen-specific trick.

Muse Glimmer 30B uses the same high-level rhythm, but its three non-global layers are not DeltaNet. They are Sliding Window Attention.

The local layers use a 2048-token sliding window. The fourth layer is full, global attention. Meta is therefore also spending roughly 75% of its mixer layers avoiding a full-context attention operation, then periodically paying for an exact global view.

The mechanisms are not interchangeable. Qwen compresses history into a recurrent DeltaNet state; Glimmer localizes attention to a fixed window. One is recurrent memory. The other is local retrieval.

And then there is a third, which is the one that convinced me the ratio is not a fashion. Laguna-S-2.1 is not dense and not a DeltaNet hybrid at all — it is a fine-grained mixture of experts, 256 experts with top-10 routing plus one shared expert, with a native 1M context. Its attention head counts alternate [48, 72, 72, 72], twelve times over:

Three quarters local, one quarter global — again. The sliding window is 512 rather than 2048, and the global layer sits at the head of each group rather than the tail, so the phase differs. The ratio does not.

That layout is also where the practical payoff becomes visible. Only the twelve global layers scale their KV cache with context length; the other thirty-six cap out at 512 tokens each. This is why a 1M-token context fits on a 24 GB consumer card at all — roughly 14 GB of KV instead of the ~56 GB a uniformly-global 48-layer model would demand. The ratio is not an aesthetic choice. It is the reason the model runs.

This makes the design space more interesting than a simple “attention versus linear attention” debate. There are at least three distinct ways to make the 75% cheaper or more local:

That last part is the piece I find most revealing. The architectural question may no longer be “How do we make every layer globally expressive?” but rather “How often do we need to restore global access?”

Qwen answers with recurrent memory plus periodic global retrieval. Muse and Laguna answer with local attention plus periodic global retrieval, at two different window sizes and on two different backbones. The three models disagree about how to process the 75% — but they agree about where full attention belongs in the stack.

Reconstructing the architecture from the outside

The interesting part of my experiment was that I didn't begin from a written description of the architecture.

I started from a GGUF.

My local quantization tooling already contained the ability to inspect GGUF headers, tensor names, shapes, metadata and tensor types. The inspection happens directly through llama.cpp's GGUF tooling rather than loading the model for inference. That information was given to GRM-2.6-Plus-0628, running locally in my own IQ4_XS quantization.

From shapes such as:

and their dimensional relationships, it reconstructed the structure of the model.

The resulting implementation contained:

• Gated DeltaNet recurrent state;
• causal convolution;
• Q/K normalization;
• gated RMSNorm;
• Gated Attention;
• partial RoPE;
• GQA;
• periodic attention layers;
• SwiGLU FFNs;
• and the corresponding residual structure.

The important thing is not that a language model can reproduce the name “Gated DeltaNet.”

It is that the architecture can be inferred from the physical structure of the weights.

The GGUF is, in a sense, an archaeological site.

The architecture is still there.

Then I did something slightly more unreasonable

I trained the reconstruction.

From scratch.

My current experimental model is 388M parameters, with the same basic 3:1 hybrid structure:

The corpus is 81% code and 19% English by volume — 70/30 by document count, which is not the same thing and the difference matters:

The training setup currently uses:

The most important part is not the absolute loss.

It is that the model actually learns.

The loss went from approximately:

Held-out loss followed the same trajectory throughout.

Neither of the two annotated numbers is decoration.

11.0904 is exactly ln(vocab_size): the loss of a model that spreads its probability mass evenly over all 65,536 tokens. It is ignorance, measured precisely. Every from-scratch run starts there, and seeing that exact value on the first line is the cheapest confirmation that the loss function and the vocabulary agree with each other.

7.7556 is the unigram baseline — the loss of a model that has learned nothing whatsoever except how often each token occurs. Crossing it is the first moment the model is doing something a frequency table cannot do.

As of writing, at 0.58B tokens seen (step 2,200), held-out loss is 1.8124 and training loss 1.7890 — a gap of 0.023, which is to say the model is nowhere near memorizing anything.

And the gradients remained completely stable throughout: gradient norm has sat between 0.19 and 0.24 for the entire run.

There is no spectacular trick hiding here.

It is simply a from-scratch model using an architecture that came from inspecting another model's weights.

The architecture works.

The other half of the experiment: quantization

This wasn't actually my first encounter with the strange asymmetry of this architecture.

I had already spent a significant amount of time quantizing Qwen3.5/3.6-style models and measuring where the damage occurs.

One tensor repeatedly stood out:

This is the output projection of the recurrent DeltaNet branch. It takes the representation generated by the recurrent memory and funnels it back into the residual stream.

I started protecting it.

Not because I had a theoretical proof that it was important.

Because the measurements said it was important.

Across my quantization ladder, I deliberately kept ssm_out at a higher precision than the surrounding tensors. ssm_alpha and ssm_beta remain in F32.

Then I performed the obvious experiment:

same quantization, same imatrix, same corpus, same chunking.

Only change:

versus

The protected version improved the 99th-percentile KLD in all fourteen tested conditions, across two context lengths and seven quantization tiers. Top-1 agreement improved in all fourteen as well.

At Q5_K_M, the result was particularly revealing.

WikiText perplexity suggested almost no meaningful difference — and actually pointed slightly in the wrong direction:

The perplexity moved 0.0085 backwards, roughly a fifth of its own standard error — which is to say, nothing at all. Over the very same pair of models, the tail-oriented KLD metric improved by 20%.

The average metric missed the structural effect.

The tail saw it.

That changed how I think about the architecture

The obvious interpretation of a hybrid model is:

DeltaNet is the cheap part. Attention is the important part.

But the more I inspect these models, the less convincing that interpretation becomes.

Qwen puts DeltaNet in 75% of the layers.

My quantization experiments show that the DeltaNet output path can be unusually sensitive to perturbation.

And the recurrent branch is responsible for maintaining a representation of the sequence over time.

Maybe the right mental model is the opposite:

DeltaNet carries the ongoing state of the computation.

Attention periodically breaks out of that compressed representation and accesses the context directly.

Attention is indispensable.

But it isn't necessarily the thing doing most of the work.

Why 3:1?

I don't think we know the full answer.

Qwen reports that the ratio emerged from systematic experiments, not from a purely theoretical derivation.

That's important.

There is a temptation to look at 3:1 and invent a beautiful explanation after the fact:

75% recurrence, 25% exact recall — therefore the brain needs three memory operations for every retrieval operation.

No.

That's storytelling.

What we know is much simpler:

Qwen tested the design space, found that the hybrid consistently worked well, and carried the 3:1 structure into the subsequent family.

The deeper reason is still something worth investigating.

And this is precisely where small-scale experiments become useful.

The experiment I actually want to run

Once the current pre-training run is complete, the cleanest experiment is not:

Can my model beat Qwen?

It cannot. Qwen has an enormously different data and training budget.

The interesting experiment is:

Then vary the ratio:

The question isn't whether DeltaNet is “better.”

The question is:

Where is the compute allocated most effectively?

That is experimentally answerable.

The absurdly small model is actually useful

One advantage of doing this at 388M parameters is that the experiment is cheap enough to repeat.

At frontier scale, an architecture ablation can cost millions.

At this scale, a single person with a consumer GPU can actually ask the question.

That changes the nature of architectural research.

You don't need to reproduce the entire Qwen training pipeline.

You need to isolate the variable.

And suddenly the architecture becomes something you can interrogate rather than something you simply download.

The hidden consequence: long context

The motivation for this architecture becomes even stronger as context windows grow.

A conventional attention layer has to deal with a growing sequence representation.

A recurrent DeltaNet state does not grow with sequence length in the same way.

That does not make long context free.

The recurrent state still has to be read and updated, and Gated DeltaNet can become memory-bound during decoding. But its state does not become a full history of every key and value.

This is an important distinction.

The future probably isn't:

It is more likely:

The expensive operation becomes a minority of the network instead of the definition of the entire network.

And this is why Qwen3.6-27B is interesting

Qwen3.6-27B is not merely a “27B Transformer that happens to be efficient.”

Its architecture is deliberately spending most of its layers on recurrent linear attention and a smaller fraction on full attention.

That matters when evaluating models by parameter count alone.

Two models with 27B parameters can have very different computational identities.

One can spend almost all of its sequence processing on quadratic attention.

Another can spend 75% of its mixer layers on a fixed-state recurrent mechanism.

Calling both simply “27B Transformers” hides the interesting part.

The architecture is now becoming its own research question

That is the part I didn't expect when I started looking at these models.

Initially, Gated DeltaNet looked like a practical optimization:

Then the Qwen papers made the more interesting claim:

under their experimental setup.

My own quantization experiments then suggested another asymmetry:

And now a from-scratch model based on the same architecture is actually learning.

Those are three separate observations pointing in the same direction.

None of them proves that Gated DeltaNet is universally superior to Transformers.

But together they suggest that the recurrent branch is doing substantially more than saving FLOPs.

Conclusion

The interesting question is no longer:

Can we replace the Transformer with linear attention?

That question is too binary.

The more useful question is:

Which operations actually require full attention, and which don't?

Qwen's answer is currently:

Do that again.

And again.

And again.

Keep attention where exact retrieval matters.

Let recurrent state carry the rest.

It is an unusually pragmatic architecture: not a rejection of the Transformer, but a refusal to pay Transformer prices everywhere.

I initially approached Gated DeltaNet as somebody trying to understand and quantize an unfamiliar architecture.

Then I inspected its tensors.

Then I reconstructed it.

Now I'm training one from scratch.

And the most interesting possibility is not that DeltaNet replaces attention.