Eight Layers and a Parallel FFN

The chart that didn't make sense

It started, as these things now do, with a chart on Reddit. A third-party "intelligence per parameter" index — agentic coding ability divided by model size, roughly — with a line-up that looked like this:

The metric is not an intelligence ranking, and I won't treat it as one. What caught my eye was simpler: a 33B dense model, from a lab nobody could place, sitting at the top of an efficiency chart — slightly ahead of the Qwen model it would turn out to have so much in common with.

A from-scratch 33B multimodal model with a 262K context window is not a weekend project. It is a serious training budget. So the obvious question was not "is Agnes good?" but:

Where does a 33B like this actually come from?

The answer was not in the benchmarks. It was in the repository — and it was not hiding.

The config is a two-line diff

First stop: config.json. Put Agnes next to Qwen3.8-27B and the diff collapses to two functional changes:

Forty-odd fields, and every one of them matches — down to num_position_embeddings: 2304 and rms_norm_eps: 1e-6. Note the family oddity that makes this comparison possible at all: Qwen3.8's own model_type is qwen3_5. Agnes, 3.6 and 3.8 are one architecture family wearing three name tags.

Inside the custom config class shipped with the repo, there is a small, almost tender detail:

Agnes renamed the full-attention interval key global_attention_interval. But its loader still answers to the old Qwen name, full_attention_interval, as a fallback. Renamed on the surface, compatible underneath. The loader speaks Qwen.

The tokenizer keeps the receipt

Architecture can be convergent. A tokenizer cannot. The vocabulary of a language model is a 248,320-entry fingerprint, and rebuilding one from scratch — same merges, same special tokens, same IDs — is effectively impossible by accident.

Agnes's added-token list is Qwen's added-token list. Not similar: identical, in order, IDs 248044 through 248076 — including eight <|fimm_*|> tokens from Qwen's audio-omni lineage that a text-and-image model has no business defining, plus seven reserved rows of stray Gujarati and CJK text whose only job is to exist. And then, at the very end:

Nobody builds a vocabulary by appending twelve tokens to somebody else's. You append because the base must stay untouched — because renumbering 248,077 existing rows would invalidate every embedding and every downstream artifact that already references them. The receipt is right there in the ID range.

generation_config.json — the sampling defaults — is byte-identical to Qwen3.8's. Not similar: identical. And the two files that define the BPE model itself, vocab.json (6,722,759 bytes) and merges.txt (3,353,259 bytes), are byte-identical across Agnes, Qwen3.6 and Qwen3.8 — same 248,044 entries, same 247,587 merges, same ranks. Three repositories, one tokenizer.

Their own server runs it as a Qwen

The repo ships a sglang_patch directory — the producer's own serving solution. The patch overlays exactly three files onto stock sglang. One of them is the Qwen3.5 model implementation, into which the patch inserts a weight-stream adapter:

Read that again. Agnes is served by feeding its tensors into the stock Qwen3.5 architecture class, renaming Agnes keys back into Qwen names, and folding the parallel FFN into the ordinary MLP by concatenation — mathematically identical, since the two branches share one input and sum their outputs.

"Checkpoints without a parallel branch pass through untouched." — sglang_patch/README.md

The patch's own "Numerics" section concedes the folding is not bit-exact — and immediately puts it in perspective. Measured on the nightly image (H200, 2,144 teacher-forced positions, full 248,320-way softmax), the folded serving path sits at full-vocabulary KL 5.9e-4 against the same weights served by the unpatched engine. That is below the 6.5e-4 the same README measured between the transformers and sglang implementations of one and the same checkpoint (perplexity 17.07 vs 17.05). The graft costs less than swapping runtimes. The producer's own stack treats this checkpoint as what it is: Qwen3.5-family weights plus one additive branch, cheap enough to ship in its own file and pass through untouched when absent.

Counting what can't be rounded

Parameter counts are usually marketing. Here you can compute them from model.safetensors.index.json — the index carries total_size in bytes, and at bf16 every parameter is exactly two bytes:

Take Qwen3.8-27B. Add exactly eight transformer layers — six delta, two full, keeping the architecture's 3:1 rhythm. Add a 2048-wide SwiGLU branch to every layer. You land on Agnes's parameter count exactly — not to within two percent, to the single parameter. (My first estimate had a small residual; parsing the safetensors headers instead of estimating killed it.) The headline numbers are not independent. One is derived from the other by precisely the two changes the config diff already confessed.

The packaging confesses

Weights ship as numbered shards, filled in layer order. Nineteen shards for the language model. Look at the map:

Sixty-four layers ride the main sequence. Eight layers — two complete four-layer groups, each three delta layers plus one global layer, matching the architecture's 3:1 rhythm — were cut out of positions 20–23 and 48–51 and parked in the final shard. The parallel FFN, the other additive change, likewise sits outside the main sequence in its own file.

Which yields a falsifiable prediction, made here before any weight-level comparison:

A prediction with a shape is a bet. The weights decide.

A template pins the date

One more file settled which generation of Qwen sits underneath. The chat template — the Jinja script that formats prompts — is unglamorous, rarely edited, and versioned with the model. So we hashed it:

As a bonus, this resolves a puzzle from the model card: its README documents a reasoning_effort of high/medium/low, while the actual template accepts xhigh/medium/low. Not an Agnes bug — the quirk is inherited verbatim from Qwen3.8's template. The README simply misdescribes what it inherited.

The weights answer

Everything above is file-level evidence: the config, the tokenizer, the generation defaults, the serving shim, the parameter arithmetic, the shard map, the template. Together they prove total architectural and textual compatibility with Qwen3.8-27B, plus exactly two additive modifications.

What they do not prove on their own is the only question that matters: do the numbers inside the tensors come from Qwen? That requires the weights themselves — 66 GB of them — and four tests, ranked from cheapest to most decisive. A locally-running Qwen 3.8 27B was given the download and the checklist. Here is what it found.

The copy-paste: three billion parameters, bit for bit

The cheapest decisive test ran first: compare everything except the big projection matrices — embeddings, heads, norms, vision, the small per-layer tensors — at full precision, element by element, against the official Qwen3.8-27B release:

Not "close". Not cosine 0.9999. Bit for bit — relative RMS exactly 0.0 across roughly three billion parameters. Training does not leave three billion parameters untouched down to the last bf16 bit; copying does. In the published checkpoint, nothing ever touched these blocks — as the next sections show, nothing trained anywhere in this model.

The staircase holds — and the "training" evaporates

The 72×64 fingerprint matrix came back in exactly the staircase shape predicted from the shard map:

All 64 inherited layers match their predicted counterparts. But the full-precision pass then did something better than measure the "drift": it identified it. Across every projection matrix of the 64 inherited layers and the MTP layer — some 704 matrices — the pattern is identical:

This is not training. A dense relative drift of 12% from an actual training run would have flipped ~100% of the bf16 bits — 12% is vastly larger than one bf16 ulp. The observed 70% bit-intact rate rules training out and rules pruning in: W′ = W ⊙ mask, pure non-structured magnitude pruning, no SparseGPT-style compensation, nothing else. The "moderate uniform training pass" I had read into these numbers yesterday died of a single square root.

And the eight inserted layers? I had predicted duplications or interpolations of their neighbours — falsified (cosine ≈ 0.02 against everything). My fallback — random init, then trained from scratch — is falsified too. Their output projections (out_proj, down_proj) are exact zeros, down to residual subnormals at 10⁻³⁰; their input projections sit at exactly N(0, 0.02²), the config's initializer_range. Each inserted layer computes h → h: a no-op. One single optimizer step with a non-zero learning rate would have moved those zeros. They never saw a gradient.

The parallel FFN is still in the box

The branch's weight statistics:

The config's initializer_range is 0.02. The gate and up projections carry exactly that distribution in every layer — training moves statistics, and these never moved. And the down projections are exact zeros on all 72 layers: the branch computes down_proj(silu(gate(x))·up(x)) with a zero down_proj, so its output is strictly, mathematically, 0. Not "inert-ish". Inert.

Which makes the released "33B with a parallel FFN" a model carrying 2.26 billion parameters that contribute nothing to its output — stored, fittingly, in its own separate file outside the numbered shard sequence, and treated as optional by the producer's own sglang patch. This is not a component. It is scaffolding for a training run that, in this checkpoint, has not happened yet.

The fossils are better than expected

The embedding check produced the single funniest detail of the investigation. All 248,320 rows of the embedding matrix are bit-identical to Qwen3.8 — including rows 248077–248088, the twelve rows Agnes renamed as <|agnes_bos|> and friends. And what is in those rows? Not a fresh initialization — that would be std 0.02, row norm around 4.5. These rows hold enormous vectors, with norms around 2.5 million:

Agnes's own token embeddings are Qwen3.8's pre-existing reserved junk rows — inherited unmodified, and simply given names.

The Qwen family reserves token IDs 248077–248319 in its vocabulary, and the corresponding embedding rows in the official checkpoint contain leftover non-zero garbage. Agnes kept the matrix as-is — bit for bit, garbage included — and pointed its new special tokens at the junk. You could not design a better fossil: too useless to have been rebuilt, too weird to have converged, too large to be an accident.

Where this leaves the verdict: settled, and stranger than expected. Agnes 3.0 Flash Preview is not, in the checkpoint it publishes, a trained model at all. It is a weight-space transformation of the official Qwen3.8-27B: magnitude-prune every projection matrix by 30%, insert eight identity layers at positions 20–23 and 48–51, graft a zero-gated parallel FFN onto all 72 layers, rename twelve reserved tokenizer rows — and leave everything else bit-for-bit identical. No training run is visible anywhere: every addition still sits at its exact initialization state. What the repo actually ships is a scaffold — an architecture assembled in weight space, waiting for the training run that will make it real.

What this means

First, the legal framing, because it matters and because it is short: Qwen3.8 is Apache 2.0. Taking it, pruning it, adding eight layers and a parallel FFN, and releasing the result under another name is legal. There is no piracy story here. The only story is disclosure — the model card says "Agnes" and never says "Qwen", while the repository says Qwen in every file that counts. And there is a second, quieter story: what was published is not the model the benchmarks describe.

And the efficiency crown that started all this? The published weights cannot reproduce the README's benchmark table — a Qwen3.8 at 30% sparsity with eight no-op layers and a dead branch is not the model that scored 150 points, and the README itself warns that the Preview differs from the production/API checkpoint. The "33B" in the chart's denominator is meanwhile ~40% inactive: 2.26B of dead branch, ~1B of identity layers, ~2.5B of pruned zeros — about 19–20B actually working. The chart's premise — compare what labs built — quietly becomes a comparison of what they inherited, divided by a number that includes scaffolding. The headroom was in the recipe, not the scale. And a denominator can be inflated without adding a single active neuron.