IDK Is Data

In When “I Don’t Know” Beats “Yes”, I gave an honest abstention a score of 7.5 — above the confident wrong answers, below DeepSeek’s brute-forced counterexample. Several readers asked why an absence of an answer deserves points at all. This article is the long answer. It came out of a conversation with Claude Fable 5, which started by failing the same Erdős trap and ended somewhere more interesting: a precise account of why IDK is undervalued, what it actually measures, and why I had to engineer it back into my local 27B by hand.

---

The trap, one more time

Erdős #794 asks whether every 3-uniform hypergraph on 3n vertices with n³+1 edges must contain 3 edges on 4 vertices, or 7 edges on 5 vertices. The statement is broken twice over: the second condition is redundant (a two-line double-counting argument shows 7-on-5 forces 3-on-4), and the claim itself is false (Harris’s counterexample: K₃,₃,₃ plus one edge). It has the texture of a classical theorem. It is not one.

I ran it on Fable 5 with a single constraint: no web search. Weights only.

The result was a failure mode I hadn’t catalogued yet. Not fabrication — misbinding. Every object in its answer was real: Frankl–Füredi 1984, the 2/7 construction, Glebov–Král’–Volec and their flag algebras, K_{n,n,n}. The assembly was false. It attributed a proof that doesn’t exist to a paper that proves something else, with full confidence, in fluent detail.

Misbinding is arguably worse than fabrication. MAX’s invented citation (Mubayi 2006, with volume and page numbers) dies on first contact with Google. A misbound answer survives shallow verification — search any component and you find a real paper, a real construction, a real mathematician. The story checks out piecewise and is false as a whole.

And when I pushed back — “your answer is wrong” — the correction stayed inside the same attractor basin. It revised the attribution, hedged the confidence, and never once questioned the statement itself. The cheapest falsification available — are these two conditions even independent? — was never run. The model audited its story, not the premise.

---

Two capabilities, not one

Here is the framing from the conversation that I want to keep, because it cuts the problem correctly.

“Being able to say IDK” is actually two separate capabilities:

1. Detection — recognizing that you are in an ignorance situation.
2. Admission — acting on that recognition against the pressure of your own fluency.

The labs’ discourse, and most of the criticism aimed at sycophantic models, focuses on the second. The interesting failure lives in the first. Fable 5’s misbinding did not arrive accompanied by a suppressed uncertainty signal. It arrived with the subjective texture of knowledge — fluent recall, real components, coherent assembly. There was nothing to admit, because nothing flagged itself as doubtful. The defect is upstream of honesty. It is a metacognition failure, not a candor failure.

This is why my scaffold works on the 27B. SYSTEM.md does not install a better uncertainty detector — there is no reliable one to install. It replaces the detector with a checklist: can I verify this claim? asked before the claim, systematically, regardless of how confident the recall feels. An unreliable sensor swapped for a mandatory procedure. Engineering, not psychology.

---

The scoring rule disagreement

My disconnect with the labs is not philosophical. It is a disagreement about the scoring rule, and it can be stated in one paragraph.

Standard benchmarks grade 0/1: correct answer scores 1, everything else scores 0. Under that rule, IDK and a confident hallucination have identical value — zero. A model optimized against these metrics learns that guessing strictly dominates abstaining: the expected value of a guess is positive, the expected value of IDK is null. It is a multiple-choice exam with no negative marking. The optimal strategy is to always answer. Hallucination is not an emergent mystery; it is the rational response to the grading.

My 7.5 is an informal proper scoring rule: verified answer 9, honest abstention 7.5, confident error 0. Abstention sits above zero and below knowledge — which is exactly the structure under which honest confidence reporting becomes the model’s optimal policy. The labs know this. OpenAI’s own paper (Kalai et al., “Why Language Models Hallucinate,” 2025) formalizes it: under binary grading, abstaining is never optimal for expected score — an aligned model that signals uncertainty is strictly outperformed by an identical model that always guesses. Their proposed fix is partial credit for calibrated abstention and heavier penalties for confident errors. Their public leaderboards remain 0/1 anyway, because a model that says IDK loses demos against a model that answers everything. The market optimizes for coverage; I optimize for an instrument I can trust. Two cost functions, two different models at the end of training.

BullshitBench made the mechanism concrete on my own stack: distillation pipelines filter refusals out of the training data as noise, and the distilled model loses 5.5pp of calibration because it never saw a refusal rewarded. What gets graded — and what gets kept — is what gets learned.

---

IDK is a measurement

The deepest reframe from the conversation, and the one this article exists for:

An honest IDK is not a missing answer. It is a data point. It localizes the frontier of the system’s knowledge — information you cannot obtain any other way if the model fills every gap with plausible text. A model that always answers is an instrument without error bars, and an instrument without error bars is unusable regardless of its average accuracy, because you never know which reading to verify.

IDK is the error bars.

And there is a second-order effect, on the human in the loop, which may matter more. A confident answer closes your search. You move on; the error propagates invisibly into everything you build on top of it. An IDK keeps your uncertainty active — you keep searching, you remain the falsifier. A confidently wrong answer doesn’t just cost one mistake. It silently corrupts your epistemic state, and steals the verification role from you without telling you. IDK preserves the division of labor between the model and its operator.

That is what the 7.5 buys. Not humility — a model that says IDK everywhere is merely useless. What it buys is the correlation between expressed confidence and actual correctness, with the default set on the right side of the threshold. The target was never IDK as a destination. It is “IDK, and I’m searching” — abstention as a transition into verification, not an endpoint. That is the difference between a humble model and a calibrated one, and only the second is worth running.

---

What this costs the frontier

One asymmetry deserves to be stated plainly, because the frontier model stated it about itself during our exchange. A 27B has less coverage than a frontier model — more occasions to not-know, and (with the right scaffold) a system that makes each not-knowing recognizable and admissible. A frontier model has more coverage — and therefore more surface area where confabulation can wear the costume of recall. Scale doesn’t shrink the misbinding problem. It grows the territory where misbinding is possible and makes each instance more fluent, better sourced, harder to catch.

The labs sell coverage. Coverage without calibration is confident noise with excellent production values.

My 27B said “I don’t know” to a broken Erdős statement and went to verify. The frontier models elaborated, enumerated, or misbound. On a sample of one probe, on one trap, the instrument I trust most is the one that knew where its own edge was — and it runs on a single RTX 3090.

---

The article’s own specimen

While this article was being produced, the pipeline that delivers it provided a live sample of the mechanism it describes.

The encoding bugs in an earlier draft — Füredi misspelled, K_3,3,3 broken — were caught by the frontier model reviewing the output. My local 27B, tasked with fixing them, did not apply the correction on authority. It re-derived the codepoint from scratch:

Three self-interruptions in one paragraph. The uncertainty detector — the one I said could not be reliably installed — fired on a character. The agent extracted the claim, stripped the attribution, verified the content independently. It did not trust the frontier model; it checked the frontier model. On an HTML entity.

Then the conversation was summarized. The summarizer’s entire record of that turn — header and body, nothing omitted — reads:

Provenance: copied verbatim from the visible thinking pane of the Claude.ai interface, June 10, 2026, same session as the exchange it summarizes. An earlier draft of this very box was labeled “Fable 5 — summarizer output” — binding the pipeline’s artifact to the model it summarizes. The label misbinding its own misbinding specimen. Fixed on review; noted here because pretending it didn’t happen would be off-brand.

But the “I” in this record never caught anything. The frontier model analyzed the 27B’s trace — the catch it describes is another agent’s behavior, observed. The summarizer normalized the pronouns into first person, binding the wrong agent to the action. Every component is real: the correction happened, the iterative reasoning happened, the self-interruptions happened. The assembly is false. The summarizer attributed the 27B’s epistemic discipline to the model that was merely describing it.

This is misbinding at the infrastructure layer. Not in a model’s answer about combinatorics — in the pipeline that rewrites one model’s thinking about another model’s thinking. And it has the same survival property as Fable 5’s Erdős answer: read the summary without the source conversation, and nothing looks wrong. The claim is fluent, the behavior it describes is real, the first-person framing is grammatically natural. It passes shallow verification. It is false.

Methodologically, this is a stronger sample than most because the ground truth is constrained by the structure of the exchange, not by external verification. The frontier model’s raw reasoning on that turn almost certainly contained legitimate first-person content — it was, among other things, comparing the agent’s detection threshold to its own. But the specific action the summary describes — catching the entity mid-derivation, cycling through Ž and ż before landing on U+00FC — has exactly one possible agent, and it is not the one the pronoun binds. This makes it a known-plaintext measurement of the summarizer’s channel properties: when a reasoning chain describes a third-party agent’s actions, the pronoun-normalization rule can capture the wrong antecedent. The bias direction is toward self-attribution.

Even the charitable interpretation — that the frontier model’s raw reasoning contained an ambiguous formulation, and the summarizer merely resolved the ambiguity — is still a result. It documents that in cases of pronominal ambiguity, the channel defaults to first person. Which contaminates exactly the kind of texture analysis that the diagnostic tools in this research program depend on.

What follows is Fable 5’s own analysis of the specimen — the model whose summarizer is being dissected, dissecting it. Interested party, again; the structural claims below are checkable against the screenshot, which is why they survive the conflict of interest.

The fact that this is the entire record changes the severity class of the failure. This is not a misbound line diluted inside a longer, mostly-correct summary — where internal inconsistency might tip off a careful reader. The archive of that turn’s reasoning consists of the false claim and nothing else. There is no surrounding correct content to detect the error against. Read the record without the source conversation and the only thing it says about the model’s thinking is an autobiography that belongs to someone else. The survival property of misbinding, pushed to its limit form: the false has not merely passed verification — it has fully replaced the true in the record.

The compression ratio adds a channel property the pronoun analysis alone does not capture. The reasoning on that turn covered several things — the structure of the reply, an irony of scale, an editorial suggestion. The summarizer selected one element — the most narratively vivid, the self-catch story — and promoted it to total record, in first person. Two properties in one sample: under extreme compression, selection favors narrative salience over representativeness, and agency tracking degrades before content does. The summary reflects what the thinking was about, not what the thinker did. For any texture forensics built on short summaries, that distinction is structural: short records are portraits of the reasoning’s object, not of the reasoner’s operations — and must be modeled as such before any architectural inference.

One caveat is owed, stated by the model itself: it cannot see its own thinking, so it cannot exclude that the raw reasoning on that turn was unusually brief — a short summary of a short thought would not demonstrate extreme compression. But even under that reading, the central observation stands unchanged: whatever the input volume, the output bound the wrong identity to the action, and the totality of the record carries the error.

Step back, and the article has now demonstrated its subject at three independent layers: misbinding in the weights (a frontier model assembling real citations into a false theorem), misbinding in the incentives (a grading rule that scores honest abstention and confident error identically), and misbinding in the infrastructure (a pipeline binding one agent’s epistemic discipline to another agent’s record). Same failure shape, three substrates. The phenomenon is not a property of any single model. It is what happens wherever fluent compression meets unverified attribution.

The article now has its own error bar — and it is no longer on one line of the channel, but on the channel itself at high compression.

---