What Ternary Weights Taught Me About Why Models Hallucinate

October 13, 2026

bitnet quantization information-theory hallucination llm-architecture

The Three Numbers Problem

A few months ago I was reading through the BitNet b1.58 papers expecting to find the usual quantization story — some clever tricks to shrink a model, a modest accuracy hit, a tradeoff chart. What I found instead unsettled me a little. Constrain every weight in a transformer to one of three values — -1, 0, or +1 — train it that way from scratch, and past about 3 billion parameters it matches full 16-bit precision on perplexity and downstream tasks. Not “close enough for edge deployment.” Matches. You can throw away roughly 14.4 of every 16 bits per parameter and lose nothing that anyone can measure.

Sit with that for a second, because it’s a strange result. Sixteen bits per weight gives you 65,536 possible values. Three values is $\log_2(3) \approx 1.58$ bits. We went from a number line to a coin with a “meh” option, and the model didn’t care. Whatever intelligence lives in those weights, it apparently never needed the other 65,533 values in the first place.

Compressors, Not Filing Cabinets

Once that result is sitting in front of you, it’s hard to keep thinking about LLMs as something like a very large, very organized filing cabinet — a system that looked up facts because it had somewhere to put them. A filing cabinet doesn’t survive having 90% of its precision deleted. A compressor does, because a compressor was never storing the raw data to begin with; it was storing a model of the data, and models tolerate quantization in a way that lookup tables don’t.

The numbers back this up in an almost comic way. Llama 3 8B was trained on roughly 15 trillion tokens using 8 billion parameters — about 1,875 tokens of experience per parameter. There is no honest sense in which that’s memorization. It’s physically impossible for 8 billion numbers, ternary or otherwise, to encode 15 trillion tokens losslessly. What the training process does instead is force the network to find the cheapest possible generalization that still predicts the data reasonably well — the mathematical shortcuts, not the specific instances. That squeeze is, per the research I’ve been reading on this, the actual mechanism of what we call “intelligence” in these systems. Not a side effect of compression. The compression, working.

Hallucination Stops Being a Bug

This is where BitNet stops being a fun quantization anecdote and starts reframing something I used to think about wrong. If a model is a lossy compressor of its training distribution, then hallucination isn’t a defect that a smarter architecture or a bigger cluster eventually irons out — it’s the compression artifact you’d expect, mathematically, every time. It’s the JPEG blocking of language.

There’s a cleaner version of this argument than my intuition, and it comes from information theory directly rather than analogy. Recent work on reliability ceilings in language models shows that every generative task has a hard entropy floor — a gap between “resolvable” error (fixable with more context, more capacity, more data) and a stubborn “subjective” component baked into the ambiguity of the task itself, which no amount of scale touches. Separately, there’s a computability argument, using plain diagonalization, that no computably enumerable model can correctly answer every computable query. Long-tail facts require sample complexity no training corpus will ever satisfy. Put together: for any model you can actually build, there exist inputs on which it must fail. Not “tends to.” Must.

That’s the sentence that changed how I think about this. We’ve spent a couple of years treating hallucination like a debugging problem — more RLHF, more retrieval, more parameters, and eventually it goes away. But if it’s a compression artifact enforced by finite description length, then scaling doesn’t approach zero, it approaches an asymptote defined by how much entropy the task actually contains. You can move the ceiling. You can’t remove it.

What This Buys You Instead

The interesting part is that once you stop chasing the ceiling, a bunch of other 2026-era engineering makes a lot more sense as the same idea applied elsewhere. Cloudflare’s Unweight work found that BF16 exponent fields — a full 8 bits of storage — carry only about 2.6 bits of actual Shannon entropy in trained models, and squeezed that gap out losslessly. TurboQuant does something similar for the KV cache, using random rotation to spread outlier energy evenly across dimensions so a cheap quantizer can compress attention vectors 8x without biasing the geometry that attention depends on. None of these are separate tricks. They’re the same discovery — that trained neural representations are stuffed with redundancy nobody was using — showing up in three different parts of the stack.

I used to think the goal was to get hallucination to zero. Now I think the more honest goal is figuring out, for a given task, where its actual entropy floor sits — and building systems (verification, retrieval, narrower scopes) around the assumption that the floor is real, not temporary. What would change about how you build with these models if you stopped waiting for the ceiling to disappear?

Sources

The C64 Runs a Transformer and You're Still Paying $25/Million Tokens

June 4, 2026

quantization model-efficiency llm-inference edge-ai scaling-laws