What Ternary Weights Taught Me About Why Models Hallucinate

Tue, 13 Oct 2026 19:41:14 +0000

The Three Numbers Problem

A few months ago I was reading through the BitNet b1.58 papers expecting to find the usual quantization story — some clever tricks to shrink a model, a modest accuracy hit, a tradeoff chart. What I found instead unsettled me a little. Constrain every weight in a transformer to one of three values — -1, 0, or +1 — train it that way from scratch, and past about 3 billion parameters it matches full 16-bit precision on perplexity and downstream tasks. Not “close enough for edge deployment.” Matches. You can throw away roughly 14.4 of every 16 bits per parameter and lose nothing that anyone can measure.

Llm-Architecture on BRYSGO

What Ternary Weights Taught Me About Why Models Hallucinate

The Three Numbers Problem