Traditional neural network quantization focuses on a simple goal: make the quantized weights as numerically close to the original weights as possible. This is typically achieved by minimizing the Mean Squared Error (MSE) between full-precision values and their discrete counterparts. However, as we push models toward ultra-low bit-widths—such as 4-bit, 2-bit, or even 1-bit Binary Neural Networks (BNNs)—this "distance-based" perspective begins to fail.
A growing body of research suggests that the secret to high-accuracy compression isn't minimizing numerical error, but rather maximizing information entropy. By ensuring that a model's limited discrete states are used as efficiently as possible, researchers are closing the gap between compressed and full-precision models.
The Information Bottleneck of Low-Bit States
In a 1-bit system, each weight can only be in one of two states: -1 or +1. If a network's weights are poorly distributed—for example, if 99% of the weights are +1 and only 1% are -1—the network has very low entropy. It is effectively "wasting" its already tiny bit-budget.
Haotong Qin and colleagues argue in IR-Net that binarization should be treated as an information retention task. When weights cluster around a single value, the model suffers from "diversity collapse," where the network loses its ability to represent complex features. To combat this, researchers like Mingzhu Shen in BBG-Net propose centering weights by subtracting their mean before binarizing. This simple balancing act ensures a 50/50 split of binary states, maximizing the Shannon entropy of the layer and allowing it to carry the maximum possible information.
From Bell Curves to Uniform Boxes
Most neural network weights naturally follow a Gaussian (bell-shaped) distribution, with most values huddled near zero. If we apply a linear quantization grid to this bell curve, the bins at the edges remain nearly empty while the center bins are overcrowded.
To solve this, Tim Dettmers and Luke Zettlemoyer introduced Quantile Quantization in their work on LLM inference scaling laws. Instead of equal-sized steps in value, they use equal-sized steps in probability. This ensures that each quantization bin contains an equal number of weights, fully saturating the information capacity of the 4-bit format.
Taking this further, Ningfeng Yang and Tor M. Aamodt developed Bell Box Quantization (BBQ). BBQ uses mathematical transforms to flatten the "Bell" of the weight distribution into a "Box" (a uniform distribution). By mapping weights to a uniform space before quantizing, they achieve information-theoretically optimal entropy, which significantly reduces perplexity in 1-bit models.
Preserving the Signal via KL Divergence
While maximizing weight entropy is crucial, the ultimate goal is preserving the model's functional behavior—its "signal." Recent work on Large Language Models (LLMs) has shifted the focus from weight reconstruction to distribution matching.
In SliM-LLM, Haotong Qin and Wei Huang use Kullback-Leibler (KL) divergence to optimize bit allocation. Instead of asking "which weights have the most error?", they ask "which bit-allocation minimizes the difference between the original and compressed output distributions?" This approach prioritizes the preservation of the semantic signal over numerical precision. Similarly, Xudong Ma in IR-QLoRA uses entropy-maximizing calibration to ensure that the diversity of internal representations is maintained even at 2-bit precision.
The Entropy-Error Trade-off
Maximizing entropy is not a free lunch. There is a fundamental tension: forcing weights into a perfectly uniform distribution to maximize entropy can sometimes increase the raw quantization error.
Ling Shao and Mingbao Lin explore this in ReCU. They treat weight standardization as a way to transform "uncontrolled" entropy into a tunable parameter. By "reviving" dead weights in the tails of the distribution and moving them toward the center, they balance the need for weight diversity with the need for stable convergence.
Expanding the State Space
Finally, some researchers are increasing entropy by expanding the available "slots" for information without increasing the bit-count. In iFairy, Feiyu Wang and colleagues move from the real number line to the complex plane. By quantizing weights to the four roots of unity ($\pm 1, \pm i$), they utilize all four states of a 2-bit budget more effectively than real-valued ternary quantization. This "raises the ceiling" of what low-bit models can represent by maximizing the directional entropy of the parameters.
By shifting the objective from "minimize error" to "maximize information," these researchers have turned quantization from a destructive process into an optimization of the model's inherent representational capacity.