Maximizing Information Entropy in Quantized Neural Networks

40 concepts29 papers1 proposal

When compressing a neural network's weights and activations into ultra-low bitwidths—such as 1-bit binarization or 2-to-4-bit quantization—the traditional objective has been to minimize geometric distance. Typically, algorithms attempt to minimize the Mean Squared Error (MSE) between the high-precision continuous values and their discrete counterparts.

However, this geometric framing suffers from a fundamental flaw when applied to extreme compression. Neural network weights naturally form a bell-shaped, Gaussian-like distribution. If we map this bell curve onto uniform quantization intervals, the outer bins are rarely used, while the central bins become overcrowded.

In information theory, an imbalanced distribution across discrete states represents a severe loss of Shannon entropy. The available bit-budget is wasted on underutilized states, leading to what researchers call information collapse, where the model's internal representational richness is severely diminished.

To solve this, a prominent research direction shifts the optimization paradigm from minimizing numerical error to maximizing information entropy. The core intuition is simple: to preserve a model's expressive capacity within a strict bit constraint, every discrete quantization level must carry the maximum possible amount of information. This means forcing the quantized states to be utilized as uniformly as possible. As shown by Qin et al., prioritizing the preservation of statistical distribution and information entropy is far more critical for maintaining model performance than minimizing the literal numerical distance between original and quantized values.

Reshaping the Bell into a Box

How do we prevent bell-shaped distributions from wasting quantization bins? The most direct approach is histogram equalization—transforming the weight distribution so that it fills the discrete bins uniformly.

Early work by Zhou et al. introduced this concept via Balanced Quantization, proving a direct positive correlation between bit utilization entropy and final model accuracy. By forcing an equal distribution of weights across quantization bins, the effective information capacity of low-bitwidth models is maximized.

More recently, Yang et al. formalized this during training with Bell Box Quantization (BBQ). They mathematically transformed weight distributions from a "Bell" (Gaussian) to a "Box" (Uniform) shape, demonstrating that higher Shannon entropy in quantized weights directly correlates with lower perplexity and higher learning capacity in compressed architectures.

Maximizing Entropy in 1-Bit Binarization

In the extreme case of 1-bit Binary Neural Networks (BNNs), weights are restricted to just two states (typically $-1$ and $+1$). Here, maximizing entropy means achieving a perfect 50/50 split between positive and negative states. If a layer's weights are predominantly $+1$, the layer carries almost no information.

To combat this, Lin et al. proposed aligning weight vectors with the binary hypercube using a rotation scheme, which naturally encourages an approximately equal distribution of binary states. Similarly, Zhang et al. utilized Information Maximized Binarization, where weights and activations are explicitly zero-centered and standardized prior to quantization. This balancing act maximizes the information entropy of the weights, ensuring a diverse representational capacity without adding floating-point overhead.

This principle extends beyond 2D images; Qin et al. demonstrated that for 3D point cloud networks, the preservation of feature distribution entropy is a vastly more significant determinant of model accuracy than the minimization of individual quantization errors.

Distribution Matching via KL Divergence

Instead of forcing strict uniformity across bins, another facet of this research direction focuses on preserving the semantic signal by minimizing the Kullback-Leibler (KL) divergence between the high-precision and quantized distributions.

For instance, Tu et al. showed that minimizing KL divergence allows binary representations to retain the structural shape and information density of the original weights far better than point-wise error minimization. In the context of Large Language Models (LLMs), Huang et al. optimized mixed-precision bit allocation by minimizing the KL divergence of the model's output activations, ensuring that the overall information flow and probability distributions remain intact.

To achieve this without relying on external calibration data, Ghaffari et al. treated quantization as a statistical classification task, using adaptive penalties as a mathematical proxy for KL minimization to safeguard the model's Shannon information content.

Entropy-Aware Regularization and Clustering

When training a network to be quantized (Quantization-Aware Training), weights must transition smoothly from continuous values to discrete bins. Without careful guidance, weights can collapse into a few redundant values during this transition.

To prevent this, the authors of SDQ: Stochastic Differentiable Quantization with Mixed Precision introduced an entropy-preserving regularization technique. This technique forces weights to cluster tightly around discrete quantization levels while encouraging a uniform distribution across those bins, preserving the network's internal representation capacity during forward propagation. Liu et al. applied a similar entropy-preserving regularization to ultra-low bit speaker verification systems, ensuring that the limited capacity of a 1-bit system is fully utilized for feature extraction.

The Fundamental Trade-Off

While maximizing entropy prevents information collapse, it introduces a delicate balancing act. As Shao observes, optimization in binary networks involves a fundamental trade-off: minimizing literal quantization error can lead to a loss of weight diversity, whereas maximizing entropy can increase the raw numerical error. Successfully navigating this trade-off—by reshaping distributions rather than blindly squashing values—allows modern quantized networks to achieve high accuracy, efficient convergence, and massive compression simultaneously.

Key Foundational Work

Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks (2017) — contributed 3 concept elements to this direction.
Forward and Backward Information Retention for Accurate Binary Neural Networks (2020) — contributed 2 concept elements to this direction.
Balanced Binary Neural Networks with Gated Residual (2020) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_information_preserving_quantization_c1681fac,
  title = {information-preserving quantization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/information-preserving-quantization/rdr_c1681facc98e4e4590bb36f010503f74},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_information_preserving_quantization_c1681fac_v1,
  title = {information-preserving quantization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/information-preserving-quantization/rdr_c1681facc98e4e4590bb36f010503f74/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake