Preserving Information in Binarized Neural Networks

14 concepts16 papers1 proposal

Deep neural networks traditionally rely on 32-bit floating-point precision to capture subtle, continuous variations in feature spaces. Binarized Neural Networks (BNNs) offer an extreme alternative: compressing weights and activations down to a single bit ($\{-1, +1\}$). As reviewed by Yuan et al., this unlocks massive hardware acceleration and memory savings by replacing expensive floating-point multiplications with bitwise XNOR and population count (popcount) operations.

However, this extreme compression introduces a severe problem: quantization-induced information loss. When a continuous signal is squashed into just two states via a rigid threshold function like $\text{sign}(x)$, the network's representational capacity degrades catastrophically. Nuanced feature distributions are flattened, and the information entropy drops. This structural bottleneck prevents standard BNNs from matching full-precision accuracy on complex tasks, a challenge highlighted across applications ranging from image classification to multi-spectral image fusion (Hou et al.).

To close this performance gap, researchers have developed techniques to maximize information retention without sacrificing underlying hardware efficiency. These approaches can be understood through four main paradigms: expanding the binary vocabulary, bypassing the bottleneck via real-valued shortcuts, adapting thresholds dynamically to the data distribution, and optimizing directly for information entropy.

Expanding the Vocabulary: Multi-Base and Residual Quantization

If a single bit cannot capture the complexity of a real-valued tensor, a natural solution is to approximate the tensor using a linear combination of multiple binary tensors. This approach preserves the efficiency of bitwise operations while drastically reducing quantization error.

In ABC-Net, Lin et al. proposed approximating full-precision weights and activations using a combination of multiple binary bases weighted by real-valued scaling factors. Instead of evaluating a single inaccurate binary representation, the network computes multiple bitwise convolutions and aggregates them.

Similarly, High-Order Residual Quantization (HORQ), introduced by Li et al., solves this recursively. The first layer computes a standard binary approximation, calculates the residual error (the difference between the real and binary values), and then binarizes that error map in a second-order operation. By cascading these binary maps, the network progressively recovers the lost information.

Real-Valued Input ───► [ Binarizer 1 ] ──────► Binary Base 1
                             │
                       (Subtract)
                             ▼
                       Residual Error ──────► [ Binarizer 2 ] ───► Binary Base 2

Other methods expand the representational space by changing the underlying numerical domain. For instance, Wan et al. introduced Ternary-Binary Networks (TBN), which use ternary inputs ($\{-1, 0, +1\}$) paired with binary weights to retain zero-valued background information. Taking this further, Li et al. proposed Binary Complex Neural Networks (BCNN), leveraging the richer geometric representation of complex numbers to encode more information per bit.

Bypassing the Bottleneck: Real-Valued Shortcuts

Rather than trying to make the binary operations themselves more precise, another strategy is to allow high-precision information to flow around the binary layers, ensuring downstream layers do not receive completely degraded signals.

The foundational realization of this approach is found in Bi-Real Net. Liu et al. introduced parameter-free identity shortcuts that connect the real-valued activations before a binary convolution directly to the activations after it. Because the shortcut simply adds the continuous input to the binarized output, it preserves the continuous distribution's gradient and magnitude information across layers.

Building on this concept, Hu et al. developed Elastic-Link (EL). They observed that modern architectures relying heavily on $1\times1$ convolutions suffer severely from binarization because the spatial receptive field is too small to compensate for the 1-bit information loss. Elastic-Link adaptively injects real-valued features into the binarized outputs, dynamically adjusting how much continuous information bypasses the bottleneck based on the layer's needs.

Dynamic and Instance-Aware Thresholds

Traditional BNNs use a static thresholding function, typically splitting values at zero ($\text{sign}(x)$). This "one-size-fits-all" approach assumes that activation distributions are perfectly symmetric and static across all inputs. In reality, feature distributions shift drastically depending on the specific input image or channel.

To address this, Bulat et al. in XNOR-Net++ moved away from rigid analytical scaling factors, opting instead to learn optimal scaling parameters directly via backpropagation. However, even learned parameters remain static during inference.

To make the binarization truly adaptive, DyBNN (Zhang et al.) introduced channel-wise, input-dependent thresholds. By computing a dynamic threshold for each channel based on the incoming feature map, the network shifts its binary boundaries to better fit the underlying distribution.

Taking this concept to higher-order statistics, INSTA-BNN (Lee et al.) incorporates an instance-aware thresholding mechanism that evaluates the skewness of each input distribution using a computationally efficient cubic proxy. By adjusting the binarization threshold per instance, it minimizes the information gap caused by squashing asymmetric distributions.

Static Threshold:   Input Distribution ───► [ Threshold = 0 ] ───► Asymmetric Bits

Dynamic Threshold:  Input Distribution ───► [ Compute Skew ] ───► [ Adjusted Threshold ] ───► Balanced Bits

Optimizing for Information Retention

Finally, information loss can be combated directly through the training objective by forcing the network to maximize the information entropy of its binary states.

In DIR-Net, Zhang et al. introduced Information Maximized Binarization (IMB). They argued that standard binarization functions fail because they do not consider the information entropy of the resulting 1-bit tensors. By designing estimators that explicitly maximize this entropy during both forward and backward propagation, they ensure that the binary states retain as much distinct feature variance as possible.

Complementing this, Xu et al. proposed Contrastive Mutual Information Maximization (CMIM). This framework uses contrastive learning during training to maximize the mutual information between the 1-bit activations and their latent, full-precision counterparts. By penalizing the network when the binary representations diverge too far in information content from the high-precision features, the model learns to compress data into bits while preserving the underlying semantic structure.

In a similar vein, Gu et al. utilized Bayesian optimization (BONNs) to model the distribution of binary weights, framing binarization as a probabilistic optimization problem that minimizes the overall quantization error.

Conclusion

As summarized by Qin, the evolution of hardware-friendly deep learning relies heavily on rethinking how information is retained during compression. Whether by expanding the binary vocabulary, routing continuous signals through shortcuts, adapting thresholds dynamically, or optimizing for mutual information, these diverse strategies all target the same fundamental flaw of BNNs: the representational bottleneck. By preserving information entropy while exploiting bitwise efficiency, modern BNNs can achieve near-full-precision performance on resource-constrained edge devices.

Key Foundational Work

Towards Accurate Binary Convolutional Neural Network (2017) — contributed 1 concept elements to this direction.
Performance Guaranteed Network Acceleration via High-Order Residual Quantization (2017) — contributed 1 concept elements to this direction.
TBN: Convolutional Neural Network with Ternary Inputs and Binary Weights (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_binary_quantization_information_loss_2b386492,
  title = {binary quantization information loss},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/binary-quantization-information-loss/rdr_2b38649209c84db183e867d4b39a0e11},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_binary_quantization_information_loss_2b386492_v1,
  title = {binary quantization information loss},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/binary-quantization-information-loss/rdr_2b38649209c84db183e867d4b39a0e11/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake