equal bits: enforcing equally distributed binary network weights

Yunqiang Li, Silvia Laura Pintea, Jan van GemertOpenReview 2022 · OpenReview

Abstract

Binary networks are extremely efficient as they use only two symbols to define the network: {+1, −1}. One can make the prior distribution of these symbols a design choice. The recent IR-Net of Qin et al. argues that imposing a Bernoulli distribution with equal priors (equal bit ratios) over the binary weights leads to maximum entropy and thus minimizes information loss. However, prior work cannot precisely control the binary weight distribution during training, and therefore cannot guarantee maximum entropy. Here, we show that quantizing using optimal transport can guarantee any bit ratio, including equal ratios. We investigate experimentally that equal bit ratios are indeed preferable and show that our method leads to optimization benefits. We show that our quantization method is effective when compared to state-of-the-art binarization methods, even when using binary weight pruning.

Introduction: The Information Loss Challenge

Binary Neural Networks (BNNs) represent the frontier of efficient deep learning, substituting standard 32-bit floating-point weights with a minimal set of values: $\{-1, +1\}$. This transition offers massive reductions in memory footprint and dramatic speedups in computation. However, the primary obstacle to widespread BNN adoption is the significant accuracy gap compared to full-precision models, largely driven by information loss during the binarization process.

Traditionally, BNNs rely on the sign() function for quantization. While simple, this approach treats weights independently and lacks a global perspective on the resulting weight distribution. If a majority of weights gravitate toward a single value (e.g., 90% positive), the network's entropy drops, leading to a loss of representational capacity. While recent advancements like IR-Net have attempted to encourage balanced bit ratios, they lack the mechanisms to strictly enforce them, often failing to reach the theoretical maximum entropy required for optimal information retention.

The Bi-Half Method: Optimal Transport for Binarization

To address these limitations, this paper introduces bi-half, a novel quantization strategy that shifts from independent thresholding to a global optimization framework based on Optimal Transport.

Instead of applying a zero-threshold, bi-half considers the weights within a filter as a collective. The method sorts the latent real-valued weights and assigns the top 50% to $+1$ and the bottom 50% to $-1$. This approach effectively minimizes the Wasserstein distance between the latent weights and a target Bernoulli distribution with equal priors. By doing so, bi-half guarantees an exact 50/50 distribution of bits at every step of training.

Uneven bit flips in IR-Net during training.

This global constraint introduces a dynamic update mechanism. Because binarization depends on a weight's rank relative to its peers, the training process involves "flipping" bits to maintain the required ratio. This provides a more stable and controlled optimization path compared to the often uneven and unpredictable flips seen in prior state-of-the-art methods like IR-Net.

Key Insights: Maximum Entropy and Search Space Regularization

The core hypothesis of bi-half is that Maximum Entropy equals Maximum Information. By forcing a perfectly balanced weight distribution, the network achieves a weight entropy of 1.0. The authors demonstrate that this high weight entropy directly correlates with higher "activation entropy," ensuring that more information is preserved as data propagates through the layers.

Entropy comparison between Sign, IR-Net, and Bi-half for weights and activations.

Beyond information theory, enforcing a 50/50 ratio acts as a powerful regularizer by significantly reducing the weight search space. In a theoretical 2D example with 12 weights, a standard BNN allows for 4,096 possible combinations. The bi-half constraint narrows this down to just 924 valid configurations. This smaller, more focused search space helps the optimizer avoid poor local minima and converge on higher-quality solutions.

Experimental Results

The bi-half method was rigorously tested against standard benchmarks and state-of-the-art (SOTA) techniques:

  • ImageNet Performance: On the challenging ImageNet dataset, bi-half achieved 60.40% Top-1 accuracy with a ResNet-18 backbone, outperforming previous leaders like RBNN (59.9%) and IR-Net (58.1%). For ResNet-34, it reached 64.17%.
  • Optimization Stability: Repeated trials on CIFAR-10 showed that bi-half consistently achieves lower training and test losses than sign-based methods, indicating superior optimization stability.
  • Pruning Compatibility: The method remains effective even in sparse regimes. When combined with weight pruning, bi-half outperformed the Multi-Prize Lottery Ticket (MPT) baseline across various pruning ratios.

Conclusion and Future Work

Bi-half demonstrates that precise control over weight distributions is a key lever for improving BNN performance. By leveraging optimal transport to guarantee maximum entropy, the method sets new benchmarks for accuracy and optimization efficiency. While the current implementation focuses on a 50/50 weight split, future research could explore the benefits of non-equal priors for specialized tasks or extend these hard constraints to activation distributions to further minimize information loss.

Summary updated

Abstract

Binary networks are extremely efficient as they use only two symbols to define the network: {+1, −1}. One can make the prior distribution of these symbols a design choice. The recent IR-Net of Qin et al. argues that imposing a Bernoulli distribution with equal priors (equal bit ratios) over the binary weights leads to maximum entropy and thus minimizes information loss. However, prior work cannot precisely control the binary weight distribution during training, and therefore cannot guarantee maximum entropy. Here, we show that quantizing using optimal transport can guarantee any bit ratio, including equal ratios. We investigate experimentally that equal bit ratios are indeed preferable and show that our method leads to optimization benefits. We show that our quantization method is effective when compared to state-of-the-art binarization methods, even when using binary weight pruning. Our code is available at https://github.com/liyunqianggyn/Equal-Bits-BNN.
BibTeX
@misc{li2022equal,
  author = {Yunqiang Li and Silvia Laura Pintea and Jan van Gemert},
  title = {{equal bits: enforcing equally distributed binary network weights}},
  year = {2022},
  url = {https://openreview.net/forum?id=nrr85QGWMf0},
  howpublished = {OpenReview},
}

Loading related content...

Figure