Abstract
Binary networks are extremely efficient as they use only two symbols to define the network: {+1, −1}. One can make the prior distribution of these symbols a design choice. The recent IR-Net of Qin et al. argues that imposing a Bernoulli distribution with equal priors (equal bit ratios) over the binary weights leads to maximum entropy and thus minimizes information loss. However, prior work cannot precisely control the binary weight distribution during training, and therefore cannot guarantee maximum entropy. Here, we show that quantizing using optimal transport can guarantee any bit ratio, including equal ratios. We investigate experimentally that equal bit ratios are indeed preferable and show that our method leads to optimization benefits. We show that our quantization method is effective when compared to state-of-the-art binarization methods, even when using binary weight pruning.
Introduction: The Information Loss Challenge
Binary Neural Networks (BNNs) represent the frontier of efficient deep learning, substituting standard 32-bit floating-point weights with a minimal set of values: $\{-1, +1\}$. This transition offers massive reductions in memory footprint and dramatic speedups in computation. However, the primary obstacle to widespread BNN adoption is the significant accuracy gap compared to full-precision models, largely driven by information loss during the binarization process.
Traditionally, BNNs rely on the sign() function for quantization. While simple, this approach treats weights independently and lacks a global perspective on the resulting weight distribution. If a majority of weights gravitate toward a single value (e.g., 90% positive), the network's entropy drops, leading to a loss of representational capacity. While recent advancements like IR-Net have attempted to encourage balanced bit ratios, they lack the mechanisms to strictly enforce them, often failing to reach the theoretical maximum entropy required for optimal information retention.
The Bi-Half Method: Optimal Transport for Binarization
To address these limitations, this paper introduces bi-half, a novel quantization strategy that shifts from independent thresholding to a global optimization framework based on Optimal Transport.
Instead of applying a zero-threshold, bi-half considers the weights within a filter as a collective. The method sorts the latent real-valued weights and assigns the top 50% to $+1$ and the bottom 50% to $-1$. This approach effectively minimizes the Wasserstein distance between the latent weights and a target Bernoulli distribution with equal priors. By doing so, bi-half guarantees an exact 50/50 distribution of bits at every step of training.
This global constraint introduces a dynamic update mechanism. Because binarization depends on a weight's rank relative to its peers, the training process involves "flipping" bits to maintain the required ratio. This provides a more stable and controlled optimization path compared to the often uneven and unpredictable flips seen in prior state-of-the-art methods like IR-Net.
Key Insights: Maximum Entropy and Search Space Regularization
The core hypothesis of bi-half is that Maximum Entropy equals Maximum Information. By forcing a perfectly balanced weight distribution, the network achieves a weight entropy of 1.0. The authors demonstrate that this high weight entropy directly correlates with higher "activation entropy," ensuring that more information is preserved as data propagates through the layers.
Beyond information theory, enforcing a 50/50 ratio acts as a powerful regularizer by significantly reducing the weight search space. In a theoretical 2D example with 12 weights, a standard BNN allows for 4,096 possible combinations. The bi-half constraint narrows this down to just 924 valid configurations. This smaller, more focused search space helps the optimizer avoid poor local minima and converge on higher-quality solutions.
Experimental Results
The bi-half method was rigorously tested against standard benchmarks and state-of-the-art (SOTA) techniques:
- ImageNet Performance: On the challenging ImageNet dataset, bi-half achieved 60.40% Top-1 accuracy with a ResNet-18 backbone, outperforming previous leaders like RBNN (59.9%) and IR-Net (58.1%). For ResNet-34, it reached 64.17%.
- Optimization Stability: Repeated trials on CIFAR-10 showed that bi-half consistently achieves lower training and test losses than sign-based methods, indicating superior optimization stability.
- Pruning Compatibility: The method remains effective even in sparse regimes. When combined with weight pruning, bi-half outperformed the Multi-Prize Lottery Ticket (MPT) baseline across various pruning ratios.
Conclusion and Future Work
Bi-half demonstrates that precise control over weight distributions is a key lever for improving BNN performance. By leveraging optimal transport to guarantee maximum entropy, the method sets new benchmarks for accuracy and optimization efficiency. While the current implementation focuses on a 50/50 weight split, future research could explore the benefits of non-equal priors for specialized tasks or extend these hard constraints to activation distributions to further minimize information loss.
Summary updated