Abstract
Binary Neural Networks (BNNs) represent a critical frontier in efficient deep learning, offering the potential to deploy sophisticated models on resource-constrained edge devices. However, the performance gap between 1-bit BNNs and their full-precision counterparts remains a significant hurdle. This paper identifies frequent weight oscillation during training as a primary cause of this gap. We present the Resilient Binary Neural Network (ReBNN), a framework designed to stabilize training through parameterized scaling factors and an adaptive, weighted reconstruction loss. ReBNN achieves state-of-the-art performance across diverse benchmarks, including a 66.9% Top-1 accuracy on ImageNet with a ResNet-18 backbone, while providing substantial hardware acceleration.
Introduction
As deep learning models grow in complexity, the demand for efficient deployment on mobile phones, IoT sensors, and other edge devices has intensified. Binary Neural Networks (BNNs) address this by quantizing 32-bit weights and activations to a single bit. This reduction theoretically slashes storage requirements by 32× and can accelerate computation by up to 58×.
Despite these advantages, BNN training is notoriously unstable. Because BNN weights are determined by their sign, even minute updates to the underlying latent real-valued weights can cause them to flip across zero. This "weight oscillation" prevents the model from converging effectively. Furthermore, traditional non-parametric scaling factors often lead to a "tri-modal" weight distribution, where many weights cluster around zero, making them highly susceptible to further oscillation. ReBNN addresses these instabilities to bridge the performance gap.
Methodology
The ReBNN framework introduces three core innovations to mitigate oscillation and improve convergence:
1. Parameterized Scaling Factors
In traditional BNNs, scaling factors are often calculated using fixed, non-parametric methods. ReBNN instead treats the scaling factor as a learnable parameter within a unified framework. This shift prevents the problematic tri-modal distribution and encourages the network to learn more robust weight representations.
2. Weighted Reconstruction Loss
To provide a more stable training objective, the authors introduce a weighted reconstruction loss term:
$$L_R(W, A) = \frac{1}{2} \sum \gamma_i^n \|w_i^n - \alpha_i^n b_i^{w^n}\|_2^2$$
In this equation, $\gamma$ serves as a "balanced parameter" that acts as a threshold to control weight oscillation. By penalizing the distance between real-valued weights ($w$) and their scaled binary counterparts ($\alpha b$), the model is encouraged to find more stable minima.
3. Resilient Gradient Update
The authors provide a mathematical foundation showing that weight oscillation occurs when the gradient magnitude exceeds the threshold $\gamma$. ReBNN employs a learning rule for $\gamma$ based on the proportion of weights that changed sign in the previous iteration and the maximum gradient magnitude. This "resilient" gradient suppresses unnecessary sign-flipping while allowing the model to remain flexible enough to learn complex features.
Experimental Results
ReBNN was evaluated across computer vision and natural language processing tasks to demonstrate its versatility.
Image Classification and Object Detection
On the ImageNet dataset, ReBNN achieved a 66.9% Top-1 accuracy using a ResNet-18 backbone. This represents a 1.0% improvement over the previous state-of-the-art, ReActNet, without increasing the parameter count. For object detection on the COCO 2014 dataset, ReBNN improved the mean Average Precision (mAP) by 3.9% over the BiDet baseline when using a Faster-RCNN framework.
Natural Language Processing
The framework's efficacy extends to Transformers. When applied to a binarized BERT architecture and tested on the GLUE benchmark (including tasks like SST-2, MNLI, and QNLI), ReBNN consistently outperformed BiBERT, proving that the resilient gradient approach is architecture-agnostic.
Hardware Efficiency
To validate practical utility, ReBNN was deployed on an ARM-based ODROID C4 using the BOLT framework. The model achieved a ~9× real-world speedup compared to full-precision models, confirming its readiness for edge AI applications.
Discussion and Insights
The success of ReBNN highlights that weight oscillation is not merely training noise but a mathematical barrier to convergence. When a weight oscillates, the loss function is likely trapped in a local maximum. By utilizing $\gamma$ as a physical stabilizer, ReBNN pushes the weight distribution from a tri-modal state toward a resilient bi-modal state. This ensures that fewer weights are "on the fence" near zero, resulting in a more robust and high-performing network.
Conclusion and Future Work
ReBNN sets a new standard for 1-bit quantization by specifically targeting the mechanics of weight instability. While the current approach involves a standard two-stage training process and requires some initial tuning of the $\gamma$ bounds, the performance gains are significant. Future research may extend this resilient gradient methodology to 2-bit or ternary networks, further narrowing the gap between ultra-low-bit models and full-precision deep learning.
Summary updated