Resilient Binary Neural Network

Sheng Xu, Yanjing Li, Teli Ma, Mingbao Lin, Hao Dong, Baochang Zhang, Peng Gao, Jinhu LvarXiv 2023 · DOI · OpenReview

Abstract

Binary Neural Networks (BNNs) represent a critical frontier in efficient deep learning, offering the potential to deploy sophisticated models on resource-constrained edge devices. However, the performance gap between 1-bit BNNs and their full-precision counterparts remains a significant hurdle. This paper identifies frequent weight oscillation during training as a primary cause of this gap. We present the Resilient Binary Neural Network (ReBNN), a framework designed to stabilize training through parameterized scaling factors and an adaptive, weighted reconstruction loss. ReBNN achieves state-of-the-art performance across diverse benchmarks, including a 66.9% Top-1 accuracy on ImageNet with a ResNet-18 backbone, while providing substantial hardware acceleration.

Introduction

As deep learning models grow in complexity, the demand for efficient deployment on mobile phones, IoT sensors, and other edge devices has intensified. Binary Neural Networks (BNNs) address this by quantizing 32-bit weights and activations to a single bit. This reduction theoretically slashes storage requirements by 32× and can accelerate computation by up to 58×.

Despite these advantages, BNN training is notoriously unstable. Because BNN weights are determined by their sign, even minute updates to the underlying latent real-valued weights can cause them to flip across zero. This "weight oscillation" prevents the model from converging effectively. Furthermore, traditional non-parametric scaling factors often lead to a "tri-modal" weight distribution, where many weights cluster around zero, making them highly susceptible to further oscillation. ReBNN addresses these instabilities to bridge the performance gap.

Methodology

The ReBNN framework introduces three core innovations to mitigate oscillation and improve convergence:

1. Parameterized Scaling Factors

In traditional BNNs, scaling factors are often calculated using fixed, non-parametric methods. ReBNN instead treats the scaling factor as a learnable parameter within a unified framework. This shift prevents the problematic tri-modal distribution and encourages the network to learn more robust weight representations.

2. Weighted Reconstruction Loss

To provide a more stable training objective, the authors introduce a weighted reconstruction loss term:

$$L_R(W, A) = \frac{1}{2} \sum \gamma_i^n \|w_i^n - \alpha_i^n b_i^{w^n}\|_2^2$$

In this equation, $\gamma$ serves as a "balanced parameter" that acts as a threshold to control weight oscillation. By penalizing the distance between real-valued weights ($w$) and their scaled binary counterparts ($\alpha b$), the model is encouraged to find more stable minima.

3. Resilient Gradient Update

The authors provide a mathematical foundation showing that weight oscillation occurs when the gradient magnitude exceeds the threshold $\gamma$. ReBNN employs a learning rule for $\gamma$ based on the proportion of weights that changed sign in the previous iteration and the maximum gradient magnitude. This "resilient" gradient suppresses unnecessary sign-flipping while allowing the model to remain flexible enough to learn complex features.

Experimental Results

ReBNN was evaluated across computer vision and natural language processing tasks to demonstrate its versatility.

Image Classification and Object Detection

On the ImageNet dataset, ReBNN achieved a 66.9% Top-1 accuracy using a ResNet-18 backbone. This represents a 1.0% improvement over the previous state-of-the-art, ReActNet, without increasing the parameter count. For object detection on the COCO 2014 dataset, ReBNN improved the mean Average Precision (mAP) by 3.9% over the BiDet baseline when using a Faster-RCNN framework.

Natural Language Processing

The framework's efficacy extends to Transformers. When applied to a binarized BERT architecture and tested on the GLUE benchmark (including tasks like SST-2, MNLI, and QNLI), ReBNN consistently outperformed BiBERT, proving that the resilient gradient approach is architecture-agnostic.

Hardware Efficiency

To validate practical utility, ReBNN was deployed on an ARM-based ODROID C4 using the BOLT framework. The model achieved a ~9× real-world speedup compared to full-precision models, confirming its readiness for edge AI applications.

Discussion and Insights

The success of ReBNN highlights that weight oscillation is not merely training noise but a mathematical barrier to convergence. When a weight oscillates, the loss function is likely trapped in a local maximum. By utilizing $\gamma$ as a physical stabilizer, ReBNN pushes the weight distribution from a tri-modal state toward a resilient bi-modal state. This ensures that fewer weights are "on the fence" near zero, resulting in a more robust and high-performing network.

Conclusion and Future Work

ReBNN sets a new standard for 1-bit quantization by specifically targeting the mechanics of weight instability. While the current approach involves a standard two-stage training process and requires some initial tuning of the $\gamma$ bounds, the performance gains are significant. Future research may extend this resilient gradient methodology to 2-bit or ternary networks, further narrowing the gap between ultra-low-bit models and full-precision deep learning.

Summary updated

Abstract

Binary neural networks (BNNs) have received ever-increasing popularity for their great capability of reducing storage burden as well as quickening inference time. However, there is a severe performance drop compared with real-valued networks, due to its intrinsic frequent weight oscillation during training. In this paper, we introduce a Resilient Binary Neural Network (ReBNN) to mitigate the frequent oscillation for better BNNs' training. We identify that the weight oscillation mainly stems from the non-parametric scaling factor. To address this issue, we propose to parameterize the scaling factor and introduce a weighted reconstruction loss to build an adaptive training objective. For the first time, we show that the weight oscillation is controlled by the balanced parameter attached to the reconstruction loss, which provides a theoretical foundation to parameterize it in back propagation. Based on this, we learn our ReBNN by calculating the balanced parameter based on its maximum magnitude, which can effectively mitigate the weight oscillation with a resilient training process. Extensive experiments are conducted upon various network models, such as ResNet and Faster-RCNN for computer vision, as well as BERT for natural language processing. The results demonstrate the overwhelming performance of our ReBNN over prior arts. For example, our ReBNN achieves 66.9% Top-1 accuracy with ResNet-18 backbone on the ImageNet dataset, surpassing existing state-of-the-arts by a significant margin. Our code is open-sourced at https://github.com/SteveTsui/ReBNN.
BibTeX
@misc{xu2023resilient,
  author = {Sheng Xu and Yanjing Li and Teli Ma and Mingbao Lin and Hao Dong and Baochang Zhang and Peng Gao and Jinhu Lv},
  title = {{Resilient Binary Neural Network}},
  year = {2023},
  eprint = {2302.00956},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.00956},
}
Figure