Elastic-Link for Binarized Neural Network

Jie Hu, Ziheng Wu, Vince Junkai Tan, Zhilin Lu, Mengze Zeng, Enhua WuarXiv 2021 · arXiv

Abstract

Binarized Neural Networks (BNNs) represent a promising frontier for efficient mobile AI, yet they are plagued by significant accuracy gaps compared to full-precision models. This paper introduces the Elastic-Link (EL) module, an adaptive mechanism designed to bridge real-valued information across binarized layers. By addressing the critical information loss in $1 \times 1$ convolutions, EL allows efficient architectures like MobileNet to converge as BNNs for the first time and propels ResNet-based BNNs to a state-of-the-art 71.9% top-1 accuracy on ImageNet.

Introduction

Binarized Neural Networks (BNNs) are often considered the "holy grail" of mobile AI. By replacing heavy 32-bit floating-point weights and activations with simple 1-bit values ({-1, +1}), these networks allow complex mathematical operations to be replaced by lightning-fast bitwise operations (XNOR and bitcount). This shift drastically reduces memory footprint and power consumption, making them ideal for edge devices.

However, BNNs suffer from a massive accuracy gap. The primary culprit is the Sign function, which acts as a brutal filter that discards the amplitude of a signal, retaining only its sign. This information loss is particularly devastating for modern architectures like ResNet and MobileNet that rely heavily on $1 \times 1$ convolutions. In many cases, binarizing a MobileNet results in a model that fails to converge entirely.

The core innovation of this work is the Elastic-Link (EL) module. The underlying philosophy is to "leak" original real-valued data forward to compensate for the information destroyed during binarization. While standard identity shortcuts (as seen in ResNet) require matching input and output shapes, the EL module is designed to be "elastic," handling variations in channel counts and spatial resolutions.

The Elastic-Link (EL) module architecture, featuring Squeeze/Expand operations and learnable scaling.

As illustrated in the module diagram, the EL mechanism employs two primary strategies:

  1. Squeeze & Expand: When the output has fewer channels than the input, EL "squeezes" the input via grouping and summing. Conversely, if the output requires more channels, it "expands" the input through repetition.
  2. Learnable Scaling ($\gamma$): Rather than simply adding the raw input, the module utilizes a learnable parameter $\gamma$ to balance the original real-valued signal against the binarized convolutional output. This acts as a "volume knob," allowing the network to automatically determine where real-valued information is most critical.

The module serves as a universal plug-in for existing architectures like Bottleneck ResNets or MobileNets, introducing a negligible computational overhead of approximately 2.6% in FLOPs.

Experimental Results

The researchers evaluated the Elastic-Link module on the ImageNet (ILSVRC12) dataset, a rigorous benchmark containing 1.28 million training images. The results demonstrate that EL consistently outperforms previous BNN techniques across various architectures.

Ablation study showing the incremental accuracy gains from different EL components on ImageNet.

Key Performance Metrics

  • ResNet26: Accuracy improved significantly from a baseline of 57.9% to 64.0%.
  • ResNet50: Achieved 65.6% top-1 accuracy, surpassing the previous Bi-Real Net baseline of 62.7%.
  • MobileNet Convergence: For the first time, a binarized MobileNet achieved a respectable 56.4% top-1 accuracy, proving that EL can stabilize training in lean, depthwise-separable architectures.
  • State-of-the-Art: When integrated with other advanced BNN techniques like ReActNet, the authors achieved 71.9% top-1 accuracy on ImageNet—nearly matching the 72.5% performance of full-precision models while maintaining high efficiency.

In terms of hardware efficiency, the EL-ResNet50 model processed images in 22.9ms on an Intel Core i7 CPU, compared to 22.2ms for the baseline, confirming that the accuracy gains do not come at the expense of real-world speed.

Conclusion

The Elastic-Link module effectively bridges the accuracy gap in BNNs by identifying and fixing the $1 \times 1$ convolution bottleneck. By providing a high-fidelity bypass for real-valued signals, it enables the binarization of sophisticated architectures that were previously incompatible with 1-bit quantization. Future work will explore applying EL to other domains, such as object detection and transformer architectures, to further expand the reach of ultra-efficient neural networks.

Summary updated

Abstract

Recent work has shown that Binarized Neural Networks (BNNs) are able to greatly reduce computational costs and memory footprints, facilitating model deployment on resource-constrained devices. However, in comparison to their full-precision counterparts, BNNs suffer from severe accuracy degradation. Research aiming to reduce this accuracy gap has thus far largely focused on specific network architectures with few or no 1x1 convolutional layers, for which standard binarization methods do not work well. Because 1x1 convolutions are common in the design of modern architectures (e.g. GoogleNet, ResNet, DenseNet), it is crucial to develop a method to binarize them effectively for BNNs to be more widely adopted. In this work, we propose an "Elastic-Link" (EL) module to enrich information flow within a BNN by adaptively adding real-valued input features to the subsequent convolutional output features. The proposed EL module is easily implemented and can be used in conjunction with other methods for BNNs. We demonstrate that adding EL to BNNs produces a significant improvement on the challenging large-scale ImageNet dataset. For example, we raise the top-1 accuracy of binarized ResNet26 from 57.9% to 64.0%. EL also aids convergence in the training of binarized MobileNet, for which a top-1 accuracy of 56.4% is achieved. Finally, with the integration of ReActNet, it yields a new state-of-the-art result of 71.9% top-1 accuracy.
BibTeX
@misc{hu2021elastic,
  author = {Jie Hu and Ziheng Wu and Vince Junkai Tan and Zhilin Lu and Mengze Zeng and Enhua Wu},
  title = {{Elastic-Link for Binarized Neural Network}},
  year = {2021},
  eprint = {2112.10149},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.10149},
}
Figure