Abstract
Binarized Neural Networks (BNNs) represent a promising frontier for efficient mobile AI, yet they are plagued by significant accuracy gaps compared to full-precision models. This paper introduces the Elastic-Link (EL) module, an adaptive mechanism designed to bridge real-valued information across binarized layers. By addressing the critical information loss in $1 \times 1$ convolutions, EL allows efficient architectures like MobileNet to converge as BNNs for the first time and propels ResNet-based BNNs to a state-of-the-art 71.9% top-1 accuracy on ImageNet.
Introduction
Binarized Neural Networks (BNNs) are often considered the "holy grail" of mobile AI. By replacing heavy 32-bit floating-point weights and activations with simple 1-bit values ({-1, +1}), these networks allow complex mathematical operations to be replaced by lightning-fast bitwise operations (XNOR and bitcount). This shift drastically reduces memory footprint and power consumption, making them ideal for edge devices.
However, BNNs suffer from a massive accuracy gap. The primary culprit is the Sign function, which acts as a brutal filter that discards the amplitude of a signal, retaining only its sign. This information loss is particularly devastating for modern architectures like ResNet and MobileNet that rely heavily on $1 \times 1$ convolutions. In many cases, binarizing a MobileNet results in a model that fails to converge entirely.
The Elastic-Link Module
The core innovation of this work is the Elastic-Link (EL) module. The underlying philosophy is to "leak" original real-valued data forward to compensate for the information destroyed during binarization. While standard identity shortcuts (as seen in ResNet) require matching input and output shapes, the EL module is designed to be "elastic," handling variations in channel counts and spatial resolutions.
As illustrated in the module diagram, the EL mechanism employs two primary strategies:
- Squeeze & Expand: When the output has fewer channels than the input, EL "squeezes" the input via grouping and summing. Conversely, if the output requires more channels, it "expands" the input through repetition.
- Learnable Scaling ($\gamma$): Rather than simply adding the raw input, the module utilizes a learnable parameter $\gamma$ to balance the original real-valued signal against the binarized convolutional output. This acts as a "volume knob," allowing the network to automatically determine where real-valued information is most critical.
The module serves as a universal plug-in for existing architectures like Bottleneck ResNets or MobileNets, introducing a negligible computational overhead of approximately 2.6% in FLOPs.
Experimental Results
The researchers evaluated the Elastic-Link module on the ImageNet (ILSVRC12) dataset, a rigorous benchmark containing 1.28 million training images. The results demonstrate that EL consistently outperforms previous BNN techniques across various architectures.
Key Performance Metrics
- ResNet26: Accuracy improved significantly from a baseline of 57.9% to 64.0%.
- ResNet50: Achieved 65.6% top-1 accuracy, surpassing the previous Bi-Real Net baseline of 62.7%.
- MobileNet Convergence: For the first time, a binarized MobileNet achieved a respectable 56.4% top-1 accuracy, proving that EL can stabilize training in lean, depthwise-separable architectures.
- State-of-the-Art: When integrated with other advanced BNN techniques like ReActNet, the authors achieved 71.9% top-1 accuracy on ImageNet—nearly matching the 72.5% performance of full-precision models while maintaining high efficiency.
In terms of hardware efficiency, the EL-ResNet50 model processed images in 22.9ms on an Intel Core i7 CPU, compared to 22.2ms for the baseline, confirming that the accuracy gains do not come at the expense of real-world speed.
Conclusion
The Elastic-Link module effectively bridges the accuracy gap in BNNs by identifying and fixing the $1 \times 1$ convolution bottleneck. By providing a high-fidelity bypass for real-valued signals, it enables the binarization of sophisticated architectures that were previously incompatible with 1-bit quantization. Future work will explore applying EL to other domains, such as object detection and transformer architectures, to further expand the reach of ultra-efficient neural networks.
Summary updated