Abstract
Binary Convolutional Neural Networks (BCNNs) offer massive efficiency gains for edge computing but typically suffer from a significant accuracy drop compared to full-precision counterparts. This paper introduces Distillation Guided Residual Learning (DGRL), a framework that bridges this performance gap. By supervising BCNNs to mimic intermediate feature maps of full-precision networks and incorporating a lightweight "Squeeze-and-Interaction" shortcut, DGRL enables binary layers to capture missing information and achieve state-of-the-art accuracy.
Introduction
As deep learning moves toward the "edge"—including mobile phones, IoT devices, and embedded systems—the demand for efficient architectures has never been higher. Standard Floating-Point CNNs (FCNNs) are often too computationally expensive for these environments. BCNNs, which constrain weights and activations to just 1-bit (±1), represent the ultimate solution for efficiency, offering up to 64x theoretical speedups and 32x memory savings.
However, BCNNs are notoriously difficult to train. Their limited representational capacity and the non-differentiable nature of the "sign" function lead to a substantial "performance gap." The authors observe that this gap manifests as significant residuals (differences) between the feature maps of a BCNN and its full-precision counterpart. As the network deepens, these errors accumulate, leading to poor final accuracy.
Methodology
The authors propose a two-pronged approach to address the limitations of binary blocks:
1. Block-wise Distillation
Instead of relying solely on final output supervision, the authors enforce a distillation loss at every convolutional block. A teacher FCNN guides a student BCNN by providing intermediate targets. To ensure robust supervision, the loss is computed using both spatial-wise and channel-wise max pooling, forcing the BCNN to capture both the spatial structure ("where") and the semantic content ("what") of the teacher's features.
2. Squeeze-and-Interaction (SI) Shortcut
Recognizing that pure binary layers may be fundamentally unable to represent complex features, the authors introduce a tiny, lightweight shortcut branch to each block.
- Squeeze: This step reduces the channel dimension to a small fraction ($S < C$).
- Interaction: A sparse matrix maps these features back to the original dimension. This branch acts as a "residual compensator," adding just enough floating-point capacity (approximately 10% overhead) to recover information lost during binarization.
3. Biased Sign Function
Standard binarization uses a fixed threshold of 0. The authors introduce a trainable threshold $t$, allowing the network to learn the optimal point to flip between -1 and 1, which significantly reduces quantization error.
Experimental Results
The researchers evaluated DGRL on two primary benchmarks: CIFAR-10 for initial ablation and ImageNet (ILSVRC12) for large-scale validation.
Key Performance Metrics
- ImageNet Accuracy: Using a ResNet-18 backbone, DGRL achieved 60.45% top-1 accuracy. This significantly outperforms the classic XNOR-Net (51.2%) and the previous state-of-the-art CI-BCNN (59.9%).
- Efficiency: Compared to a full-precision ResNet-18 (69.3% accuracy), DGRL reduces FLOPs by roughly 40x and memory usage by 29x, while trailing by only ~9% in accuracy.
- Inference Speed: On a 1080TI GPU, the SI shortcut added negligible latency, increasing processing time from 1.875ms to only 2.063ms per image.
Discussion and Insights
The core innovation of this work is the "Residual Perspective." By treating the performance gap as a modelable signal, the authors use tiny auxiliary branches to handle the error that binary layers cannot resolve.
Global channel selection analysis revealed that shallow blocks (closer to the input) benefit most from SI shortcuts. These early layers form the foundation for discriminative features; if they are poorly represented, the deeper network cannot recover. Furthermore, providing intermediate "hints" via distillation smooths the optimization landscape, which is otherwise noisy due to discrete binary constraints.
Conclusion and Future Work
DGRL sets a new benchmark for binary networks, proving that strategic residual compensation and intermediate distillation can drastically improve 1-bit model performance. While a 9% gap remains compared to full-precision models, future work may extend this logic to object detection and semantic segmentation, or explore further quantizing the SI shortcut itself to move toward entirely binary inference.
Summary updated