Distillation Guided Residual Learning for Binary Convolutional Neural Networks

Jianming Ye, Shiliang Zhang, Jingdong Wang

Abstract

Binary Convolutional Neural Networks (BCNNs) offer massive efficiency gains for edge computing but typically suffer from a significant accuracy drop compared to full-precision counterparts. This paper introduces Distillation Guided Residual Learning (DGRL), a framework that bridges this performance gap. By supervising BCNNs to mimic intermediate feature maps of full-precision networks and incorporating a lightweight "Squeeze-and-Interaction" shortcut, DGRL enables binary layers to capture missing information and achieve state-of-the-art accuracy.

Introduction

As deep learning moves toward the "edge"—including mobile phones, IoT devices, and embedded systems—the demand for efficient architectures has never been higher. Standard Floating-Point CNNs (FCNNs) are often too computationally expensive for these environments. BCNNs, which constrain weights and activations to just 1-bit (±1), represent the ultimate solution for efficiency, offering up to 64x theoretical speedups and 32x memory savings.

However, BCNNs are notoriously difficult to train. Their limited representational capacity and the non-differentiable nature of the "sign" function lead to a substantial "performance gap." The authors observe that this gap manifests as significant residuals (differences) between the feature maps of a BCNN and its full-precision counterpart. As the network deepens, these errors accumulate, leading to poor final accuracy.

Methodology

The authors propose a two-pronged approach to address the limitations of binary blocks:

1. Block-wise Distillation

Instead of relying solely on final output supervision, the authors enforce a distillation loss at every convolutional block. A teacher FCNN guides a student BCNN by providing intermediate targets. To ensure robust supervision, the loss is computed using both spatial-wise and channel-wise max pooling, forcing the BCNN to capture both the spatial structure ("where") and the semantic content ("what") of the teacher's features.

2. Squeeze-and-Interaction (SI) Shortcut

Recognizing that pure binary layers may be fundamentally unable to represent complex features, the authors introduce a tiny, lightweight shortcut branch to each block.

  • Squeeze: This step reduces the channel dimension to a small fraction ($S < C$).
  • Interaction: A sparse matrix maps these features back to the original dimension. This branch acts as a "residual compensator," adding just enough floating-point capacity (approximately 10% overhead) to recover information lost during binarization.
Visual comparison of feature map residuals. Row (b) shows the large errors in standard BCNNs, while row (c) demonstrates how the proposed DGRL method minimizes these residuals to better match the FCNN teacher in row (a).

3. Biased Sign Function

Standard binarization uses a fixed threshold of 0. The authors introduce a trainable threshold $t$, allowing the network to learn the optimal point to flip between -1 and 1, which significantly reduces quantization error.

Experimental Results

The researchers evaluated DGRL on two primary benchmarks: CIFAR-10 for initial ablation and ImageNet (ILSVRC12) for large-scale validation.

Key Performance Metrics

  • ImageNet Accuracy: Using a ResNet-18 backbone, DGRL achieved 60.45% top-1 accuracy. This significantly outperforms the classic XNOR-Net (51.2%) and the previous state-of-the-art CI-BCNN (59.9%).
  • Efficiency: Compared to a full-precision ResNet-18 (69.3% accuracy), DGRL reduces FLOPs by roughly 40x and memory usage by 29x, while trailing by only ~9% in accuracy.
  • Inference Speed: On a 1080TI GPU, the SI shortcut added negligible latency, increasing processing time from 1.875ms to only 2.063ms per image.

Discussion and Insights

The core innovation of this work is the "Residual Perspective." By treating the performance gap as a modelable signal, the authors use tiny auxiliary branches to handle the error that binary layers cannot resolve.

Global channel selection analysis revealed that shallow blocks (closer to the input) benefit most from SI shortcuts. These early layers form the foundation for discriminative features; if they are poorly represented, the deeper network cannot recover. Furthermore, providing intermediate "hints" via distillation smooths the optimization landscape, which is otherwise noisy due to discrete binary constraints.

Conclusion and Future Work

DGRL sets a new benchmark for binary networks, proving that strategic residual compensation and intermediate distillation can drastically improve 1-bit model performance. While a 9% gap remains compared to full-precision models, future work may extend this logic to object detection and semantic segmentation, or explore further quantizing the SI shortcut itself to move toward entirely binary inference.

Summary updated

Abstract

It is challenging to bridge the performance gap between binary convolutional neural network (BCNN) and floating-point CNN (FCNN). This performance gap is mainly caused by the inferior modeling capability and training strategy of BCNN, which leads to substantial residuals in intermediate feature maps between BCNN and FCNN. To minimize the performance gap, we enforce BCNN to produce similar intermediate feature maps with the ones of FCNN. This intuition leads to a more effective training strategy for BCNN, i.e., optimizing each binary convolutional block with blockwise distillation loss derived from FCNN. The goal of minimizing the residuals in intermediate feature maps also motivates us to update the binary convolutional block architecture to facilitate the optimization of blockwise distillation loss. Specifically, a lightweight shortcut branch is inserted into each binary convolutional block to complement residuals at each block. Benefited from its squeeze-and-interaction (SI) structure, this shortcut branch introduces a fraction of parameters, e.g., less than 10% overheads, but effectively boosts the modeling capability of binary convolution blocks in BCNN. Extensive experiments on ImageNet demonstrate the superior performance of our method in both classification efficiency and accuracy, e.g., BCNN trained with our methods achieves the accuracy of 60.45% on ImageNet, better than many state-of-the-art ones.
BibTeX
@article{ye2022distillation,
  author = {Jianming Ye and Shiliang Zhang and Jingdong Wang},
  title = {{Distillation Guided Residual Learning for Binary Convolutional Neural Networks}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2022},
  url = {https://openreview.net/forum?id=N8qOZ4jFct},
}
Figure