Searching for Robust Binary Neural Networks via Bimodal Parameter Perturbation

Daehyun Ahn, Hyungjun Kim, Taesu Kim, Eunhyeok Park, Jae-Joon KimWACV 2023 · OpenReview

Abstract

Binary neural networks (BNNs) are advantageous in performance and memory footprint but suffer from low accuracy due to their limited expression capability. Recent works have tried to enhance the accuracy of BNNs via a gradient-based search algorithm and showed promising results. However, the mixture of architecture search and binarization induce the instability of the search process, resulting in convergence to the suboptimal point. To address this issue, we propose a BNN architecture search framework with bimodal parameter perturbation. The bimodal parameter perturbation can improve the stability of gradient-based architecture search by reducing the sharpness of the loss surface along both weight and architecture parameter axes. In addition, we refine the inverted bottleneck convolution block for having robustness with BNNs. The synergy of the refined space and the stabilized search process allows us to find out the accurate BNNs with high computation efficiency. Experimental results show that our framework finds the best architecture on CIFAR-100 and ImageNet datasets in the existing search space for BNNs. We also tested our framework on another search space based on the inverted bottleneck convolution block, and the selected BNN models using our approach achieved the highest accuracy on both datasets with a much smaller number of equivalent operations than previous works.

Introduction

Binary Neural Networks (BNNs) represent weights and activations using a single bit, offering massive savings in memory and power—ideal for edge devices. However, they usually suffer from a significant "accuracy gap" compared to full-precision models. To close this gap, researchers have turned to Differentiable Architecture Search (DARTS) to automatically discover optimal network structures. However, combining NAS with binarization is notoriously unstable. The search often "collapses," choosing suboptimal operations because the loss landscape is too sharp. When the algorithm selects a final discrete architecture from a relaxed search space, performance often plummets because the model was not robust to structural discretization.

Bimodal Parameter Perturbation

The core innovation of this work is Bimodal Parameter Perturbation. During the search phase, the algorithm does not merely seek the lowest loss; it actively searches for the "flattest" regions of the loss landscape. This is achieved by adding random uniform noise to both the weights ($W$) and the architecture parameters ($A$).

Visualizing the loss landscape on the architecture (A) and weight (W) axes. Perturbation forces the search toward flat minima ($A^{*,flat}$), which are more robust to discretization than sharp minima ($A^{*,sharp}$).

By perturbing both axes, the method ensures that the chosen operations remain effective even after the architecture is discretized and weights are binarized. Mathematically, this noise acts as a regularizer for the Hessian matrix, penalizing sharp minima and rewarding flat ones that generalize better. This dual-axis perturbation prevents the "collapsing cell" problem where DARTS favors simple skip-connections over more complex, effective operations.

Refined Binary Search Space

In addition to the search algorithm, the authors refined the Inverted Bottleneck (MBConv) block to better suit binary operations. Standard MBConv blocks often perform poorly when binarized. The proposed modifications include:

  • ReAct Sign and PReLU: Utilizing learnable thresholds to increase the network's expressive capability.
  • Partial Activation Duplication: Maintaining identity connections even during channel dimension changes to prevent signal loss.
  • Group Binary Convolutions: Replacing depthwise convolutions with more hardware-friendly and binarization-efficient group convolutions.
The proposed Normal and Reduction blocks. The design incorporates ReAct Sign, PReLU, and a concatenation-based identity path in the reduction block to preserve signal integrity.

Experimental Results

The framework was evaluated on CIFAR-100 and ImageNet-1K. On ImageNet-1K, the proposed method achieved a Top-1 accuracy of 68.2%, outperforming previous state-of-the-art NAS-based BNNs such as BATS (66.1%) and BNAS-F (59.0%). Remarkably, this was achieved with only 89.52M equivalent operations, significantly lower than competitors like BNAS-H (1275M). Furthermore, the resulting models are genuinely "lean," with only 24% of equivalent operations being floating-point, primarily in shortcuts.

Conclusion

This paper demonstrates that the instability inherent in BNN architecture search can be overcome by searching for flat minima. The synergy of bimodal parameter perturbation and a refined, robust search space enables the discovery of BNNs that achieve high accuracy with exceptional computational efficiency. While noise injection adds a modest 20% overhead to search time, the resulting gains in model robustness and performance are substantial.

Summary updated

Abstract

Binary neural networks (BNNs) are advantageous in performance and memory footprint but suffer from low accuracy due to their limited expression capability. Recent works have tried to enhance the accuracy of BNNs via a gradient-based search algorithm and showed promising results. However, the mixture of architecture search and binarization induce the instability of the search process, resulting in convergence to the suboptimal point. To address this issue, we propose a BNN architecture search framework with bimodal parameter perturbation. The bimodal parameter perturbation can improve the stability of gradient-based architecture search by reducing the sharpness of the loss surface along both weight and architecture parameter axes. In addition, we refine the inverted bottleneck convolution block for having robustness with BNNs. The synergy of the refined space and the stabilized search process allows us to find out the accurate BNNs with high computation efficiency. Experimental results show that our framework finds the best architecture on CIFAR-100 and ImageNet datasets in the existing search space for BNNs. We also tested our framework on another search space based on the inverted bottleneck convolution block, and the selected BNN models using our approach achieved the highest accuracy on both datasets with a much smaller number of equivalent operations than previous works.
BibTeX
@inproceedings{ahn2023searching,
  author = {Daehyun Ahn and Hyungjun Kim and Taesu Kim and Eunhyeok Park and Jae-Joon Kim},
  title = {{Searching for Robust Binary Neural Networks via Bimodal Parameter Perturbation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=5fx42N53_s},
}
Figure