Abstract
Binary neural networks (BNNs) are advantageous in performance and memory footprint but suffer from low accuracy due to their limited expression capability. Recent works have tried to enhance the accuracy of BNNs via a gradient-based search algorithm and showed promising results. However, the mixture of architecture search and binarization induce the instability of the search process, resulting in convergence to the suboptimal point. To address this issue, we propose a BNN architecture search framework with bimodal parameter perturbation. The bimodal parameter perturbation can improve the stability of gradient-based architecture search by reducing the sharpness of the loss surface along both weight and architecture parameter axes. In addition, we refine the inverted bottleneck convolution block for having robustness with BNNs. The synergy of the refined space and the stabilized search process allows us to find out the accurate BNNs with high computation efficiency. Experimental results show that our framework finds the best architecture on CIFAR-100 and ImageNet datasets in the existing search space for BNNs. We also tested our framework on another search space based on the inverted bottleneck convolution block, and the selected BNN models using our approach achieved the highest accuracy on both datasets with a much smaller number of equivalent operations than previous works.
Introduction
Binary Neural Networks (BNNs) represent weights and activations using a single bit, offering massive savings in memory and power—ideal for edge devices. However, they usually suffer from a significant "accuracy gap" compared to full-precision models. To close this gap, researchers have turned to Differentiable Architecture Search (DARTS) to automatically discover optimal network structures. However, combining NAS with binarization is notoriously unstable. The search often "collapses," choosing suboptimal operations because the loss landscape is too sharp. When the algorithm selects a final discrete architecture from a relaxed search space, performance often plummets because the model was not robust to structural discretization.
Bimodal Parameter Perturbation
The core innovation of this work is Bimodal Parameter Perturbation. During the search phase, the algorithm does not merely seek the lowest loss; it actively searches for the "flattest" regions of the loss landscape. This is achieved by adding random uniform noise to both the weights ($W$) and the architecture parameters ($A$).
By perturbing both axes, the method ensures that the chosen operations remain effective even after the architecture is discretized and weights are binarized. Mathematically, this noise acts as a regularizer for the Hessian matrix, penalizing sharp minima and rewarding flat ones that generalize better. This dual-axis perturbation prevents the "collapsing cell" problem where DARTS favors simple skip-connections over more complex, effective operations.
Refined Binary Search Space
In addition to the search algorithm, the authors refined the Inverted Bottleneck (MBConv) block to better suit binary operations. Standard MBConv blocks often perform poorly when binarized. The proposed modifications include:
- ReAct Sign and PReLU: Utilizing learnable thresholds to increase the network's expressive capability.
- Partial Activation Duplication: Maintaining identity connections even during channel dimension changes to prevent signal loss.
- Group Binary Convolutions: Replacing depthwise convolutions with more hardware-friendly and binarization-efficient group convolutions.
Experimental Results
The framework was evaluated on CIFAR-100 and ImageNet-1K. On ImageNet-1K, the proposed method achieved a Top-1 accuracy of 68.2%, outperforming previous state-of-the-art NAS-based BNNs such as BATS (66.1%) and BNAS-F (59.0%). Remarkably, this was achieved with only 89.52M equivalent operations, significantly lower than competitors like BNAS-H (1275M). Furthermore, the resulting models are genuinely "lean," with only 24% of equivalent operations being floating-point, primarily in shortcuts.
Conclusion
This paper demonstrates that the instability inherent in BNN architecture search can be overcome by searching for flat minima. The synergy of bimodal parameter perturbation and a refined, robust search space enables the discovery of BNNs that achieve high accuracy with exceptional computational efficiency. While noise injection adds a modest 20% overhead to search time, the resulting gains in model robustness and performance are substantial.
Summary updated