MeliusNet: Can Binary Neural Networks Achieve MobileNet-level Accuracy?

Joseph Bethge, Christian Bartz, Haojin Yang, Ying Chen, Christoph MeinelarXiv 2020 · OpenReview

Abstract

Binary Neural Networks (BNNs) are neural networks which use binary weights and activations instead of the typical 32-bit floating point values. They have reduced model sizes and allow for efficient inference on mobile or embedded devices with limited power and computational resources. However, the binarization of weights and activations leads to feature maps of lower quality and lower capacity and thus a drop in accuracy compared to traditional networks. Previous work has increased the number of channels or used multiple binary bases to alleviate these problems. In this paper, we instead present an architectural approach: MeliusNet. It consists of alternating a DenseBlock, which increases the feature capacity, and our proposed ImprovementBlock, which increases the feature quality. Experiments on the ImageNet dataset demonstrate the superior performance of our MeliusNet over a variety of popular binary architectures with regards to both computation savings and accuracy. Furthermore, with our method we trained BNN models, which for the first time can match the accuracy of the popular compact network MobileNet-v1 in terms of model size, number of operations and accuracy.

Introduction

Binary Neural Networks (BNNs) represent a significant frontier in edge AI, offering the potential for up to a 50x speedup and 99% energy savings by using 1-bit weights and activations. However, the primary hurdle for BNNs has been accuracy. Binarization typically results in low-quality, noisy features and low representational capacity. MeliusNet addresses these challenges through a novel architectural approach that balances feature capacity and quality, finally bridging the performance gap between ultra-efficient 1-bit networks and popular compact models like MobileNet-v1.

Methods: The MeliusNet Architecture

The core of MeliusNet is an alternating structure of two specialized building blocks designed to overcome the inherent limitations of binarization.

Capacity and Quality Blocks

To address the issues of low feature capacity and quality in BNNs, MeliusNet utilizes a dual-block approach:

  • DenseBlock (Capacity): Drawing inspiration from DenseNet, this block concatenates 64 newly computed binary channels to the existing feature map. This expansion directly increases the network's "capacity" to represent complex patterns.
  • ImprovementBlock (Quality): This block focuses on refining the newly added features. It applies a residual connection specifically to the new channels (indicated by the green area in the diagram below), improving their "quality" without significantly increasing the computational load.
MeliusNet building blocks and macro-architecture. (a) illustrates the DenseBlock and ImprovementBlock mechanisms for increasing capacity and quality. (b) shows the full network structure including transition layers.

Grouped Stem Optimization

A significant portion of computational overhead in BNNs often resides in the first 32-bit layer. MeliusNet addresses this by replacing the standard 7x7 convolution with a series of 3x3 grouped convolutions. This "Grouped Stem" reduces the 32-bit computational overhead by over 40%, significantly improving overall efficiency while maintaining or even increasing accuracy.

Key Insights

  • The Quality-Capacity Balance: MeliusNet's alternating design ensures that as the network gains "room" to learn (capacity), the features remain precise (quality). The authors argue that without this balance, networks either remain noisy or lack the complexity to learn advanced representations.
  • Training Stability: The authors found that the RAdam optimizer is particularly effective for BNNs. It better handles the frequent sign-flipping of binary weights during early training stages compared to standard SGD, leading to more stable convergence.
  • Architecture Specificity: Interestingly, MeliusNet is specifically tailored for the constraints of 1-bit logic; experiments showed it does not provide the same benefits to standard 32-bit networks, suggesting it is a specialized solution for quantized domains.

Results

Evaluated on the ImageNet (ILSVRC12) dataset, MeliusNet sets a new benchmark for what 1-bit networks can achieve:

  • MobileNet-level Accuracy: MeliusNet-42 achieved 69.2% Top-1 accuracy, matching MobileNet-v1 0.75. MeliusNet-59 reached 71.0%, matching the full MobileNet-v1 1.0.
  • Efficiency: MeliusNet-22 outperforms previous state-of-the-art BNNs like Bi-RealNet and BinaryDenseNet while maintaining a compact memory footprint (~4MB).
  • Stem Performance: The grouped stem significantly reduced total operations (e.g., from 1.63 to 1.14 x 10^8 OPs in a ResNet baseline) while simultaneously increasing accuracy.

Conclusion

MeliusNet demonstrates that through careful architectural design, Binary Neural Networks can finally match the performance of popular compact full-precision models like MobileNet. While it currently lags behind more advanced models like MobileNet-v2 or v3, the combination of MeliusNet's blocks with techniques like knowledge distillation or multiple binary bases offers a promising path toward even more efficient and accurate edge AI.

Summary updated

Abstract

Binary Neural Networks (BNNs) are neural networks which use binary weights and activations instead of the typical 32-bit floating point values. They have reduced model sizes and allow for efficient inference on mobile or embedded devices with limited power and computational resources. However, the binarization of weights and activations leads to feature maps of lower quality and lower capacity and thus a drop in accuracy compared to traditional networks. Previous work has increased the number of channels or used multiple binary bases to alleviate these problems. In this paper, we instead present an architectural approach: MeliusNet. It consists of alternating a DenseBlock, which increases the feature capacity, and our proposed ImprovementBlock, which increases the feature quality. Experiments on the ImageNet dataset demonstrate the superior performance of our MeliusNet over a variety of popular binary architectures with regards to both computation savings and accuracy. Furthermore, with our method we trained BNN models, which for the first time can match the accuracy of the popular compact network MobileNet-v1 in terms of model size, number of operations and accuracy. Our code is published online at https://github.com/hpi-xnor/BMXNet-v2
BibTeX
@misc{bethge2020meliusnet,
  author = {Joseph Bethge and Christian Bartz and Haojin Yang and Ying Chen and Christoph Meinel},
  title = {{MeliusNet: Can Binary Neural Networks Achieve MobileNet-level Accuracy?}},
  year = {2020},
  eprint = {2001.05936},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.05936},
}
Figure