BiMLP: Compact Binary Architectures for Vision Multi-Layer Perceptrons

Yixing Xu, Xinghao Chen, Yunhe Wang

Introduction

Binarization—reducing weights and activations to a single bit—is a critical technique for model compression, offering up to 64x savings in FLOPs and 32x in memory. While binary Convolutional Neural Networks (CNNs) have seen significant progress, vision Multi-Layer Perceptrons (vMLPs) have historically struggled to maintain performance under binarization.

The core issue is a "capacity gap": a $1 \times 1$ binary kernel (the building block of MLPs) has significantly less representational power than the $k \times k$ binary kernels used in CNNs. Simply increasing channel counts to compensate leads to a quadratic explosion in computational cost. BiMLP addresses this by redesigning the vMLP architecture to maximize representation power without sacrificing efficiency through multi-branch structures and optimized connectivity.

Methodology

The authors propose three primary architectural innovations to create BiMLP:

Multi-branch Binary MLP (MBB) Block

Instead of a single wide layer, BiMLP utilizes multiple parallel branches to merge outputs. This enriches the feature representation at a lower computational cost than simply widening the network. The design features two types of blocks that alternate between spatial mixing and channel mixing to ensure robust feature extraction.

The proposed Multi-branch binary MLP (MBB) blocks compared to standard vision MLP blocks.

Universal Shortcut (Uni-shortcut)

Standard residual connections require matching input and output dimensions. In MLPs, channel counts often change between stages. The Uni-shortcut handles these transitions by either averaging or repeating channels, ensuring effective information flow across the entire network regardless of shape changes.

Optimized Downsampling

Traditional $3 \times 3$ stride-2 convolutions are computationally expensive and difficult to binarize effectively. BiMLP replaces these with a combination of a $1 \times 1$ convolution and multi-scale max-pooling (using $2 \times 2$ to $7 \times 7$ kernels). This strategy maintains the receptive field while achieving a 41.1% reduction in downsampling complexity with only a negligible (0.3%) drop in accuracy.

Key Results and Insights

BiMLP was evaluated on the ImageNet-1k benchmark, setting a new standard for binary architectures:

  • BiMLP-S: Achieves 70.0% top-1 accuracy with only 0.156G OPs, outperforming MeliusNet-42 by 0.8% while using less than half the operations.
  • BiMLP-M: Reaches 72.7% top-1 accuracy with 0.188G OPs. This is 1.3% better than the state-of-the-art ReActNet-C while being 12.1% more efficient.
Accuracy vs. OPs of different binary models on ImageNet-1k. BiMLP (red) sits at the top-left frontier.

The results highlight that the failure of previous binary MLPs was not due to the MLP concept itself, but rather the inherent weakness of $1 \times 1$ binarization. By adopting a branching strategy over simple widening, BiMLP recovers lost representation power efficiently.

Conclusion and Future Work

BiMLP demonstrates that specialized architectural design can unlock the potential of 1-bit vision MLPs, allowing them to outperform advanced binary CNNs. While the results are promising, future work remains in binarizing the first, last, and downsampling layers—which currently remain in full precision—and evaluating the scalability of these architectures to larger model sizes. Real-world speedups will also depend on the continued development of specialized hardware capable of efficiently executing binary operations.

Summary updated

Abstract

This paper studies the problem of designing compact binary architectures for vision multi-layer perceptrons (MLPs). We provide extensive analysis on the difficulty of binarizing vision MLPs and find that previous binarization methods perform poorly due to limited capacity of binary MLPs. In contrast with the traditional CNNs that utilizing convolutional operations with large kernel size, fully-connected (FC) layers in MLPs can be treated as convolutional layers with kernel size $1\times1$. Thus, the representation ability of the FC layers will be limited when being binarized, and places restrictions on the capability of spatial mixing and channel mixing on the intermediate features. To this end, we propose to improve the performance of binary MLP (BiMLP) model by enriching the representation ability of binary FC layers. We design a novel binary block that contains multiple branches to merge a series of outputs from the same stage, and also a universal shortcut connection that encourages the information flow from the previous stage. The downsampling layers are also carefully designed to reduce the computational complexity while maintaining the classification performance. Experimental results on benchmark dataset ImageNet-1k demonstrate the effectiveness of the proposed BiMLP models, which achieve state-of-the-art accuracy compared to prior binary CNNs.The MindSpore code is available at \url{https://gitee.com/mindspore/models/tree/master/research/cv/BiMLP}.
BibTeX
@inproceedings{xu2022bimlp,
  author = {Yixing Xu and Xinghao Chen and Yunhe Wang},
  title = {{BiMLP: Compact Binary Architectures for Vision Multi-Layer Perceptrons}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=7jxsM0E2xPm},
}
Figure