Introduction
Binarization—reducing weights and activations to a single bit—is a critical technique for model compression, offering up to 64x savings in FLOPs and 32x in memory. While binary Convolutional Neural Networks (CNNs) have seen significant progress, vision Multi-Layer Perceptrons (vMLPs) have historically struggled to maintain performance under binarization.
The core issue is a "capacity gap": a $1 \times 1$ binary kernel (the building block of MLPs) has significantly less representational power than the $k \times k$ binary kernels used in CNNs. Simply increasing channel counts to compensate leads to a quadratic explosion in computational cost. BiMLP addresses this by redesigning the vMLP architecture to maximize representation power without sacrificing efficiency through multi-branch structures and optimized connectivity.
Methodology
The authors propose three primary architectural innovations to create BiMLP:
Multi-branch Binary MLP (MBB) Block
Instead of a single wide layer, BiMLP utilizes multiple parallel branches to merge outputs. This enriches the feature representation at a lower computational cost than simply widening the network. The design features two types of blocks that alternate between spatial mixing and channel mixing to ensure robust feature extraction.
Universal Shortcut (Uni-shortcut)
Standard residual connections require matching input and output dimensions. In MLPs, channel counts often change between stages. The Uni-shortcut handles these transitions by either averaging or repeating channels, ensuring effective information flow across the entire network regardless of shape changes.
Optimized Downsampling
Traditional $3 \times 3$ stride-2 convolutions are computationally expensive and difficult to binarize effectively. BiMLP replaces these with a combination of a $1 \times 1$ convolution and multi-scale max-pooling (using $2 \times 2$ to $7 \times 7$ kernels). This strategy maintains the receptive field while achieving a 41.1% reduction in downsampling complexity with only a negligible (0.3%) drop in accuracy.
Key Results and Insights
BiMLP was evaluated on the ImageNet-1k benchmark, setting a new standard for binary architectures:
- BiMLP-S: Achieves 70.0% top-1 accuracy with only 0.156G OPs, outperforming MeliusNet-42 by 0.8% while using less than half the operations.
- BiMLP-M: Reaches 72.7% top-1 accuracy with 0.188G OPs. This is 1.3% better than the state-of-the-art ReActNet-C while being 12.1% more efficient.
The results highlight that the failure of previous binary MLPs was not due to the MLP concept itself, but rather the inherent weakness of $1 \times 1$ binarization. By adopting a branching strategy over simple widening, BiMLP recovers lost representation power efficiently.
Conclusion and Future Work
BiMLP demonstrates that specialized architectural design can unlock the potential of 1-bit vision MLPs, allowing them to outperform advanced binary CNNs. While the results are promising, future work remains in binarizing the first, last, and downsampling layers—which currently remain in full precision—and evaluating the scalability of these architectures to larger model sizes. Real-world speedups will also depend on the continued development of specialized hardware capable of efficiently executing binary operations.
Summary updated