MLP-Mixer: An all-MLP Architecture for Vision

Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, Alexey Dosovitskiy

Abstract

Convolutional Neural Networks (CNNs) are the go-to model for computer vision. Recently, attention-based networks, such as the Vision Transformer, have also become popular. In this paper we show that while convolutions and attention are both sufficient for good performance, neither of them are necessary. We present MLP-Mixer, an architecture based exclusively on multi-layer perceptrons (MLPs). MLP-Mixer contains two types of layers: one with MLPs applied independently to image patches (i.e. "mixing" the per-location features), and one with MLPs applied across patches (i.e. "mixing" spatial information). When trained on large datasets, or with modern regularization schemes, MLP-Mixer attains competitive scores on image classification benchmarks, with pre-training and inference cost comparable to state-of-the-art models. We hope that these results spark further research beyond the realms of well established CNNs and Transformers.

Introduction

For years, the computer vision community has relied on specific architectural "priors" or inductive biases. First, Convolutional Neural Networks (CNNs) dominated by assuming that pixels near each other are more related (locality). More recently, Vision Transformers (ViT) took over by using self-attention to model global dependencies. The authors of MLP-Mixer challenge this status quo by asking if these specialized structures are actually required. They propose a competitive model using the most basic building block—the Multi-Layer Perceptron (MLP)—without local constraints or attention mechanisms.

Architecture

The MLP-Mixer architecture is elegantly simple. It processes an image as a sequence of non-overlapping patches, which are linearly projected into "tokens." These tokens are organized in a table where rows represent spatial locations (patches) and columns represent feature channels.

The MLP-Mixer architecture overview, showing the sequence of patches and the interleaved mixing layers.

The core consists of two types of MLP layers:

  1. Token-mixing MLPs: These act on the columns, allowing communication between different spatial locations (patches) while working on each channel independently.
  2. Channel-mixing MLPs: These act on the rows, allowing communication between different features within a single patch, working on each location independently.

By interleaving these operations, the model learns both local features and global spatial relationships. Unlike Transformers, Mixer does not use position embeddings because the token-mixing MLPs are inherently sensitive to input order. The architecture is "isotropic," maintaining consistent dimensionality throughout, unlike the pyramidal structure of traditional CNNs.

Evaluation and Key Insights

The authors evaluated MLP-Mixer across various scales, from ImageNet-1k to the massive JFT-300M dataset. Several key insights emerged from this study:

  • Inductive Bias vs. Scale: MLP-Mixer has less built-in knowledge about image structure than a CNN. While it struggles with scarce data, it scales exceptionally well. As dataset size grows, the gap between Mixer and state-of-the-art models shrinks.
  • Separation of Mixing: Explicitly separating spatial and channel mixing creates a computationally efficient way to capture global context.
  • Computational Efficiency: Relying on basic matrix multiplications makes Mixer highly optimized for modern hardware.
  • Permutation Invariance: Shuffling patches and pixels within them doesn't affect Mixer's performance (once retrained), whereas a CNN's performance would collapse due to its reliance on spatial order.

Results

When pre-trained on JFT-300M, the Mixer-H/14 model achieved 87.94% top-1 accuracy on ImageNet. Furthermore, Mixer-H/14 is significantly faster than its competitors, running 2.5x faster than ViT-H/14 and nearly 2x faster than BiT ResNets with comparable accuracy.

Performance trade-offs of MLP-Mixer compared to Transformers and ResNets in terms of training cost and inference throughput.

The results demonstrate that Mixer benefits more from larger datasets than ResNets, suggesting that for "billion-scale" datasets, simpler architectures may be preferable.

Conclusion

While Mixer requires "big data" to learn spatial features that CNNs get for "free," its simplicity and scaling properties are promising. The MLP-Mixer proves that neither convolutions nor self-attention are strictly necessary for high-performance computer vision. Future work could explore "pyramidal" Mixer designs or apply the concept to domains like NLP where Transformers currently dominate.

Summary updated

Abstract

Convolutional Neural Networks (CNNs) are the go-to model for computer vision. Recently, attention-based networks, such as the Vision Transformer, have also become popular. In this paper we show that while convolutions and attention are both sufficient for good performance, neither of them are necessary. We present MLP-Mixer, an architecture based exclusively on multi-layer perceptrons (MLPs). MLP-Mixer contains two types of layers: one with MLPs applied independently to image patches (i.e. "mixing" the per-location features), and one with MLPs applied across patches (i.e. "mixing" spatial information). When trained on large datasets, or with modern regularization schemes, MLP-Mixer attains competitive scores on image classification benchmarks, with pre-training and inference cost comparable to state-of-the-art models. We hope that these results spark further research beyond the realms of well established CNNs and Transformers.
BibTeX
@inproceedings{tolstikhin2021mlp,
  author = {Ilya O. Tolstikhin and Neil Houlsby and Alexander Kolesnikov and Lucas Beyer and Xiaohua Zhai and Thomas Unterthiner and Jessica Yung and Andreas Steiner and Daniel Keysers and Jakob Uszkoreit and Mario Lucic and Alexey Dosovitskiy},
  title = {{MLP-Mixer: An all-MLP Architecture for Vision}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=y5vVvzO6xY},
}
Figure