Abstract
In-context learning (ICL) is widely considered a unique capability of Transformer architectures. This study challenges this assumption by demonstrating that simple Multi-Layer Perceptrons (MLPs) and MLP-Mixers can learn in-context as effectively as Transformers on synthetic regression and classification tasks. Furthermore, MLPs are shown to outperform Transformers on specific relational reasoning tasks, suggesting that the attention mechanism is not a strict requirement for emergent ICL behavior.
Introduction
In-context learning—the ability of a model to solve a new task at inference time by observing a few examples—is the core mechanism behind the success of Large Language Models (LLMs). Because ICL emerged alongside the Transformer, many researchers assumed that the attention mechanism was the essential ingredient for this behavior. This paper investigates whether "simpler" architectures like MLPs have been unfairly overlooked by stripping away the complexity of natural language and focusing on controlled, synthetic tasks.
Methods
The researchers conducted a head-to-head comparison between three primary architectures: vanilla MLPs, MLP-Mixers, and Transformers. To ensure a fair comparison, models were evaluated based on total compute budget (measured in PFLOPs) rather than parameter count, accounting for varying training efficiencies. The study utilized a suite of synthetic benchmarks, including ICL regression, ICL classification, and relational reasoning tasks such as Match-to-Sample and Oddball puzzles.
Key Insights and Results
The experimental results demonstrate that given sufficient data diversity and compute, vanilla MLPs and MLP-Mixers reach the same optimal performance (Bayes optimality) as Transformers on standard ICL tasks.
The IWL-to-ICL Phase Transition
All three architectures exhibit a distinct phase transition. When trained on low-diversity data, models employ In-Weight Learning (IWL), memorizing specific tasks. As data diversity increases, they switch to In-Context Learning (ICL), learning the underlying algorithm to solve tasks from the prompt. Transformers appear to transition at slightly lower levels of data diversity, suggesting a helpful inductive bias, though not a necessary one.
Relational Reasoning and Efficiency
In a surprising finding, MLPs significantly outperformed Transformers on relational tasks, such as identifying "oddballs." Transformers struggled to generalize to out-of-distribution (OOD) examples, whereas MLPs handled them with ease. Additionally, on simple non-ICL tasks, Transformers were orders of magnitude less efficient than MLPs, sometimes requiring "chunking" of inputs to match MLP performance.
Context Length and Architecture
Vanilla MLPs face a limitation with long context lengths because they flatten the input sequence. However, MLP-Mixers overcome this by alternating MLPs across token and feature dimensions, maintaining near-optimal performance as context scales. This suggests that structural modifications to MLPs can mitigate their inherent weaknesses in sequence processing.
Discussion
While these results are provocative, the authors acknowledge that the tasks were synthetic and low-dimensional. The scalability of MLPs to high-dimensional, messy data like natural language at the scale of state-of-the-art LLMs remains an open question. Nonetheless, this work encourages the exploration of "all-MLP" alternatives and suggests that the "Bitter Lesson"—that general-purpose architectures eventually overtake specialized ones with scale—may apply to the domain of in-context learning.
Summary updated