In-context learning (ICL) is the most striking emergent property of large language models: the ability to solve a new task from a few examples in a prompt without any weight updates. For years, this was treated as a "black box" of pattern matching. However, a growing body of research suggests that Transformers are not just retrieving patterns; they are internally executing optimization algorithms.
This perspective, often called the mesa-optimization hypothesis, posits that the forward pass of a Transformer implements a mini-training loop, effectively "learning" the task provided in the prompt on the fly.
The Forward Pass as Gradient Descent
The foundational intuition for this idea comes from Johannes von Oswald and his colleagues in Transformers Learn In-Context by Gradient Descent. They demonstrated that the mathematical structure of a linear self-attention layer can be mapped directly onto a single step of Gradient Descent (GD).
In this framing, the Transformer's weights act as a "meta-optimizer." During pretraining, the model doesn't just learn to predict the next token; it learns a set of weights that, when applied to new data in a prompt, execute an update rule. The "model" being trained is represented by the activations flowing through the layers, and the "loss" is the error on the provided in-context examples.
Refining the Algorithm
If a single layer is a single step of GD, then a deep Transformer is a multi-step optimizer. But standard GD is often slow and inefficient. Recent work suggests that Transformers learn more sophisticated versions of this internal algorithm:
- Preconditioning: Kwangjun Ahn and Suvrit Sra proved in Transformers learn to implement preconditioned gradient descent for in-context learning that models don't just run basic GD; they learn to "precondition" the updates, automatically adapting to the noise and distribution of the prompt data.
- Second-Order Convergence: Deqing Fu and Vatsal Sharan argue in Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression that Transformers can even emulate iterative Newton-style methods, achieving much faster convergence than simple first-order gradient steps.
- Task Allocation: In multi-head architectures, this optimization is distributed. Heejune Sheen and Zhuoran Yang found that different attention heads spontaneously specialize in different sub-tasks, a phenomenon they call "task allocation" (Training Dynamics of Multi-Head Softmax Attention for In-Context Learning).
Scaling to Non-Linearity
Most early proofs of mesa-optimization focused on linear regression. However, real-world language is non-linear. Xiang Cheng and Suvrit Sra bridged this gap in Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context. They showed that Transformers can implement Functional Gradient Descent (FGD), allowing them to optimize over a space of functions rather than just linear weights.
This is further supported by Zhaiming Shen, who demonstrated that Transformers act as kernel regression machines (Understanding In-Context Learning on Structured Manifolds). By treating the attention mechanism as a learned kernel, the model can achieve optimal prediction rates for complex, non-linear functions by identifying the underlying low-dimensional manifold of the data.
Architectural Universality
Is this "internal optimization" unique to the Transformer's attention mechanism? Evidence suggests it is a more general property of sequence models trained on diverse tasks.
Jiarui Jiang and Wei Huang showed in Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression that Mamba—a state-space model (SSM) with a very different architecture—also converges to an internal implementation of online gradient descent. This suggests that "learning to optimize" is a fundamental strategy that high-capacity models use to solve the problem of in-context adaptation.
The Limits of the Analogy
While the mesa-optimization framework provides a powerful lens, researchers like Miao Lu and Difan Zou caution that linear regression is only a proxy (Towards Theoretical Understanding of Transformer Test-Time Computing). Natural language involves discrete symbols, hierarchical logic, and vast amounts of pretrained world knowledge that simple gradient descent on a prompt cannot fully explain.
The current frontier lies in understanding how these internal "optimizers" interact with the model's "static" knowledge—how the model decides when to use a "statistical shortcut" versus when to induce a new functional rule (Function Induction and Task Generalization).