Transformers as In-Context Mesa Optimizers

This direction is split into other directions: linear task simplification. Its last version stays readable and citable here.

Version 1 of 182 papers82 conceptsfrozen 2026-10-04

In-context learning (ICL) is the most striking emergent property of large language models: the ability to solve a new task from a few examples in a prompt without any weight updates. For years, this was treated as a "black box" of pattern matching. However, a growing body of research suggests that Transformers are not just retrieving patterns; they are internally executing optimization algorithms.

This perspective, often called the mesa-optimization hypothesis, posits that the forward pass of a Transformer implements a mini-training loop, effectively "learning" the task provided in the prompt on the fly.

The Forward Pass as Gradient Descent

The foundational intuition for this idea comes from Johannes von Oswald and his colleagues in Transformers Learn In-Context by Gradient Descent. They demonstrated that the mathematical structure of a linear self-attention layer can be mapped directly onto a single step of Gradient Descent (GD).

In this framing, the Transformer's weights act as a "meta-optimizer." During pretraining, the model doesn't just learn to predict the next token; it learns a set of weights that, when applied to new data in a prompt, execute an update rule. The "model" being trained is represented by the activations flowing through the layers, and the "loss" is the error on the provided in-context examples.

Refining the Algorithm

If a single layer is a single step of GD, then a deep Transformer is a multi-step optimizer. But standard GD is often slow and inefficient. Recent work suggests that Transformers learn more sophisticated versions of this internal algorithm:

Scaling to Non-Linearity

Most early proofs of mesa-optimization focused on linear regression. However, real-world language is non-linear. Xiang Cheng and Suvrit Sra bridged this gap in Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context. They showed that Transformers can implement Functional Gradient Descent (FGD), allowing them to optimize over a space of functions rather than just linear weights.

This is further supported by Zhaiming Shen, who demonstrated that Transformers act as kernel regression machines (Understanding In-Context Learning on Structured Manifolds). By treating the attention mechanism as a learned kernel, the model can achieve optimal prediction rates for complex, non-linear functions by identifying the underlying low-dimensional manifold of the data.

Architectural Universality

Is this "internal optimization" unique to the Transformer's attention mechanism? Evidence suggests it is a more general property of sequence models trained on diverse tasks.

Jiarui Jiang and Wei Huang showed in Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression that Mamba—a state-space model (SSM) with a very different architecture—also converges to an internal implementation of online gradient descent. This suggests that "learning to optimize" is a fundamental strategy that high-capacity models use to solve the problem of in-context adaptation.

The Limits of the Analogy

While the mesa-optimization framework provides a powerful lens, researchers like Miao Lu and Difan Zou caution that linear regression is only a proxy (Towards Theoretical Understanding of Transformer Test-Time Computing). Natural language involves discrete symbols, hierarchical logic, and vast amounts of pretrained world knowledge that simple gradient descent on a prompt cannot fully explain.

The current frontier lies in understanding how these internal "optimizers" interact with the model's "static" knowledge—how the model decides when to use a "statistical shortcut" versus when to induce a new functional rule (Function Induction and Task Generalization).

Papers in this version

Showing 60 of 82 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_transformers_as_in_context_mesa_optimize_1145f50c,
  title = {Transformers as In-Context Mesa Optimizers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/transformers-as-in-context-mesa-optimizers/rdr_1145f50c2cc04c459ea1ff768beac676},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_transformers_as_in_context_mesa_optimize_1145f50c_v1,
  title = {Transformers as In-Context Mesa Optimizers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/transformers-as-in-context-mesa-optimizers/rdr_1145f50c2cc04c459ea1ff768beac676/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}