Transformers as In-Context Second-Order Optimizers

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 115 papers24 conceptsfrozen 2026-10-04

When a Transformer solves a new task using only a few examples in its prompt—a phenomenon known as In-Context Learning (ICL)—it is effectively performing an optimization process on the fly. For several years, the prevailing theory was that the Transformer’s forward pass approximates Gradient Descent (GD), iteratively refining its internal representations to minimize a loss function.

However, recent research suggests that Transformers are far more efficient than simple gradient descent. Instead of taking small, first-order steps, these models appear to implement second-order optimization methods, such as Newton’s Method. By accounting for the "curvature" of the data, Transformers achieve convergence rates that are exponentially faster than first-order methods.

The Efficiency Gap

The primary limitation of Gradient Descent is its linear convergence rate. In complex landscapes, GD can struggle with ill-conditioned data, requiring many iterations to find a solution. If Transformers were merely emulating GD, they would require a vast number of layers to solve even simple regression tasks.

Empirical evidence suggests they don't. Deqing Fu, Tian-Qi Chen, and Vatsal Sharan have shown that Transformers achieve superlinear convergence rates for in-context linear regression. In their work, Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression, they demonstrate that the model’s performance improves much faster across layers than a first-order algorithm would allow. A single Transformer layer can represent the progress of multiple iterations of a more sophisticated algorithm.

Emulating Newton’s Method

Why does this happen? The architecture of the Transformer—specifically the attention mechanism—is mathematically capable of performing the matrix operations required for second-order optimization.

While GD only uses the gradient (the slope), second-order methods like Newton’s Method use the Hessian (the curvature). This allows the algorithm to jump much closer to the optimum in fewer steps. Angeliki Giannou, Dimitris Papailiopoulos, and Jason D. Lee proved in How Well Can Transformers Emulate In-context Newton's Method? that Transformers can be constructed to solve logistic regression with quadratic convergence. They show that the forward pass can explicitly compute the necessary pre-conditioning matrices to navigate the optimization landscape efficiently.

Similarly, Max Vladymyrov and Johannes von Oswald argue in Linear Transformers are Versatile In-Context Learners that even simplified "Linear Transformers" can implement these internal second-order updates, allowing them to solve regression problems with high precision using very few layers.

Implicit Discovery of Numerical Algorithms

Perhaps most surprising is that Transformers aren't just capable of these algorithms; they "invent" them during training. When trained on matrix completion tasks, Patrick Lutz, Hadi Daneshmand, and colleagues found that linear transformers discover a unified numerical algorithm they call EAGLE.

As detailed in Linear Transformers Implicitly Discover Unified Numerical Algorithms, this discovered algorithm mimics Newton-Schulz iterations. It automatically adapts to hardware constraints and achieves quadratic convergence, solving ill-conditioned problems that would baffle standard gradient-based approaches.

From Internal Logic to External Training

The realization that Transformers are "natural" second-order optimizers has implications for how we train them. If the model's internal representation-refinement process acts like a pre-conditioned gradient descent—as suggested by Lav R. Varshney and Xinbo Wu in A Meta-Learning Perspective on Transformers for Causal Language Modeling—it follows that our external optimizers should leverage similar curvature information.

Researchers are already extending this intuition to the training phase:

  • Aida: Guoqiang Zhang and Kenta Niwa developed Aida, which generalizes AdamW by incorporating higher-order magnitude scaling. This relaxation of stability constraints leads to better validation performance in Transformer architectures.
  • Fast Natural Gradient Descent (FNGD): To overcome the computational cost of second-order methods, Ce Zhu and Yipeng Liu propose Inverse-Free Fast Natural Gradient Descent, which approximates the curvature of the loss landscape without the expensive matrix inversions usually required.

Summary of Intuition

The shift from viewing Transformers as first-order learners to second-order optimizers explains the "unreasonable" effectiveness of in-context learning. By internally approximating algorithms like Newton's Method, Transformers don't just follow the gradient; they reshape the optimization space itself, allowing them to learn from a handful of examples in a single forward pass.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_transformers_as_in_context_second_order__0b8b02b2,
  title = {Transformers as In-Context Second-Order Optimizers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/transformers-as-in-context-second-order-optimizers/rdr_0b8b02b26e0d442ea29886a25e1004bc},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_transformers_as_in_context_second_order__0b8b02b2_v1,
  title = {Transformers as In-Context Second-Order Optimizers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/transformers-as-in-context-second-order-optimizers/rdr_0b8b02b26e0d442ea29886a25e1004bc/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}