When a Transformer solves a new task using only a few examples in its prompt—a phenomenon known as In-Context Learning (ICL)—it is effectively performing an optimization process on the fly. For several years, the prevailing theory was that the Transformer’s forward pass approximates Gradient Descent (GD), iteratively refining its internal representations to minimize a loss function.
However, recent research suggests that Transformers are far more efficient than simple gradient descent. Instead of taking small, first-order steps, these models appear to implement second-order optimization methods, such as Newton’s Method. By accounting for the "curvature" of the data, Transformers achieve convergence rates that are exponentially faster than first-order methods.
The Efficiency Gap
The primary limitation of Gradient Descent is its linear convergence rate. In complex landscapes, GD can struggle with ill-conditioned data, requiring many iterations to find a solution. If Transformers were merely emulating GD, they would require a vast number of layers to solve even simple regression tasks.
Empirical evidence suggests they don't. Deqing Fu, Tian-Qi Chen, and Vatsal Sharan have shown that Transformers achieve superlinear convergence rates for in-context linear regression. In their work, Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression, they demonstrate that the model’s performance improves much faster across layers than a first-order algorithm would allow. A single Transformer layer can represent the progress of multiple iterations of a more sophisticated algorithm.
Emulating Newton’s Method
Why does this happen? The architecture of the Transformer—specifically the attention mechanism—is mathematically capable of performing the matrix operations required for second-order optimization.
While GD only uses the gradient (the slope), second-order methods like Newton’s Method use the Hessian (the curvature). This allows the algorithm to jump much closer to the optimum in fewer steps. Angeliki Giannou, Dimitris Papailiopoulos, and Jason D. Lee proved in How Well Can Transformers Emulate In-context Newton's Method? that Transformers can be constructed to solve logistic regression with quadratic convergence. They show that the forward pass can explicitly compute the necessary pre-conditioning matrices to navigate the optimization landscape efficiently.
Similarly, Max Vladymyrov and Johannes von Oswald argue in Linear Transformers are Versatile In-Context Learners that even simplified "Linear Transformers" can implement these internal second-order updates, allowing them to solve regression problems with high precision using very few layers.
Implicit Discovery of Numerical Algorithms
Perhaps most surprising is that Transformers aren't just capable of these algorithms; they "invent" them during training. When trained on matrix completion tasks, Patrick Lutz, Hadi Daneshmand, and colleagues found that linear transformers discover a unified numerical algorithm they call EAGLE.
As detailed in Linear Transformers Implicitly Discover Unified Numerical Algorithms, this discovered algorithm mimics Newton-Schulz iterations. It automatically adapts to hardware constraints and achieves quadratic convergence, solving ill-conditioned problems that would baffle standard gradient-based approaches.
From Internal Logic to External Training
The realization that Transformers are "natural" second-order optimizers has implications for how we train them. If the model's internal representation-refinement process acts like a pre-conditioned gradient descent—as suggested by Lav R. Varshney and Xinbo Wu in A Meta-Learning Perspective on Transformers for Causal Language Modeling—it follows that our external optimizers should leverage similar curvature information.
Researchers are already extending this intuition to the training phase:
- Aida: Guoqiang Zhang and Kenta Niwa developed Aida, which generalizes AdamW by incorporating higher-order magnitude scaling. This relaxation of stability constraints leads to better validation performance in Transformer architectures.
- Fast Natural Gradient Descent (FNGD): To overcome the computational cost of second-order methods, Ce Zhu and Yipeng Liu propose Inverse-Free Fast Natural Gradient Descent, which approximates the curvature of the loss landscape without the expensive matrix inversions usually required.
Summary of Intuition
The shift from viewing Transformers as first-order learners to second-order optimizers explains the "unreasonable" effectiveness of in-context learning. By internally approximating algorithms like Newton's Method, Transformers don't just follow the gradient; they reshape the optimization space itself, allowing them to learn from a handful of examples in a single forward pass.