Transformers as Iterative Optimization Engines

This direction is merged into another direction: deep unfolding. Its last version stays readable and citable here.

Version 1 of 122 papers28 conceptsfrozen 2026-10-04

How does a Transformer, with its weights frozen after training, "learn" to solve a new task just by looking at a few examples in a prompt? This phenomenon, In-Context Learning (ICL), has traditionally been treated as a mysterious emergent property. However, a growing body of research suggests a more mechanical explanation: the forward pass of a Transformer is not just a sequence of feature transformations, but an unrolled execution of an iterative optimization algorithm.

In this view, the Transformer’s architecture acts as a programmable computer that uses its layers to refine a solution to the problem presented in its context window.

The Layer-to-Iteration Mapping

The fundamental intuition behind this research direction is a structural mapping: each layer (or block of layers) in a Transformer corresponds to a single iteration of a mathematical optimization algorithm. When you feed a prompt into a deep Transformer, the data doesn't just pass through; it is being "optimized" step-by-step.

Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra demonstrated in Transformers learn to implement preconditioned gradient descent for in-context learning that for linear regression tasks, Transformers naturally learn weights that execute Preconditioned Gradient Descent. Instead of a simple gradient step, the model learns to adapt to the noise and distribution of the data, refining its internal "weight estimate" as the signal moves deeper into the network.

This isn't limited to linear problems. Xiang Cheng and colleagues further showed in Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context that the forward pass can implement Functional Gradient Descent, allowing the model to optimize its way toward a Bayes-optimal predictor for non-linear tasks.

Beyond First-Order Methods

While Gradient Descent is the most common analogy, Transformers are capable of much more sophisticated numerical methods. If Gradient Descent is a slow walk down a hill, second-order methods like Newton’s Method are a direct leap toward the valley floor.

Researchers Deqing Fu, Tian-Qi Chen, and Vatsal Sharan argue in Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression that Transformers don't just settle for simple descent; they can achieve quadratic convergence. This means the error drops exponentially faster as depth increases. Similarly, Angeliki Giannou and Jason D. Lee proved in How Well Can Transformers Emulate In-context Newton's Method? that Transformers can be explicitly constructed to emulate Newton’s Method, reaching high precision for complex tasks like logistic regression in very few layers.

A Universal Algorithmic Engine

The "internal optimizer" hypothesis extends far beyond regression. The Transformer architecture appears to be a general-purpose engine for various classical algorithms:

Depth as Computational Budget

If layers are iterations, then depth is the model's "computational budget" for solving a task. A shallow model might only have enough "steps" to perform a crude approximation, while a deep model can refine its answer to near-perfection.

Blake Bordelon and Cengiz Pehlevan explored this in Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Time. They found that while width and context length help with memory and data coverage, depth is specifically required when the model faces diverse tasks with shifting statistics. Each additional layer acts as a refinement step that allows the model to "untangle" complex, structured problems.

To push this even further, Haodong Liang and Lifeng Lai investigated Looped Transformers in Transformers Handle Endogeneity in In-Context Linear Regression. By feeding the output of a block back into itself, the model can perform an arbitrary number of iterations, allowing it to solve econometrics problems like Instrumental Variable (IV) regression with exponential convergence.

Why This Works: The Energy Functional View

Why does the Transformer architecture lend itself so well to optimization? Abdel-Rahim Mezidi and Massimiliano Pontil suggest a "Bregman Proximal" viewpoint in A Bregman Proximal Viewpoint on Neural Operators. They argue that neural layers can be seen as solving a regularized optimization problem at each step. Similarly, Yueming Lyu and Ivor W. Tsang propose the Neural Optimization Kernel (NOK) framework, where each layer is a step in a monotonic descent algorithm that minimizes a specific energy functional (Neural Optimization Kernel: Towards Robust Deep Learning).

In essence, the Transformer isn't just a "black box" predicting the next token. It is a dynamic system that, through its depth, unrolls a sophisticated mathematical solver to find the best possible answer hidden within the context.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_transformers_as_iterative_optimization_e_673a37cb,
  title = {Transformers as Iterative Optimization Engines},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/transformers-as-iterative-optimization-engines/rdr_673a37cbc2f5486fab74ec53fdf7b638},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_transformers_as_iterative_optimization_e_673a37cb_v1,
  title = {Transformers as Iterative Optimization Engines},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/transformers-as-iterative-optimization-engines/rdr_673a37cbc2f5486fab74ec53fdf7b638/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}