Nonlinear In-Context Learning in Transformers

31 concepts30 papers

In-context learning (ICL) has revolutionized how we interact with large language models, allowing them to adapt to entirely new tasks using just a few examples provided in a prompt. To understand why this phenomenon occurs, early theoretical work relied heavily on a simplifying assumption: treating ICL as a form of linear regression.

For instance, von Oswald et al. demonstrated in Transformers Learn In-Context by Gradient Descent that a transformer's forward pass can mathematically simulate steps of gradient descent on linear data. Similarly, Ahn et al. showed in Transformers learn to implement preconditioned gradient descent for in-context learning that linear transformer weights naturally converge during pretraining to execute preconditioned gradient descent at inference time.

While these linear models provided elegant proofs, they created a significant functional complexity gap. Real-world language, logical reasoning, and complex mathematical operations are fundamentally non-linear. As pointed out by Saunshi et al. in On the Role of Depth and Looping for In-Context Learning with Task Diversity and Chen et al. in Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality (extended abstract), linear proxies fail to capture how models handle higher-order dependencies or generalize to non-linear function spaces.

To bridge this gap, a growing body of research explores the exact mechanisms that enable transformers to perform non-linear function approximation in-context.

Simulating Internal Neural Networks

How does a transformer transition from solving simple linear equations to approximating complex, non-linear functions? The answer lies in a mechanism known as meta-optimization.

Instead of merely updating a single weight vector during the forward pass, the attention mechanism can act as an optimizer that trains an implicit internal neural network. In In-context Learning on Function Classes Unveiled for Transformers, Wang et al. proved that transformers can implement approximate gradient descent on these internal sub-networks. This allows the model to dynamically construct complex, non-linear mappings and even switch between entirely different task classes (such as regression and classification) on the fly, purely through activation dynamics.

Bypassing the Curse of Dimensionality via Kernel Methods

When moving from linear to non-linear function spaces, standard statistical models typically suffer from the "curse of dimensionality"—requiring an exponentially larger number of examples as the complexity or dimensionality of the input grows. Yet, transformers manage to learn non-linear functions from remarkably short prompts.

This efficiency occurs because transformers implicitly operate as non-parametric kernel regression machines. In Understanding In-Context Learning on Structured Manifolds: Bridging Attention to Kernel Methods, Shen et al. established a formal bridge between attention layers and classical kernel methods. They demonstrated that transformers can achieve optimal regression rates for non-linear functions by exploiting the intrinsic low-dimensional manifold of the data, rather than being bogged down by the raw input size.

Building on this, Oko et al. showed in Pretrained Transformer Efficiently Learns Low-Dimensional Target Functions In-Context that the required prompt length depends strictly on this intrinsic dimensionality. This allows pretrained transformers to bypass the curse of dimensionality and achieve what Kim et al. describe in Transformers are Minimax Optimal Nonparametric In-Context Learners as "minimax optimal" error rates across highly complex, non-linear task spaces.

The Architecture of Nonlinearity: FFNs and Curvature

To unlock true non-linear function approximation, specific architectural components must step in. While pure attention layers excel at routing information and computing linear combinations, the feed-forward networks (FFNs) provide the necessary non-linear transformations.

In On the Role of Transformer Feed-Forward Layers in Nonlinear In-Context Learning, Sun et al. highlighted that FFNs are essential for adapting to non-linear relationships like quadratic or cubic functions. Without them, the model remains trapped within linear task assumptions.

Furthermore, the geometry of the target function dictates how easily a transformer can learn it. In Provable In-Context Learning of Nonlinear Regression with Transformers, Li et al. provided the first formal analysis of non-linear regression dynamics in ICL. They discovered a critical phase transition governed by the target function's curvature (its Lipschitz constant). When a function is highly curved, the transformer dynamically adjusts its attention weights to focus intensely on the most localized, relevant prompt examples, effectively performing a localized non-linear interpolation.

Empirical Reality of Non-linear ICL

This theoretical machinery matches striking empirical capabilities observed in large models. For example, Goulart et al. demonstrated in Can a Large Language Model Learn Matrix Functions In Context? that large language models can approximate highly intricate, non-linear matrix operations—such as Singular Value Decomposition (SVD)—directly within the context window, frequently outperforming specialized neural networks trained from scratch.

By shifting the theoretical paradigm from simple linear task simplification to non-linear function approximation, researchers are finally uncovering how the same basic transformer architecture seamlessly scales from basic pattern matching to sophisticated mathematical and linguistic reasoning.

Key Foundational Work

Transformers Learn Nonlinear Features In Context (2024) — contributed 2 concept elements to this direction.
How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations (2024) — contributed 1 concept elements to this direction.
A Closer Look at In-Context Learning under Distribution Shifts (2023) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_linear_task_simplification_aa2e0aed,
  title = {linear task simplification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-task-simplification/rdr_aa2e0aedd62340a7b6b5ecfc960d01eb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_linear_task_simplification_aa2e0aed_v1,
  title = {linear task simplification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-task-simplification/rdr_aa2e0aedd62340a7b6b5ecfc960d01eb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake