Provable Low-Frequency Bias of In-Context Learning of Representations

Yongyi Yang, Hidenori Tanaka, Wei HuSubmission to ICLR 2026 · Rejected

TL;DR

When fed sequences generated by random walks on graphs, large language models (LLMs) spontaneously organize their hidden representations to mirror the graph's underlying geometry. This paper provides the first rigorous theoretical explanation for this phenomenon, proving that representations undergo a "double convergence" across context length and layer depth that acts as a low-frequency spectral filter on graph signals.

What Problem Does This Paper Address?

Recent empirical work (notably Park et al., 2024) discovered a striking emergent behavior called In-Context Learning of Representations (ICLR): when a pretrained LLM processes a sequence of words generated by a random walk on a hidden planar graph, its internal representations discard their original semantic meanings and reorganize to reflect the graph’s geometric topology.

While fascinating, several fundamental puzzles remained unexplained:

  1. Why does in-context learning suppress pretrained word semantics in favor of prompt structure?
  2. Why do representations compress at the graph periphery instead of forming an exact grid?
  3. Why does representation "energy" decay across layers without ever collapsing to zero?

This paper establishes a mathematical framework explaining the mechanics behind these empirical observations.

Methods

The authors formalize the forward dynamics of deep Transformers using a framework called Double Convergence:

  • Context-wise Convergence: As sequence length $n$ grows, hidden vectors for identical tokens converge to position-independent limiting representations (Theorem 1). This relies on defining balanced attention maps that generalize induction heads by distributing attention proportionally across functionally related tokens (e.g., neighbors in the sequence).
  • Layer-wise Convergence: Across network depth $L$, balanced attention acts as a linear transformation on these limiting representations. By decomposing attention into four core types—self-connections ($A$), neighbor-connections ($B$), global connections ($O$), and attention sinks ($T$)—the authors show that attention layers behave like normalized graph shift operators.
  • Neuron-wise Transformations (FFNs & Normalizations): Non-linearities are modeled as "great mappings" that preserve subspace distances, showing that double convergence is invariant to standard layer-wise feedforward blocks.

Combining these reveals that deep layer-wise forward propagation is mathematically equivalent to power iteration on a normalized graph operator $M = D^{-1/2} W D^{-1/2}$, projecting representations into the top eigenspace of $M$.

Datasets

  • Synthetic Graph Sequences: Random walks on 2D planar grid graphs (replicating Park et al., 2024) where each node maps to a unique vocabulary token.
  • Noisy Random Walks: The authors introduce a corrupted variant where 1%, 2.5%, or 5% of transitions jump to a uniformly random vocabulary token rather than a valid graph neighbor.
  • Model Tested: All empirical validations were run on pretrained Llama-3.1-8B via the NNSight interpretability library.

Key Insights and Ideas

  • Low-Frequency Bias as a Spectral Filter: The top eigenvectors of the normalized adjacency matrix $M$ correspond directly to the lowest eigenvectors of the normalized graph Laplacian $\tilde{L} = I - M$. Thus, the transformer naturally acts as a low-pass graph filter, amplifying smooth, low-energy signals (adjacent nodes have similar representations) and eliminating high-frequency components.
  • Semantic Erasure: A word's original embedding contains a mix of frequencies. Because the double convergence selectively attenuates high-frequency modes, initial semantic features are progressively washed out across layers.
  • Stationary Distribution Causes Peripheral Distortion: The compression seen at graph boundaries is not merely an artifact of random walk visitation counts. Rather, context-wise attention creates a reweighted adjacency matrix $W_{x,y} = \tilde{W}_{x,y} \pi_x \pi_y$ governed by the stationary distribution $\pi$. The eigenvectors of this reweighted matrix inherently pinch peripheral nodes.
  • Energy Decays but Never Collapses: Graph Dirichlet energy drops because high-frequency components shrink to zero. However, energy does not vanish because representations accumulate in the lowest non-trivial eigenspaces (components 1 and 2), preventing total collapse.

Key Results

  • Empirical Validation of Attention Assumptions: Analysis of Llama-3.1-8B shows that over 72% of actual attention weights across heads correspond directly to the authors' structured classes ($A$, $B$, and $T$).
  • Spectral Alignment: Principal Component Analysis (PCA) of hidden representations in Llama-3.1-8B closely matches the analytically derived 2nd and 3rd eigenvectors of $M$, confirming the spectral convergence prediction.
  • Intrinsic Denoising Robustness: When corrupting inputs with 1% to 5% uniform random noise (introducing high-frequency perturbations), Llama-3.1-8B's predictions completely filter out invalid transitions at later context lengths, achieving 100% adherence to the underlying graph structure.

Limitations and Future Work

  • Exogenous Attention Maps: The primary simplification is treating attention maps as structurally fixed/given functions of tokens rather than deriving them end-to-end from Query-Key dot products.
  • DGP Specificity: The tightest convergence guarantees assume Markovian random walks over graphs. Extending this to broader non-Markovian data distributions remains an open challenge (though preliminary distribution-free steps are explored in Appendix E).
  • Pretraining Emergence: The paper explains how structured attention maps induce representation geometry during inference, but does not prove why gradient descent during pretraining causes LLMs to develop such induction heads in the first place.

Summary updated

Abstract

In-context learning (ICL) enables large language models (LLMs) to acquire new behaviors from the input sequence alone without any parameter updates. Recent studies have shown that ICL can surpass the original meaning learned in the pretraining stage through internalizing the structure of the data-generating process (DGP) of the prompt into the hidden representations. However, the mechanisms by which LLMs achieve this ability are left open. In this paper, we present the first rigorous explanation of such phenomena by introducing a unified framework of double convergence, where hidden representations converge both over context and across layers. This double convergence process leads to an implicit bias towards smooth (low-frequency) representations, which we prove analytically and verify empirically. Our theory explains several open empirical observations, including why learned representations exhibit globally structured but locally distorted geometry, and why their total energy decays without vanishing. Moreover, our theory predicts that ICL has an intrinsic robustness towards high-frequency noise, which we empirically confirm. These results provide new insights into the underlying mechanisms of ICL, and a theoretical foundation to study it that hopefully extends to more general data distributions and settings.
BibTeX
@inproceedings{yang2025provable,
  author = {Yongyi Yang and Hidenori Tanaka and Wei Hu},
  title = {{Provable Low-Frequency Bias of In-Context Learning of Representations}},
  booktitle = {NeurReps @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=53njoEoBZo},
}
Figure