Language Models Struggle to Use Representations Learned In-Context

Michael A. Lepori, Tal Linzen, Ann Yuan, Katja Filippova

TL;DR

While large language models (LLMs) can learn to represent complex new structures (like spatial grids) entirely from context, they frequently fail to actually use these representations to solve tasks. This paper demonstrates that these "in-context representations" are often inert, meaning the model encodes the information in its hidden states but cannot successfully deploy it for prediction or reasoning, especially when faced with a delay or a novel task.

What Problem Does This Paper Address?

Artificial General Intelligence requires agents that can adapt to radically new environments upon deployment. A key component of this is in-context representation learning: the ability to induce a "world model" of a new environment just by observing data (like a sequence of states) without updating the model's weights.

Previous research showed that LLMs can do this; for example, if you give an LLM a random walk over a hidden 2D grid of words, its internal hidden states will eventually reflect the geometry of that grid. However, the authors of this paper ask a critical follow-up question: Can the model actually use these learned representations? They investigate the gap between encoding information and deploying it for downstream tasks, a property they call "flexible deployability."

Methods

The authors evaluate several open-weights models (Gemma-3, OLMo-2) and frontier reasoning models (Gemini 2.5, GPT-5) using two primary synthetic tasks:

  1. Graph Tracing (Next-Token Prediction): Models are given a random walk over a latent topology (1D lines or 2D grids). The authors compare a Prefilled condition (predicting the next token immediately) vs. an Instruction condition (where a user prompt is inserted between the walk and the prediction).
  2. Adaptive World Modeling (AWM): A novel task where the model must first infer a topology from a random walk and then apply a few-shot rule (e.g., "move two steps down in the grid") to a new starting word.

To measure internal representations, they use two metrics:

  • Dirichlet Energy (DE): Measures how close the representations of adjacent states are in the latent space.
  • Distance Correlation (DC): Measures how well the Euclidean distances between representations correlate with the actual distances in the underlying topology.
An illustration of how a model might learn a topology internally but fail to use it when prompted to generate an answer.

Datasets

The study uses synthetic datasets consisting of random walks over:

  • 1D Lines: 16 or 25 states.
  • 2D Grids: 4x4 or 5x5 configurations. States are represented by common, semantically unrelated one-token words (e.g., "city," "ink," "toy"). The models tested include the Gemma-3 family, OLMo-2-13b, Gemini-2.5-Pro/Flash, and GPT-5/GPT-5-mini.

Key Insights and Ideas

  • The "Inert" Representation: The most striking finding is that a model can have a high-quality internal map of a grid (high DC, low DE) but still fail to predict the next neighbor if there is a slight delay or a change in prompt format. The representation exists but is "inert" for the task at hand.
  • The Instruction Penalty: Open-weights models perform well when the task is "prefilled" (immediate), but their performance collapses in the "Instruction" condition. Analysis shows the internal representations are actually better in the Instruction condition, proving the failure is one of deployment, not encoding.
  • Dimensionality Collapse: While frontier reasoning models (like GPT-5) are much better at 1D line topologies, their performance "entirely collapses" when moving to 2D grids. This suggests that current scaling and reasoning techniques (like Chain-of-Thought) still struggle with complex spatial/topological induction.
  • CoT as a Patch: Reasoning models seem to use their externalized thinking chains to compensate for inert hidden representations, but this is an imperfect workaround that fails as task complexity increases.

Key Results

  • Next-Token Prediction: Open-weights models show a massive performance gap between Prefilled and Instruction conditions (see Figure 4). For example, on a 5x5 grid, accuracy drops from near-perfect to near-zero in the Instruction condition.
  • AWM Performance: On the novel AWM task, open-weights models barely beat a naive baseline that just guesses a previously seen neighbor.
  • Reasoning Models: Gemini 2.5 Pro and GPT-5 achieve high accuracy on 1D lines (~80-90%) but drop to near 0% on 2D grids for complex rules (see Figure 8).
  • Explicit Knowledge vs. Induction: When models are given the grid coordinates explicitly in the prompt (bypassing the need for in-context induction), they succeed. This confirms the bottleneck is the model's inability to bridge the gap between induced latent structures and active reasoning.
A comparison of how different models handle the Adaptive World Modeling task across different rules and topologies.

Limitations and Future Work

  • No Solution Provided: The paper identifies a significant "deployment gap" but does not offer a specific architectural fix.
  • Opaque Reasoning: It remains unclear exactly how Chain-of-Thought (CoT) interacts with hidden representations. In some cases, ablating CoT actually improved performance for Gemini Flash, suggesting the "thinking" process can sometimes interfere with latent knowledge.
  • Future Directions: The authors suggest that future work should focus on training regimens or mechanistic interventions (like making representations available to earlier layers) to ensure that in-context learned information is actually usable by the model's "reasoning" components.

Summary updated

Abstract

Though large language models (LLMs) have enabled great success across a wide variety of tasks, they still appear to fall short of one of the loftier goals of artificial intelligence research: creating an artificial system that can adapt its behavior to radically new contexts upon deployment. One important step towards this goal is to create systems that can induce rich representations of data that are seen in-context, and then flexibly deploy these representations to accomplish goals. Recently, Park et al. (2024) demonstrated that current LLMs are indeed capable of inducing such representation from context (i.e., in-context representation learning). The present study investigates whether LLMs can use these representations to complete simple downstream tasks. We first assess whether open-weights LLMs can use in-context representations for next-token prediction, and then probe models using a novel task, adaptive world modeling. In both tasks, we find evidence that open-weights LLMs struggle to deploy representations of novel semantics that are defined in-context, even if they encode these semantics in their latent representations. Furthermore, we assess closed-source, state-of-the-art reasoning models on the adaptive world modeling task, demonstrating that even the most performant LLMs cannot reliably leverage novel patterns presented in-context. Overall, this work seeks to inspire novel methods for encouraging models to not only encode information presented in-context, but to do so in a manner that supports flexible deployment of this information.
BibTeX
@inproceedings{lepori2026language,
  author = {Michael A. Lepori and Tal Linzen and Ann Yuan and Katja Filippova},
  title = {{Language Models Struggle to Use Representations Learned In-Context}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.676},
}
Figure