Out-of-Context Reasoning in Large Language Models

Jonathan Shaki, Emanuele La Malfa, Michael Wooldridge, Sarit Kraus

TL;DR

This paper investigates "out-of-context reasoning" in Large Language Models (LLMs)—the ability to reason over facts memorized during training rather than provided directly in a prompt. By introducing a lightweight "out-of-context representation learning" technique that trains only new token embeddings, the authors demonstrate that LLMs can master abstract relational properties like transitivity and symmetry. Crucially, the study reveals that this reasoning is baked directly into the geometric structure of the embedding space during training, rather than computed dynamically at inference time.

What Problem Does This Paper Address?

Most current LLM benchmarks evaluate in-context learning, where all the rules, facts, and context needed to solve a problem are explicitly spelled out in the prompt. However, true systemic intelligence requires out-of-context reasoning—the ability to recall facts from memory and compose them logically to infer completely unseen relationships.

Prior research into out-of-context learning often focused on overly complex tasks or linguistically ambiguous relations (such as the verb "is"), which triggered the infamous "reversal curse" (where a model trained on "A is B" fails to realize "B is A"). This paper isolates the core mechanics of out-of-context reasoning by stripping away linguistic noise and testing LLMs on fundamental, unambiguous mathematical binary relations: equality ($=$), inequality ($<$), and proper inclusion ($\subset$).

Methods

The authors introduce a novel, lightweight framework called out-of-context representation learning. Instead of fine-tuning the entire model—which alters core parameters and risks overfitting—or relying on in-context prompts, they augment the LLM's vocabulary with entirely new, abstract tokens.

During the training phase, the weights of the LLM remain completely frozen. Only the dense vector embeddings of these newly introduced tokens are optimized via gradient descent to satisfy specific relational axioms (e.g., $e_1 < e_2$ and $e_2 < e_3$). During inference, the model is presented with unseen pairs (e.g., evaluating if $e_1 < e_3$) to test if it can successfully generalize properties like reflexivity, symmetry, and transitivity.

Figure 1: Left: an example of a binary relation that a model would learn without issues (top). On the other hand, both the examples at the bottom raise issues in terms of the acceptability of the answer (which nonetheless would follow from an axiomatic system). Right: our solution encompasses all three cases without falling into the biases of existing representations.

Datasets

The authors generated synthetic Question-Answering (Q&A) datasets consisting of positive and negative queries (e.g., "Is $a < b$?" or "Is it wrong that $a < b$?"). To understand how data composition affects learning, they structured the training data into two distinct settings:

  1. Setting I (Minimal sufficient hypotheses): The training data contains only the absolute minimum facts required to logically deduce the answers (e.g., a simple chain of inequalities like $e_i < e_{i+1}$).
  2. Setting II (Illustrative information): The training data is enriched with contrastive examples and facts demonstrating other properties of the relation, creating a balanced dataset.

The experiments were executed across three open-weight models: Llama-3-8B, Llama-3.2-1B, and Mistral-7B-v0.3.

Key Insights and Ideas

  • Reasoning Happens During Training, Not Inference: Because the model's internal weights are frozen during testing and no reasoning chains (like Chain-of-Thought) are provided in the prompt, the logical compositionality is entirely handled by the geometric arrangement of the learned token embeddings during the training phase.
  • Geometric Organization of Abstract Concepts: When an LLM successfully learns a relation out-of-context, it naturally organizes the new token embeddings into highly structured spatial layouts. For example, inequality ($<$) tokens spontaneously arrange themselves in a circular manifold, beautifully mirroring how the LLM natively represents actual numbers.

PCA showing circular arrangement of total order embeddings PCA showing native number embeddings in Llama-3-8B

  • The Distance Effect: Mirroring a well-documented phenomenon in human psychology, LLMs exhibit higher accuracy when evaluating relations between elements that are logically further apart in a chain (e.g., comparing $e_1 < e_9$ vs $e_1 < e_3$), even though longer distances technically require more compositional steps.
  • Demystifying the Reversal Curse: The authors show that the reversal curse is largely a byproduct of the linguistic ambiguity of the word "is" (which can act as an equality operator or a noun copula). When using mathematically precise relations and representation learning, even small models successfully master symmetry.

Key Results

  • Failure in the Minimal Setting: In the minimal sufficient hypotheses setting, all models failed to generalize effectively, typically collapsing to match the baseline training statistics due to the lack of contrastive examples.
  • Success in the Illustrative Setting: When provided with diverse, contrastive data, all models performed statistically significantly better than chance. Llama-3-8B led the pack with an average accuracy of 0.65 (against a data-distribution baseline of 0.49), followed by Llama-3.2-1B at 0.62, and Mistral-7B-v0.3 at 0.60.
  • Outperforming In-Context Learning: Out-of-context representation learning consistently yielded higher accuracy than standard in-context learning across the majority of relational test cases, proving to be a highly efficient way to inject relational knowledge.

Limitations and Future Work

  • Noisy Short-Distance Representations: The learned embedding spaces are inherently noisy at close proximity, which explains why the models struggle with short-distance comparisons.
  • Lack of Explicit Reasoning Chains: The current training paradigm does not utilize or generate explicit multi-step reasoning paths. The authors note that exploring whether models can generate chains of thought out-of-context without explicit training remains an open challenge.
  • Data Contamination Risks: Future work will need to rigorously evaluate the robustness of out-of-context representation learning against pre-training data contamination by validating these findings on plain-text corpora.

Summary updated

Abstract

We study how large language models (LLMs) reason about memorized knowledge through simple binary relations such as equality ($=$), inequality ($<$), and inclusion ($\subset$). Unlike in-context reasoning, the axioms (e.g., $a < b, b < c$) are only seen during training and not provided in the task prompt (e.g., evaluating $a < c$). The tasks require one or more reasoning steps, and data aggregation from one or more sources, showing performance change with task complexity. We introduce a lightweight technique, out-of-context representation learning, which trains only new token embeddings on axioms and evaluates them on unseen tasks. Across reflexivity, symmetry, and transitivity tests, LLMs mostly perform statistically significant better than chance, making the correct answer extractable when testing multiple phrasing variations, but still fall short of consistent reasoning on every single query. Analysis shows that the learned embeddings are organized in structured ways, suggesting real relational understanding. Surprisingly, it also indicates that the core reasoning happens during the training, not inference.
BibTeX
@misc{shaki2025out,
  author = {Jonathan Shaki and Emanuele La Malfa and Michael Wooldridge and Sarit Kraus},
  title = {{Out-of-Context Reasoning in Large Language Models}},
  year = {2025},
  eprint = {2503.10408},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.10408},
}
Figure