Understanding the Logical Capabilities of Large Language Models via Out-of-Context Representation Learning

Jonathan Shaki, Emanuele La Malfa, Michael J. Wooldridge, Sarit Kraus

Abstract

We study the capabilities of Large Language Models (LLM) on binary relations, a ubiquitous concept in math employed in most reasoning, math and logic benchmarks. This work focuses on equality, inequality, and inclusion, along with the properties they satisfy, such as ir/reflexivity, a/symmetry, transitivity, and logical complexity (e.g., number of reasoning ``hops''). We propose an alternative to in-context learning that trains only the representations of newly introduced tokens, namely out-of-context representation learning. This method mitigates linguistic biases already present in a model and, differently from in-context learning, does not rely on external information or illustrations. We argue out-of-context representation learning as a better alternative to in-context learning and fine-tuning to evaluate the capabilities of LLMs on logic tasks that are the building blocks of more complex reasoning benchmarks.

Introduction

Most benchmarks for Large Language Models (LLMs) evaluate in-context reasoning, where the model is provided with all necessary facts within a prompt. However, human reasoning often relies on "out-of-context" knowledge—information that has been internalized and memorized. A significant challenge in LLM reasoning is the "reversal curse", where a model trained on "A is B" fails to infer "B is A". This limitation often stems from the linguistic ambiguity of natural language. This paper explores whether LLMs can master abstract binary relations (equality, inequality, and inclusion) when these relations are learned as part of the model's internal vocabulary rather than being described in a prompt.

Methodology: Out-of-Context Representation Learning (OCRL)

The authors propose Out-of-Context Representation Learning (OCRL) as a lightweight alternative to full fine-tuning. In OCRL, the model's weights remain frozen while only the embeddings (mathematical vectors) of newly introduced, meaningless tokens (e.g., new-token-1) are trained on logical axioms.

Overview of the OCRL training process and how it addresses the 'reversal curse' compared to standard linguistic training.

By training only the representations, the model is forced to internalize relational logic without relying on pre-existing linguistic biases. After training, the model is tested on unseen token pairs to evaluate its ability to generalize learned logical properties, such as transitivity (e.g., if $a=b$ and $b=c$, then $a=c$).

Experimental Setup

The study utilized synthetic datasets centered on three fundamental relations:

  • Equality ($=$): Testing reflexivity, symmetry, and transitivity.
  • Inequality ($<$): Testing strict total order (asymmetry and transitivity).
  • Inclusion ($\subseteq$): Testing proper subset relations.

Experiments were conducted on Llama-3-8B, Llama-3.2-1B, and Mistral-7B-v0.3 across two settings: a "Minimal" setting with bare-bones facts and an "Illustrative" setting with contrastive examples to help balance internal representations.

Results and Discussion

Structured Latent Space

A key finding is that LLMs organize learned embeddings into highly structured latent spaces. Principal Component Analysis (PCA) reveals that the model arranges tokens in manifolds that mirror their logical order. For instance, tokens representing a sequence of numbers are arranged in a clear arc, despite the model never receiving explicit instructions on spatial arrangement.

PCA visualization of learned embeddings for numbers 0-99, demonstrating the emergence of a structured manifold.

Reasoning During Training

The results suggest that reasoning may occur during the weight-update phase of learning rather than solely at inference. Models often performed better via OCRL than via In-Context Learning (ICL) for complex relations like transitivity. Furthermore, the "distance effect"—where models more accurately judge relations between "far apart" elements in a chain—indicates the formation of a "fuzzy" internal mental line.

Model Performance and the Reversal Curse

In the "Illustrative" setting, Llama-3-8B achieved an average accuracy of 0.65, significantly beating random baselines. By using abstract tokens and OCRL, models successfully learned symmetric relations, suggesting that the "reversal curse" is primarily a linguistic byproduct of natural language usage rather than a fundamental architectural flaw in Transformers.

Conclusion

OCRL provides a robust framework for evaluating the core logical capabilities of LLMs. While models still struggle with fine-grained comparisons due to "noisy" embeddings, the ability to internalize and structure abstract relations suggests a path toward more reliable reasoning. Future work will explore whether these internalized representations can support more complex "Chain-of-Thought" generation.

Summary updated

Abstract

We study the capabilities of Large Language Models (LLM) on binary relations, a ubiquitous concept in math employed in most reasoning, math and logic benchmarks. This work focuses on equality, inequality, and inclusion, along with the properties they satisfy, such as ir/reflexivity, a/symmetry, transitivity, and logical complexity (e.g., number of reasoning ``hops''). We propose an alternative to in-context learning that trains only the representations of newly introduced tokens, namely out-of-context representation learning. This method mitigates linguistic biases already present in a model and, differently from in-context learning, does not rely on external information or illustrations. We argue out-of-context representation learning as a better alternative to in-context learning and fine-tuning to evaluate the capabilities of LLMs on logic tasks that are the building blocks of more complex reasoning benchmarks.
BibTeX
@misc{shaki2025understanding,
  author = {Jonathan Shaki and Emanuele La Malfa and Michael J. Wooldridge and Sarit Kraus},
  title = {{Understanding the Logical Capabilities of Large Language Models via Out-of-Context Representation Learning}},
  year = {2025},
  eprint = {2503.10408},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.10408},
}
Figure