Abstract
We study the capabilities of Large Language Models (LLM) on binary relations, a ubiquitous concept in math employed in most reasoning, math and logic benchmarks. This work focuses on equality, inequality, and inclusion, along with the properties they satisfy, such as ir/reflexivity, a/symmetry, transitivity, and logical complexity (e.g., number of reasoning ``hops''). We propose an alternative to in-context learning that trains only the representations of newly introduced tokens, namely out-of-context representation learning. This method mitigates linguistic biases already present in a model and, differently from in-context learning, does not rely on external information or illustrations. We argue out-of-context representation learning as a better alternative to in-context learning and fine-tuning to evaluate the capabilities of LLMs on logic tasks that are the building blocks of more complex reasoning benchmarks.
Introduction
Most benchmarks for Large Language Models (LLMs) evaluate in-context reasoning, where the model is provided with all necessary facts within a prompt. However, human reasoning often relies on "out-of-context" knowledge—information that has been internalized and memorized. A significant challenge in LLM reasoning is the "reversal curse", where a model trained on "A is B" fails to infer "B is A". This limitation often stems from the linguistic ambiguity of natural language. This paper explores whether LLMs can master abstract binary relations (equality, inequality, and inclusion) when these relations are learned as part of the model's internal vocabulary rather than being described in a prompt.
Methodology: Out-of-Context Representation Learning (OCRL)
The authors propose Out-of-Context Representation Learning (OCRL) as a lightweight alternative to full fine-tuning. In OCRL, the model's weights remain frozen while only the embeddings (mathematical vectors) of newly introduced, meaningless tokens (e.g., new-token-1) are trained on logical axioms.
By training only the representations, the model is forced to internalize relational logic without relying on pre-existing linguistic biases. After training, the model is tested on unseen token pairs to evaluate its ability to generalize learned logical properties, such as transitivity (e.g., if $a=b$ and $b=c$, then $a=c$).
Experimental Setup
The study utilized synthetic datasets centered on three fundamental relations:
- Equality ($=$): Testing reflexivity, symmetry, and transitivity.
- Inequality ($<$): Testing strict total order (asymmetry and transitivity).
- Inclusion ($\subseteq$): Testing proper subset relations.
Experiments were conducted on Llama-3-8B, Llama-3.2-1B, and Mistral-7B-v0.3 across two settings: a "Minimal" setting with bare-bones facts and an "Illustrative" setting with contrastive examples to help balance internal representations.
Results and Discussion
Structured Latent Space
A key finding is that LLMs organize learned embeddings into highly structured latent spaces. Principal Component Analysis (PCA) reveals that the model arranges tokens in manifolds that mirror their logical order. For instance, tokens representing a sequence of numbers are arranged in a clear arc, despite the model never receiving explicit instructions on spatial arrangement.
Reasoning During Training
The results suggest that reasoning may occur during the weight-update phase of learning rather than solely at inference. Models often performed better via OCRL than via In-Context Learning (ICL) for complex relations like transitivity. Furthermore, the "distance effect"—where models more accurately judge relations between "far apart" elements in a chain—indicates the formation of a "fuzzy" internal mental line.
Model Performance and the Reversal Curse
In the "Illustrative" setting, Llama-3-8B achieved an average accuracy of 0.65, significantly beating random baselines. By using abstract tokens and OCRL, models successfully learned symmetric relations, suggesting that the "reversal curse" is primarily a linguistic byproduct of natural language usage rather than a fundamental architectural flaw in Transformers.
Conclusion
OCRL provides a robust framework for evaluating the core logical capabilities of LLMs. While models still struggle with fine-grained comparisons due to "noisy" embeddings, the ability to internalize and structure abstract relations suggests a path toward more reliable reasoning. Future work will explore whether these internalized representations can support more complex "Chain-of-Thought" generation.
Summary updated