Abstract
We study how large language models (LLMs) reason about memorized knowledge through simple binary relations such as equality ($=$), inequality ($<$), and inclusion ($\subset$). Unlike in-context reasoning, the axioms (e.g., $a < b, b < c$) are only seen during training and not provided in the task prompt (e.g., evaluating $a < c$). The tasks require one or more reasoning steps, and data aggregation from one or more sources, showing performance change with task complexity. We introduce a lightweight technique, out-of-context representation learning, which trains only new token embeddings on axioms and evaluates them on unseen tasks. Across reflexivity, symmetry, and transitivity tests, LLMs mostly perform statistically significant better than chance, making the correct answer extractable when testing multiple phrasing variations, but still fall short of consistent reasoning on every single query. Analysis shows that the learned embeddings are organized in structured ways, suggesting real relational understanding. Surprisingly, it also indicates that the core reasoning happens during the training, not inference.
Introduction
The evaluation of reasoning in Large Language Models (LLMs) has primarily focused on in-context learning, where all necessary premises are provided within the prompt. While LLMs excel at these tasks, their ability to reason over out-of-context knowledge—facts encountered during pre-training or fine-tuning but absent from the immediate prompt—remains poorly understood. Previous research has identified significant hurdles, such as the "reversal curse," where models fail to infer "B is A" after learning "A is B." This study investigates whether these failures are fundamental architectural limitations or artifacts of linguistic ambiguity.
Methodology: Out-of-Context Representation Learning
To isolate reasoning from linguistic noise, we propose a novel method called out-of-context representation learning. This technique involves introducing new, abstract tokens (e.g., $e_1, e_2$) into the model's vocabulary. We freeze the model's weights and train only the embeddings of these new tokens on a set of logical axioms.
By training only the representations, we can observe how the model's internal state adapts to new relational information. Evaluation is then performed by prompting the model with unseen questions (e.g., "Is $e_1 < e_3$?") without providing the original axioms in the context.
Geometric Reasoning and the Latent Space
A key finding of this work is that the "reasoning" process appears to occur during the training phase rather than at inference time. Through Principal Component Analysis (PCA) of the learned embeddings, we observe that the model organizes these tokens into highly structured geometric arrangements that reflect the underlying logical properties.
For instance, tokens trained on a strict total order ($<$) arrange themselves in a linear or circular manifold, effectively creating an internal "number line." This spatial organization allows the model to "read off" the answer during inference based on the relative positions of the embeddings.
Key Insights
- The Distance Effect: Models exhibit higher accuracy when comparing tokens that are logically further apart in a chain (e.g., $e_1$ vs $e_9$) compared to adjacent tokens. This mirrors human cognitive biases in numerical comparison.
- Linguistic Artifacts: The "reversal curse" is largely mitigated when using precise mathematical relations instead of ambiguous natural language copulas like "is."
- Requirement for Illustrative Data: Models fail to generalize when provided only with minimal sufficient hypotheses. They require "illustrative" or contrastive data (showing both true and false relations) to correctly internalize the mathematical properties.
Results
We evaluated Llama-3-8B, Llama-3.2-1B, and Mistral-7B-v0.3. In the illustrative data setting, Llama-3-8B achieved the highest average accuracy at 65%, significantly outperforming the 49% baseline. Notably, for transitivity tasks in strict total order, accuracy reached up to 99% for large logical distances. Interestingly, this out-of-context approach often outperformed standard in-context learning, suggesting that embedding-based memorization can be a robust way to store and reason over relational data.
Conclusion
Our results demonstrate that LLMs are capable of out-of-context reasoning for simple binary relations. The discovery that reasoning is encoded into the geometry of the embedding space during training offers a new perspective on how these models internalize knowledge. Future work will explore the scalability of this approach to more complex logical structures and its robustness against natural language variations.
Summary updated