The Reversal Curse in Large Language Models

This direction is merged into another direction: reversal curse. Its last version stays readable and citable here.

Version 1 of 112 papers24 conceptsfrozen 2026-10-04

If you tell a large language model (LLM) that "Olaf Scholz was the ninth Chancellor of Germany," it will likely answer correctly when asked who the ninth chancellor was. However, if you ask "Who was Olaf Scholz the ninth of?", the model may struggle or fail entirely. This phenomenon, widely termed the reversal curse, reveals a fundamental asymmetry in how transformer-based models store and retrieve factual knowledge.

Despite their vast parameters, LLMs often fail to recognize that a relationship between two entities remains valid when the order of the query is reversed. This suggests that instead of building a cohesive internal world model, these systems often store facts as unidirectional string associations.

The Asymmetry of Factual Storage

The core of the problem lies in the training objective of modern LLMs: next-token prediction. When a model learns a fact during pre-training, it learns to predict the sequence "B" following "A." However, as Jun-Yu Ma and colleagues demonstrate in Untying the Reversal Curse via Bidirectional Language Model Editing, this does not automatically grant the model the ability to predict "A" given "B."

This "coherency gap" means that models lack a logically consistent internal representation. Paul Youssef and Jörg Schlötterer highlight this in The Queen of England is not England's Queen, noting that even when a model knows "Valetta is the capital of Malta," it frequently fails to realize that "Malta is the country whose capital is Valetta." The knowledge is present, but it is trapped in a one-way street.

Localized but Directional Memories

To understand why this happens, we must look at where knowledge lives inside a transformer. Research by Kevin Meng and David Bau in Locating and Editing Factual Associations in GPT found that factual associations are often stored in localized "early sites"—specifically the feed-forward networks (MLPs) of middle layers.

These layers act as linear associative memories. While highly efficient for surgical updates (using methods like ROME), these memories are fundamentally asymmetric. Updating a fact in the "forward" direction (Subject $\rightarrow$ Object) does not inherently modify the weights responsible for the "reverse" direction (Object $\rightarrow$ Subject). Consequently, the model's internal "map" is composed of directional links rather than a unified relational network.

The Reasoning and Consistency Gap

The implications of this curse extend beyond simple name-flipping. It creates a "reasoning gap" where models fail to infer basic logical inverses. In Can Language Models Act as Knowledge Bases at Scale?, Qiyuan He and Yizhong Wang show that models can memorize millions of triplets (e.g., "A is the father of B") but fail to infer the reciprocal relationship ("B is the child of A").

This lack of global consistency is further evidenced by Yujia Hu and Simon Razniewski in Enabling LLM Knowledge Analysis via Extensive Materialization. By extracting 101 million relational triples from GPT-4o-mini to create GPTKB, they found massive structural weaknesses where the model recognized a parent company but failed to identify its own subsidiary.

Even in long-context scenarios, where all relevant information is present in the prompt, models struggle. Meiyun Wang and Takeshi Kojima found in Lost in the Distance that models are significantly better at predicting the consequence of a relationship than identifying its antecedent, even when both are within the context window.

Strategies for Bidirectional Consistency

Researchers are now developing methods to force models toward logical symmetry:

  1. Bidirectional Editing: Jun-Yu Ma proposed BIRD, an editing method that explicitly optimizes for bidirectional relationships during model updates, ensuring that changing "A is B" also updates the model's response for "B is A."
  2. Deductive Closure Training (DCT): Afra Feyza Akyürek and Jacob Andreas introduced Deductive Closure Training, which uses the model's own reasoning to generate implications and contradictions. By fine-tuning the model on a logically consistent subset of its own thoughts, they improved factual accuracy and updatability by up to 26%.

These efforts suggest that while the reversal curse is a natural byproduct of how LLMs are currently trained, it is not an insurmountable law of architecture. By moving from sequence-based memorization to consistency-based training, we can begin to build models that understand relationships as holistic concepts rather than one-way strings.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_the_reversal_curse_in_large_language_mod_c9e5998f,
  title = {The Reversal Curse in Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-reversal-curse-in-large-language-models/rdr_c9e5998fe43f4411bb6c8168af591bb7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_reversal_curse_in_large_language_mod_c9e5998f_v1,
  title = {The Reversal Curse in Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-reversal-curse-in-large-language-models/rdr_c9e5998fe43f4411bb6c8168af591bb7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}