Bidirectional Logical Generalization in Large Language Models

This direction is merged into another direction: reversal curse. Its last version stays readable and citable here.

Version 1 of 142 papers51 conceptsfrozen 2026-10-04

If a human learns that "Olaf Scholz was the Chancellor of Germany," they instantly and effortlessly understand that "The Chancellor of Germany was Olaf Scholz." However, as demonstrated by Lukas Berglund and colleagues in The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A", large language models (LLMs) lack this basic logical symmetry. This phenomenon, known as the Reversal Curse, reveals a fundamental gap between statistical pattern matching and true relational understanding.

The Mechanism of Directional Brittleness

The Reversal Curse is not a mere data deficiency but a byproduct of the standard auto-regressive training objective. Models are trained to predict the next token in a sequence, which inherently prioritizes a forward-moving causal chain. When a model internalizes a fact like "A is B," it optimizes its weights to predict "B" given "A." However, this update does not automatically strengthen the weights required to predict "A" given "B."

This "directional brittleness" persists across model scales. As noted in A Survey of Reasoning with Foundation Models by Jiankai Sun and others, increasing the number of parameters does not inherently grant models the ability to perform basic logical symmetry. The knowledge remains trapped in the specific sequence in which it was encountered.

The Geometry of Asymmetry

Why does the internal representation fail to generalize? Hanji Du argues in Chiral Symmetry Breaking in Transformers that the problem is geometric. Because of causal masking, an entity’s latent representation differs significantly depending on whether it appears as a subject (the start of the sequence) or an object (the end). This structural asymmetry prevents the model from recognizing that the two representations refer to the same concept.

Researchers like Boshi Wang and Huan Sun frame this as a failure of conceptual binding. Instead of storing facts as flexible, bidirectional relationships, Transformers store them as surface-level word associations. Without a dedicated "concept space" that exists independently of word order, the model cannot bridge the gap between "A $\rightarrow$ B" and "B $\rightarrow$ A."

Consequences Across Domains

The implications of the Reversal Curse extend far beyond simple trivia:

Strategies for Mitigation

To overcome this limitation, researchers are moving toward training objectives that enforce symmetry:

  1. Bidirectional Training: Frameworks like BICO, introduced by Kaiyi Zhang, use bidirectional attention during fine-tuning to help models master reversed relationships.
  2. Latent Perturbation: Minki Kang developed LaPael (Latent Paraphrasing), which injects semantic noise into internal layers to simulate paraphrasing, encouraging the model to learn the underlying fact rather than the specific token order.
  3. Architectural Shifts: The Chiral Transformer uses Householder reflections to align subject and object representations, while Dong-Kyum Kim proposes bilinear relational structures to ensure that knowledge edits propagate logically in both directions.

Ultimately, solving the Reversal Curse requires moving beyond next-token prediction toward architectures that treat information as a network of symmetric, logical associations rather than a one-way street of text.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bidirectional_logical_generalization_in__d13df6b1,
  title = {Bidirectional Logical Generalization in Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bidirectional-logical-generalization-in-large-language-models/rdr_d13df6b115ec4613a80a25b1a43e8ed5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_bidirectional_logical_generalization_in__d13df6b1_v1,
  title = {Bidirectional Logical Generalization in Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bidirectional-logical-generalization-in-large-language-models/rdr_d13df6b115ec4613a80a25b1a43e8ed5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}