If you tell a standard Large Language Model (LLM) that "Olaf Scholz was the ninth Chancellor of Germany," it will likely answer correctly when asked who the ninth chancellor was. However, if you ask "Who was Olaf Scholz?" in the context of German chancellors, the model may struggle or fail entirely if it hasn't seen that specific phrasing.
This phenomenon, famously dubbed the Reversal Curse by Lukas Berglund, Meg Tong, and colleagues in The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A", reveals a fundamental blind spot in how modern AI learns. Despite their scale, models trained to predict the next token are inherently "one-way" learners.
The Factorization of Knowledge
To understand why this happens, we must look at the training objective. Standard LLMs use Next-Token Prediction (NTP). Mathematically, they factorize the joint probability of a sequence $P(w_1, w_2, ..., w_n)$ into a chain of conditional probabilities: $P(w_1) \cdot P(w_2|w_1) \cdot P(w_3|w_1, w_2)$, and so on.
As Diane Bouchacourt and Adina Williams argue in The Factorization Curse, this specific left-to-right factorization is not a neutral choice. It creates a directional bias where the model learns to predict the future from the past but never the past from the future. When a model learns "A is B," it optimizes the probability of seeing "B" given the context of "A." It is never incentivized to establish the reciprocal connection—that "A" should follow "B."
The Mechanics of One-Way Gradients
The failure isn't just a lack of data; it's a byproduct of how neural networks update their weights. In Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics, Baihe Huang and Stuart D. Russell provide a theoretical proof for this asymmetry.
During autoregressive training on the sequence "A is B," the gradient updates modify the model's internal representations so that the prefix "A" triggers the completion "B." Because the causal mask in the Transformer architecture prevents the second token ("B") from serving as context for the first ("A") during a single forward pass, the model strengthens the mapping from $A \rightarrow B$ without ever touching the reciprocal $B \rightarrow A$ weights. As Lukas Berglund notes, the representation of the subject is updated to include the object, but the object's representation remains unchanged regarding the subject.
Beyond Facts: Reasoning and Code
This directional bias extends far beyond simple trivia. It affects the very nature of machine "reasoning."
- Analogy Solving: Claire E Stevenson and Melanie Mitchell found that LLMs struggle with abstract analogies that require bidirectional navigation of sequences, a task children perform with ease. The models are "right-handed," proficient at identifying what comes next but poor at identifying what came before.
- Code Understanding: In software development, definitions usually precede usage. Danning Xie and colleagues demonstrated in the CORE benchmark that LLMs are significantly better at identifying forward dependencies than "backward" ones (where a later statement influences an earlier one, such as in loops).
- Retrieval Asymmetry: Even on simple copying tasks, Yana Veitsman shows in Born a Transformer -- Always a Transformer? that pretraining creates a massive performance gap between forward-looking induction and backward-looking retrieval.
Restoring Symmetry
Researchers are exploring several avenues to break this "curse" and move toward more human-like, bidirectional understanding.
One straightforward approach is Reverse Training. Olga Golovneva and Jason Weston propose training models on both forward and reversed versions of the same data. By doubling the tokens and providing both $A \rightarrow B$ and $B \rightarrow A$ sequences, the model is forced to learn bidirectional relationships. Similarly, Xunjian Yin introduced LEDOM, a model trained purely in reverse, which can act as a verifier for standard forward models by checking if a generated answer can "reconstruct" the original question.
Other methods target the architecture or the objective itself. Kaiyi Zhang and Ji-Rong Wen developed BICO, a fine-tuning framework that uses bidirectional attention to bridge the logic gap in causal models. Interestingly, Sangwoo Shin found that Masked Diffusion Models are naturally more resilient to the reversal curse because their "any-order" training and encoder-style architecture create a structural coupling between tokens, regardless of their sequence position.
Ultimately, the research suggests that while next-token prediction is a powerful engine for generating fluent text, it is an incomplete foundation for logical symmetry. To truly "know" a fact, a model must be able to access it from any direction.