Abstract
Large Language Models (LLMs) exhibit a fundamental limitation known as the "Reversal Curse": learning that "A has feature B" does not automatically enable the model to infer that "B is a feature of A." This paper proposes reverse training, a novel approach where models are trained on both forward and strategically reversed versions of the same data. By treating reversed text as a "second language," this method allows models to learn bidirectional relationships, effectively doubling training tokens without compromising standard performance.
The Reversal Curse
Despite their scale, LLMs are autoregressive "one-way" learners. They predict the next token from left to right, which often prevents them from generalizing facts in reverse. This is particularly evident with rare facts that appear only a few times in a corpus, typically in a single direction. While humans intuitively understand relational equivalence (if A=B, then B=A), LLMs treat these as distinct, unrelated sequences.
As shown in Figure 1, even advanced models like GPT-4 and Llama-2 struggle with simple reversals of well-known sequences, such as the lines of the U.S. national anthem.
Methods: Bidirectional Learning
The authors introduce four types of reverse training transformations to "nurse" this curse. In each case, the model is trained on both the original string and its reversed counterpart.
- Token Reversal: Reversing the order of individual tokens (BPE units).
- Word Reversal: Reversing the order of words while maintaining the internal tokenization of each word.
- Entity-Preserving Reversal: This method identifies entities (e.g., "Mary Lee Pfeiffer") and preserves their internal word order while reversing the rest of the sentence. This prevents the model from having to learn "mangled" versions of names.
- Random Segment Reversal: The sequence is divided into random chunks of 1 to $k$ tokens. The internal order of each chunk is kept, but the sequence of chunks is reversed, separated by a special
[REV]token.
Figure 2 illustrates how these transformations maintain the "units of knowledge" (entities) while reversing the overall structure, facilitating the model's ability to bridge the directional gap.
Key Insights
The effectiveness of reverse training rests on several core ideas:
- The "Second Language" Hypothesis: Reversed text acts like a different language. Since LLMs can transfer knowledge across languages (e.g., English to French), reverse training forces a similar transfer from "Reverse-English" to "Forward-English."
- Entity Integrity: Simple reversal often fails because it mangles names (e.g., "Isaac Newton" becomes "Newton Isaac"). Entity-preserving and random segment reversals solve this by keeping these knowledge units intact.
- Data-Bound Efficiency: In data-scarce scenarios, training once on forward data and once on reversed data is significantly more effective than training on the same forward data for two epochs.
Results and Performance
The researchers evaluated reverse training across symbolic tasks, biographies, and real-world celebrity data.
- Symbolic Reversal: Standard training models achieved 0% accuracy, while entity-preserving reverse training reached 100% accuracy.
- Real-world Knowledge: On a celebrity parent task using a 1.4B Llama-2 model, entity-preserving reversal improved "Parent → Celebrity" accuracy from 0.9% to 10.4% (Best@10).
- General Performance: Crucially, reverse training does not degrade performance on standard benchmarks. On evaluations like MMLU, reverse-trained models slightly outperformed data-matched standard models (48.8 vs 47.5 average).
Conclusion
Reverse training provides a robust framework for overcoming the directional limitations of autoregressive models. By making the training process inherently bidirectional, models can better internalize facts regardless of their presentation order. Future work will explore scaling these methods to larger models and more complex logical reasoning tasks.
Summary updated