TL;DR
This paper investigates the impact of mixing forward and reverse reasoning data during LLM fine-tuning. They find that while reverse reasoning data (r1k) improves performance, simply mixing it with forward data degrades performance due to conflicting supervision signals. They propose that better alignment strategies are needed to handle such mixed-direction reasoning.
What Problem Does This Paper Address?
Large Language Models (LLMs) have shown impressive reasoning capabilities, often achieved through fine-tuning on small, high-quality datasets. However, most existing methods focus on unidirectional reasoning, where models learn to solve problems step-by-step from question to answer (e.g., calculating kinetic energy from temperature). This overlooks the inherently bidirectional nature of human cognition, where we often reason backward from a desired outcome to known facts (e.g., inferring temperature from kinetic energy). The paper aims to explore whether incorporating this "backward" or "reverse" reasoning supervision can enhance LLM performance and generalization, and critically, what happens when this reverse data is mixed with traditional forward reasoning data during fine-tuning.
Methods
The core of the paper's methodology involves:
- Constructing a Reverse Reasoning Dataset (r1k): The authors generated a new dataset,
r1k, by inverting 1,000 forward reasoning examples from the existings1kdataset. This inversion process involved using the DeepSeek-R1 model to generate reverse questions and their corresponding chain-of-thought (CoT) solutions. This approach avoids expensive manual data collection. - Multi-Stage Fine-Tuning: They employed a two-stage fine-tuning pipeline:
- Supervised Fine-Tuning (SFT): Models were initially fine-tuned using standard cross-entropy loss on different combinations of forward (
s1k), reverse (r1k), and mixed datasets. - Direct Preference Optimization (DPO): Following SFT, DPO was applied to further align model responses. This involved creating preference pairs where the model learns to favor the correct directional response (forward or reverse, depending on the original question) over the incorrect one.
- Analysis of Learning Dynamics: The authors tracked Average Log Probability (ALP) of preferred (
y+) and less preferred (y-) responses during fine-tuning. They also analyzed the margin between these ALPs (ALP(y+) - ALP(y-)) as a proxy for how well the model distinguishes between correct and incorrect reasoning paths, and how it handles hallucinations.
Datasets
- s1k: An existing dataset of 1,000 high-quality forward reasoning examples used as a baseline and for constructing the new dataset.
- r1k: A newly constructed dataset of 1,000 reverse reasoning examples, created by inverting
s1kexamples. - Mixed Datasets (D, D0.5k): Datasets formed by combining
s1kandr1kin varying proportions (e.g.,Dis the full union,D0.5kis 500 forward and 500 reverse). - Evaluation Benchmarks: AIME24-NoFigures, Math 500, and GPQA were used to evaluate downstream performance.
Key Insights and Ideas
- Reverse Data Boosts Performance: Fine-tuning solely on the
r1kdataset leads to significant performance improvements (1.6%-6.8%) compared to using only the forwards1kdataset. This highlights the value of bidirectional reasoning supervision. - Mixing Data is Problematic: Simply mixing forward and reverse data during SFT leads to a degradation in performance. This is attributed to conflicting supervision signals that confuse the model and weaken its ability to distinguish between reasoning directions.
- DPO Partially Helps, But Not Enough: DPO can partially recover the directional distinction lost during mixed-data SFT. However, it struggles with suboptimal initialization and can still suppress less preferred reasoning paths in a way that shifts probability mass towards irrelevant outputs.
- Conflicting Signals Hinder Learning: The paper hypothesizes that mixed data forces the model to optimize in competing directions, hindering the formation of coherent preferences and effective learning. This explains why smaller, consistently supervised datasets can sometimes outperform larger, mixed ones.
- Teacher Model Matters: The choice of teacher model for generating the initial reasoning data (e.g., Gemini vs. DeepSeek-R1) impacts downstream accuracy, with DeepSeek-R1 showing slightly better results.
Key Results
- SFT on
r1kimproved accuracy by 1.6%-6.8% overs1kacross evaluated benchmarks. - Mixing forward and reverse data during SFT resulted in a significant drop in performance, with degradation increasing as the proportion of mixed data increased.
- DPO applied after mixed-data SFT showed some performance improvements but remained inferior to SFT trained solely on
r1k. - Analysis of Average Log Probability (ALP) showed that models trained on
r1kexhibited lower hallucination rates and better generalization under out-of-distribution scenarios.
Limitations and Future Work
- Automated Data Generation: The
r1kdataset was generated automatically, which might introduce subtle errors or inconsistencies in reasoning quality. Human validation is suggested for future work. - Strict Directional Preference: The DPO formulation assumes a strict preference between forward and reverse outputs, which might oversimplify scenarios where both directions offer complementary insights.
- Alternative Alignment Strategies: The authors suggest that alternative alignment strategies beyond the standard SFT+DPO pipeline might be more robust for handling conflicting supervision in mixed-data settings. Future work could explore these.
Summary updated