When Inverse Data Outperforms: Exploring the Pitfalls of Mixed Data in Multi-Stage Fine-Tuning

Mengyi Deng, Xin Li, Tingyu Zhu, Zhicheng Yang, Zhijiang Guo, Wei Wang

TL;DR

This paper investigates the impact of mixing forward and reverse reasoning data during LLM fine-tuning. They find that while reverse reasoning data (r1k) improves performance, simply mixing it with forward data degrades performance due to conflicting supervision signals. They propose that better alignment strategies are needed to handle such mixed-direction reasoning.

What Problem Does This Paper Address?

Large Language Models (LLMs) have shown impressive reasoning capabilities, often achieved through fine-tuning on small, high-quality datasets. However, most existing methods focus on unidirectional reasoning, where models learn to solve problems step-by-step from question to answer (e.g., calculating kinetic energy from temperature). This overlooks the inherently bidirectional nature of human cognition, where we often reason backward from a desired outcome to known facts (e.g., inferring temperature from kinetic energy). The paper aims to explore whether incorporating this "backward" or "reverse" reasoning supervision can enhance LLM performance and generalization, and critically, what happens when this reverse data is mixed with traditional forward reasoning data during fine-tuning.

Methods

The core of the paper's methodology involves:

  • Constructing a Reverse Reasoning Dataset (r1k): The authors generated a new dataset, r1k, by inverting 1,000 forward reasoning examples from the existing s1k dataset. This inversion process involved using the DeepSeek-R1 model to generate reverse questions and their corresponding chain-of-thought (CoT) solutions. This approach avoids expensive manual data collection.
  • Multi-Stage Fine-Tuning: They employed a two-stage fine-tuning pipeline:
  • Supervised Fine-Tuning (SFT): Models were initially fine-tuned using standard cross-entropy loss on different combinations of forward (s1k), reverse (r1k), and mixed datasets.
  • Direct Preference Optimization (DPO): Following SFT, DPO was applied to further align model responses. This involved creating preference pairs where the model learns to favor the correct directional response (forward or reverse, depending on the original question) over the incorrect one.
  • Analysis of Learning Dynamics: The authors tracked Average Log Probability (ALP) of preferred (y+) and less preferred (y-) responses during fine-tuning. They also analyzed the margin between these ALPs (ALP(y+) - ALP(y-)) as a proxy for how well the model distinguishes between correct and incorrect reasoning paths, and how it handles hallucinations.

Datasets

  • s1k: An existing dataset of 1,000 high-quality forward reasoning examples used as a baseline and for constructing the new dataset.
  • r1k: A newly constructed dataset of 1,000 reverse reasoning examples, created by inverting s1k examples.
  • Mixed Datasets (D, D0.5k): Datasets formed by combining s1k and r1k in varying proportions (e.g., D is the full union, D0.5k is 500 forward and 500 reverse).
  • Evaluation Benchmarks: AIME24-NoFigures, Math 500, and GPQA were used to evaluate downstream performance.

Key Insights and Ideas

  • Reverse Data Boosts Performance: Fine-tuning solely on the r1k dataset leads to significant performance improvements (1.6%-6.8%) compared to using only the forward s1k dataset. This highlights the value of bidirectional reasoning supervision.
  • Mixing Data is Problematic: Simply mixing forward and reverse data during SFT leads to a degradation in performance. This is attributed to conflicting supervision signals that confuse the model and weaken its ability to distinguish between reasoning directions.
  • DPO Partially Helps, But Not Enough: DPO can partially recover the directional distinction lost during mixed-data SFT. However, it struggles with suboptimal initialization and can still suppress less preferred reasoning paths in a way that shifts probability mass towards irrelevant outputs.
  • Conflicting Signals Hinder Learning: The paper hypothesizes that mixed data forces the model to optimize in competing directions, hindering the formation of coherent preferences and effective learning. This explains why smaller, consistently supervised datasets can sometimes outperform larger, mixed ones.
  • Teacher Model Matters: The choice of teacher model for generating the initial reasoning data (e.g., Gemini vs. DeepSeek-R1) impacts downstream accuracy, with DeepSeek-R1 showing slightly better results.

Key Results

  • SFT on r1k improved accuracy by 1.6%-6.8% over s1k across evaluated benchmarks.
  • Mixing forward and reverse data during SFT resulted in a significant drop in performance, with degradation increasing as the proportion of mixed data increased.
  • DPO applied after mixed-data SFT showed some performance improvements but remained inferior to SFT trained solely on r1k.
  • Analysis of Average Log Probability (ALP) showed that models trained on r1k exhibited lower hallucination rates and better generalization under out-of-distribution scenarios.

Limitations and Future Work

  • Automated Data Generation: The r1k dataset was generated automatically, which might introduce subtle errors or inconsistencies in reasoning quality. Human validation is suggested for future work.
  • Strict Directional Preference: The DPO formulation assumes a strict preference between forward and reverse outputs, which might oversimplify scenarios where both directions offer complementary insights.
  • Alternative Alignment Strategies: The authors suggest that alternative alignment strategies beyond the standard SFT+DPO pipeline might be more robust for handling conflicting supervision in mixed-data settings. Future work could explore these.

Summary updated

Abstract

Existing work has shown that o1-level performance can be achieved with limited data distillation, but most existing methods focus on unidirectional supervised fine-tuning (SFT), overlooking the intricate interplay between diverse reasoning patterns. In this paper, we construct r1k, a high-quality reverse reasoning dataset derived by inverting 1,000 forward examples from s1k, and examine how SFT and Direct Preference Optimization (DPO) affect alignment under bidirectional reasoning objectives. SFT on r1k yields a 1.6%--6.8% accuracy improvement over s1k across evaluated benchmarks. However, naively mixing forward and reverse data during SFT weakens the directional distinction. Although DPO can partially recover this distinction, it also suppresses less preferred reasoning paths by shifting the probability mass toward irrelevant outputs. These findings suggest that mixed reasoning data introduce conflicting supervision signals, underscoring the need for robust and direction-aware alignment strategies.
BibTeX
@inproceedings{deng2025when,
  author = {Mengyi Deng and Xin Li and Tingyu Zhu and Zhicheng Yang and Zhijiang Guo and Wei Wang},
  title = {{When Inverse Data Outperforms: Exploring the Pitfalls of Mixed Data in Multi-Stage Fine-Tuning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure