DLoFT: Gradient-Decoupled Fine-Tuning for Generalizable Long Chain-of-Thought Reasoning

Sitong Wu, Haoru Tan, Jingyao Li, Shaofeng Zhang, Xiaojuan Qi, Bei Yu, Jiaya JiaNeurIPS 2025 · OpenReview

TL;DR

DLoFT (Decoupled LongCoT Fine-Tuning) is a novel training algorithm designed to prevent Large Language Models (LLMs) from "memorizing" specific facts during reasoning tasks. By mathematically separating the process of thinking (the reasoning paradigm) from the content of the problem (the specific facts), DLoFT allows models to learn generalizable reasoning skills that work across diverse, unseen domains. In evaluations, it outperforms standard fine-tuning by over 16 points on out-of-distribution tasks.

The Overfitting Problem in Long-Form Reasoning

The recent success of models like OpenAI’s o1 has popularized Long Chain-of-Thought (LongCoT)—a paradigm where models engage in a systematic "thinking" phase (reflecting, correcting errors, and exploring alternatives) before providing a final answer.

To replicate this capability in open-weight models, researchers typically use Supervised Fine-Tuning (SFT) on high-quality reasoning traces. However, the authors identify a major pitfall in this approach: overfitting to problem-specific knowledge.

When a model is fine-tuned on complex math problems, for example, it doesn't just learn how to reason; it memorizes specific math heuristics and facts. When you evaluate that model on a different field like medicine or law, its performance collapses because it hasn't internalized the general logic of long-form reasoning—only the content of the math training data. Standard fixes like "early stopping" fall short because they often halt training before the model can fully learn the LongCoT behavior.

Comparison of SFT and DLoFT training dynamics, showing how standard SFT overfits and degrades out-of-domain accuracy while DLoFT maintains it.

As seen in the training dynamics, there is a specific "turning point" during standard SFT where in-distribution accuracy keeps rising while out-of-distribution accuracy begins to plummet. This confirms the model is trading general logic for specific memorization.

Methods: Decoupling the Gradient

To solve this, the authors propose DLoFT, which decouples the training gradient into two orthogonal components:

  1. Paradigm-relevant gradient ($g_{par}$): Captures the general "how-to-think" structure of LongCoT.
  2. Content-relevant gradient ($g_{con}$): Reflects the problem-specific knowledge and facts.

The key insight is leveraging the two-phase structure of LongCoT data. A typical response consists of an exploratory Thinking Phase followed by a deterministic Solution Phase. The authors argue that while the Thinking Phase contains both the reasoning paradigm and specific content, the Solution Phase primarily encodes the specific content and facts.

Vector diagram illustrating the decoupling of the full gradient into paradigm-relevant and content-relevant components.

The DLoFT Algorithm: * Step 1: Compute the full gradient ($g_{full}$) using the entire response (Thinking + Solution). * Step 2: Compute a "reference" gradient ($g_{ref}$) using only the Solution Phase. * Step 3: Use a projection operation to isolate the component of the full gradient that aligns with the reference (the content-relevant gradient, $g_{con}$). * Step 4: Use orthogonalization (subtracting the content component from the full gradient) to isolate the "pure" reasoning paradigm gradient ($g_{par}$). * Step 5: Update the model parameters using only this purified reasoning gradient.

By forcing the update to be orthogonal to the content-specific gradient, DLoFT ensures the model's weights change in a way that favors the structure of the response (e.g., "Wait, let me re-check that step") rather than the facts (e.g., "The derivative of $x^2$ is $2x$").

Datasets and Evaluation

The researchers tested DLoFT across 20 different domains. Training utilized the s1K dataset (1,000 high-quality mixed-domain samples) alongside domain-specific 5K subsets from OpenR1-Math, OpenThoughts-Code, and Medical-o1.

Evaluation was conducted across a massive battery of benchmarks including AIME24 (math), LiveCodeBench (coding), MedQA (medicine), and SuperGPQA (graduate-level reasoning across STEM, social sciences, and humanities).

Key Results

  • Superior Generalization: On generalist models (Qwen2.5-7B/32B), DLoFT outperformed standard SFT by an average of +16.4 points on out-of-distribution domains.
  • In-Domain Gains: Even on the topics it was explicitly trained on, DLoFT saw an average improvement of +11.5 points. This suggests that focusing purely on the reasoning paradigm actually helps the model solve the original problems more effectively.
  • Better RL Foundation: DLoFT is particularly effective as a "cold-start" for Reinforcement Learning (RL). It provides a more stable foundation for subsequent RL training compared to standard SFT, which can bake in bad habits or narrow knowledge.
  • Efficiency: DLoFT adds negligible overhead. It only requires one extra (shorter) backward pass for the solution phase and a simple vector subtraction, making it a highly practical drop-in replacement for SFT.

Limitations and Future Work

While highly effective, the authors note a few limitations. The experiments were conducted on 7B and 32B parameter models, leaving its efficacy at "frontier" scales (hundreds of billions of parameters) untested due to resource constraints. Additionally, while DLoFT reduces the amount of data needed, it still fundamentally relies on high-quality LongCoT traces to define the reasoning paradigm.

Future directions include exploring how this gradient decoupling mechanism could be applied to other multi-step tasks beyond reasoning, such as long-form creative writing or complex tool-use.

Summary updated

Abstract

Long chain-of-thought (LongCoT) has emerged as a powerful reasoning paradigm for enabling large language models (LLMs) to solve complex tasks through a systematic and thorough thinking phase. Although supervised fine-tuning (SFT) on high-quality LongCoT traces has proven effective to activate LongCoT abilities, we find that models trained in this way tend to overfit problem-specific knowledge and heuristics, leading to degraded out-of-distribution performance. To address this issue, we propose a Decoupled LongCoT Fine-Tuning (DLoFT) algorithm, which enables the model to learn generalizable LongCoT reasoning abilities while preventing overfitting to the reasoning content with problem-specific information. The key idea is to decouple the gradient into two orthogonal components: 1) a paradigm-relevant gradient corresponding to the general LongCoT paradigm and 2) a content-relevant gradient reflecting the problem-specific information, where only the former gradient is used to update model parameters. Specifically, by leveraging the unique two-phase composition (thinking and solution) of the LongCoT response, our gradient decoupling mechanism isolates the content-relevant gradient via a projection operation and separates the paradigm-relevant gradient through orthogonalization. Our DLoFT ensures the model concentrate on internalizing the LongCoT paradigm rather than memorizing problem-specific knowledge and heuristics. Extensive experiments demonstrate that our DLoFT significantly improves the generalization behavior of LongCoT abilities compared to SFT while maintaining strong in-distribution performance.
BibTeX
@inproceedings{wu2025dloft,
  author = {Sitong Wu and Haoru Tan and Jingyao Li and Shaofeng Zhang and XIAOJUAN QI and Bei Yu and Jiaya Jia},
  title = {{DLoFT: Gradient-Decoupled Fine-Tuning for Generalizable Long Chain-of-Thought Reasoning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=5ZDT1dxojA},
}
Figure