RL Fine-Tuning Heals OOD Forgetting in SFT

Hangzhan Jin, Sitao Luan, Sicheng Lyu, Guillaume Rabusseau, Reihaneh Rabbany, Doina Precup, Mohammad Hamdaqa

Introduction

The standard pipeline for training modern Large Language Models (LLMs) follows a well-established two-stage process: Supervised Fine-Tuning (SFT) on high-quality demonstrations, followed by Reinforcement Learning (RL) to align the model with human preferences or specific rewards. While this combination is highly effective, the underlying mechanics have often been reduced to the simple mantra: "SFT memorizes, while RL generalizes."

This paper challenges that oversimplification. By investigating the internal dynamics of models like LLaMA-3.2-11B and Qwen-2.5-7B, the researchers reveal a phenomenon termed "OOD forgetting." They demonstrate that SFT often causes models to lose their out-of-distribution (OOD) reasoning capabilities long before training loss indicates any issues. Crucially, the study finds that RL does not necessarily grant models new "generalization magic"; instead, it acts as a healer, restoring the OOD capabilities that were inadvertently discarded during the SFT phase.

The Silent Failure of SFT

A critical finding of this research is that OOD performance is a fragile metric during supervised training. In tasks requiring robust reasoning—such as spatial navigation or complex arithmetic—OOD performance typically peaks very early in the SFT process. For example, in the "GeneralPoints" arithmetic task, LLaMA-3.2 reached its OOD peak around checkpoint 140 before its performance cratered.

This "OOD forgetting" is particularly dangerous because it is invisible to standard monitoring tools. While OOD accuracy drops, both In-Distribution (ID) accuracy and standard test loss curves continue to improve or remain stable. The model essentially over-specializes on the specific formatting and superficial patterns of the SFT data, sacrificing the flexible reasoning it possessed at the start of the fine-tuning process.

The Geometry of Forgetting: Rotation vs. Magnitude

To understand why this forgetting occurs, the researchers employed Singular Value Decomposition (SVD) to dissect the model's weight matrices. They focused on two components:

  1. Singular Values: Representing the magnitude or importance of specific features.
  2. Singular Vectors: Representing the direction or orientation of those features.

Contrary to the belief that fine-tuning significantly alters the importance of different features, the study found that singular values remained remarkably stable, fluctuating by a mere 0.005. Instead, the primary changes occurred in the singular vectors. SFT aggressively rotates these vectors toward the specific distribution of the training data. This rotation re-orients the model's internal representations to favor the SFT format, which simultaneously breaks the alignment required for OOD tasks.

RL as a Healer

The paper posits the "Healer Hypothesis": RL's primary contribution to generalization is the restoration of these rotated vectors. In the GeneralPoints task, RL (via PPO) was able to pull OOD accuracy back from near-zero to levels matching the early-SFT peak.

By analyzing the statistics of reward signals, the researchers found that RL acts as a "soft" corrective force. It re-aligns the singular vectors back toward a more robust configuration—similar to the state of the model during its early-SFT peak—without destroying the ID accuracy gained during the later stages of SFT. To prove this, the researchers performed weight editing experiments, manually rolling back the top singular vectors of a late-stage SFT model to their early-SFT state. This manual restoration successfully recovered OOD performance, confirming that the geometric orientation of weights is the key driver of OOD robustness.

The RL Boundary

RL is not a universal fix; its effectiveness depends heavily on the state of the model post-SFT. The researchers identified a "Goldilocks zone" for RL intervention:

  • Too Early: If SFT is stopped too soon (ID accuracy below 40%), the model has not yet learned the basic task format, leaving RL with no foundation to build upon.
  • Too Late: If SFT is allowed to run too long (ID accuracy near 100%), the model becomes "stiff" or over-specialized. In this state, the singular vectors are so heavily rotated that standard RL fails to move the needle.

The optimal window for RL effectiveness generally occurs when ID accuracy is between 40% and 80%.

Conclusion and Future Directions

This study shifts the understanding of the SFT-RL relationship from one of "learning vs. refining" to one of "specialization vs. restoration." By identifying that OOD forgetting is a matter of vector rotation rather than feature magnitude, the authors open the door for more efficient training methods. Future work may involve "rotation-aware" SFT—regularization techniques that penalize excessive singular vector shifts—potentially preventing OOD forgetting from occurring in the first place and reducing the reliance on computationally expensive RL stages.

Summary updated

Abstract

The two-stage fine-tuning paradigm of Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) has empirically shown better reasoning performance than one-stage SFT for the post-training of Large Language Models (LLMs). However, the evolution and mechanism behind the synergy of SFT and RL are still under-explored and inconclusive. To figure out this issue, we dissect the Out-Of-Distribution (OOD) and In-Distribution (ID) reasoning performance of LLaMA-3.2-11B and Qwen-2.5-7B at different checkpoints of the fine-tuning (full-parameter, rather than LoRA) process, and conduct fine-grained analysis. We find the well-known claim "SFT memorizes, RL generalizes" is over-simplified, and discover that: (1) OOD performance peaks at the early stage of SFT and then declines (OOD forgetting), the best SFT checkpoint cannot be captured by training/test loss; (2) the subsequent RL stage does not generate fundamentally better OOD capability, instead it plays an OOD restoration role, recovering the lost reasoning ability during SFT; (3) The recovery ability has boundaries, i.e., if SFT trains for too short or too long, RL cannot recover the lost OOD ability; (4) To uncover the underlying mechanisms behind the forgetting and restoration process, we employ SVD analysis on parameter matrices, manually edit them, and observe their impacts on model performance. Unlike the common belief that the shift of model capacity mainly results from the changes of singular values, we find that they are actually quite stable throughout fine-tuning. Instead, the OOD behavior strongly correlates with the rotation of singular vectors. In a nutshell, SFT performs hard alignment of the crucial parameter directions to the target tasks, leading to rapid and greedy adjustment, but also quick forgetting; RL then conditionally re-aligns singular vectors softly and slowly towards a more robust configuration, healing the forgetting and learning the downstream tasks simultaneously. Our findings re-identify the roles of SFT and RL in the two-stage fine-tuning and discover the rotation of singular vectors as the key mechanism.
BibTeX
@misc{jin2025rl,
  author = {Hangzhan Jin and Sitao Luan and Sicheng Lyu and Guillaume Rabusseau and Reihaneh Rabbany and Doina Precup and Mohammad Hamdaqa},
  title = {{RL Fine-Tuning Heals OOD Forgetting in SFT}},
  year = {2025},
  eprint = {2509.12235},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.12235},
}
Figure