TL;DR
The conventional wisdom that "Supervised Fine-Tuning (SFT) memorizes while Reinforcement Learning (RL) generalizes" is an oversimplification. Tracking full-parameter fine-tuning checkpoint-by-checkpoint reveals that out-of-distribution (OOD) reasoning peaks early in SFT and then degrades—a phenomenon termed OOD forgetting. RL does not endow models with fundamentally new OOD reasoning beyond the peak SFT checkpoint; rather, it acts as an automatic regularizer that "heals" the reasoning capabilities lost during late-stage SFT by softly re-aligning rotated singular vectors.
What Problem Does This Paper Address?
The standard recipe for post-training reasoning models (exemplified by DeepSeek-R1) is a two-stage pipeline: SFT warm-up followed by RL (such as PPO). The community often assumes that SFT merely fits the training distribution while RL provides the "magic" needed to unlock out-of-distribution generalization.
However, prior comparisons typically measure performance only at the end of SFT ($SFT_{End}$) versus the end of RL ($RL_{End}$). The authors investigate what actually happens across intermediate checkpoints:
- Does SFT genuinely fail to learn generalizable reasoning, or does it learn and then lose it?
- Does RL unlock capabilities beyond what SFT achieved at its peak?
- What parameter-space mechanisms drive these performance shifts?
Methods
- Dense Checkpoint Tracking: The authors conduct full-parameter SFT and RL (PPO) using LLaMA-3.2-11B and Qwen-2.5-7B, continuously evaluating in-distribution (ID) and out-of-distribution (OOD) reasoning across hundreds of checkpoints.
- RL Intervention Across the Timeline: Instead of applying RL only at the end of SFT, PPO is initialized from multiple SFT checkpoints to map out the "recovery boundary."
- Spectral Weight Surgery (SVD): The authors apply Singular Value Decomposition ($M = U\Sigma V^\top$) to the self-attention projection matrices ($W_Q, W_K, W_V$) and MLP matrices ($W_{MLP}$). By swapping singular values ($\Sigma$) or singular vector orientations ($U, V$) between early, late, and RL-tuned checkpoints, they isolate whether changes in magnitude or directional rotation drive OOD performance.
- Advantage & Distributional Analysis: The authors analyze the PPO advantage distribution (entropy, skewness, and KL-divergence against matched Gaussians) to pinpoint why RL only succeeds within specific SFT checkpoint windows.
Datasets
The authors test on three controlled environments designed to disentangle superficial format matching from true compositional reasoning:
- GeneralPoints (Arithmetic Reasoning): Based on the "24-point" poker game.
- ID: Face cards ('J', 'Q', 'K') are interpreted as 10.
- OOD: Face cards ('J', 'Q', 'K') are mapped to 11, 12, and 13.
- Navigation (Spatial Reasoning):
- ID: Models follow instructions using absolute compass headings (
turn direction(north)). - OOD: Models must output relative directions (
turn direction(right)). - Rank-Determinant Computation (Cross-Concept Math):
- ID: Training on computing matrix rank for non-square matrices (e.g., $4 \times 5$).
- OOD: Evaluating on computing matrix determinants for square matrices (e.g., $2 \times 2$).
Key Insights and Ideas
- SFT Suffers from "OOD Forgetting": Format learning converges very early (~checkpoint 50). Between checkpoints 50 and 140, genuine reasoning peaks ($SFT_{MaxOOD}$). Beyond that, OOD accuracy drops steadily even as ID loss continues to decrease and ID accuracy rises. Standard validation loss completely misses this OOD degradation.
- Token-Level Loss Penalizes Valid Novelty: In problems with multiple valid solutions, standard token-level cross-entropy assigns high loss to any valid formula not matching the specific ground-truth reference in the dataset. This greedy objective drives the model to over-specialize to dataset artifacts, inducing forgetting.
- RL Heals, but Has Boundaries: RL does not surpass the peak OOD accuracy seen at $SFT_{MaxOOD}$. Instead, RL acts like an automated regularizer (analogous to early stopping or dropout) that restores degraded performance back to near-peak levels. However, RL requires a balanced advantage signal: if applied too early (< 400 checkpoints) or too late (> 1200 checkpoints), policy updates collapse.
- Rotation Matters, Not Singular Values: Contrary to common assumptions that neural capacity shifts reside in singular values, the singular values barely fluctuate throughout fine-tuning ($\Delta \sigma \in [0, 0.005]$). Swapping singular values between checkpoints causes almost zero performance change. Model adaptation and OOD forgetting are almost exclusively driven by the rotation of singular vectors (primarily the top 512–1024 ranks).
Key Results
- OOD Accuracy Trajectory (GeneralPoints):
- LLaMA-3.2-11B reaches an OOD peak of ~17.5% at SFT checkpoint 140 ($SFT_{MaxOOD}$), degrades to ~6.5% at checkpoint 1100 ($SFT_{End}$), and recovers to ~16.5% after RL ($RL_{End}$).
- Qwen-2.5-7B reaches ~17.5% at checkpoint 120, falls to ~13% at checkpoint 800, and is restored to ~17% after RL.
- RL Effectiveness Window: RL successfully recovers OOD accuracy only when initialized from checkpoints where ID accuracy is between roughly 0.4 and 0.8 (advantage distribution entropy $> 2.55$ and KL divergence against a normal distribution $< 0.16$).
- Rotation-Aware SFT: Regularizing the rotation of the top-512 singular vectors during SFT significantly mitigates OOD forgetting across training without needing an RL stage.
Limitations and Future Work
- Task Scope: Experiments focus on controlled, rule-based algorithmic reasoning benchmarks (arithmetic, navigation, basic linear algebra). Validating whether this rotation mechanism and recovery ceiling hold on complex, natural-language math/coding benchmarks (e.g., MATH, LiveCodeBench) remains open.
- Geometric Mechanics of Rotation: While Principal Angle analysis (PABS) confirms that singular vectors rotate significantly in both SFT and RL, the exact geometric differences between "destructive SFT rotation" and "restorative RL rotation" in high-dimensional space remain unresolved.
- Model Scale: Studies are limited to 7B–11B parameter models; behaviors on larger frontier scales (>70B) are untested.
Summary updated