Learning Dynamics of LLM Finetuning

Yi Ren, Danica J. Sutherland

TL;DR

This paper introduces a formal framework to study the learning dynamics of LLM finetuning—specifically how training on one example shifts the model's predictions across the entire output space. The authors identify a counter-intuitive "squeezing effect" in Direct Preference Optimization (DPO) that causes the model to lose confidence in even desired outputs. To counter this, they propose a simple data-augmentation fix that significantly improves alignment performance.

What Problem Does This Paper Address?

While finetuning techniques like Supervised Finetuning (SFT) and Direct Preference Optimization (DPO) are industry standards, our understanding of why they behave the way they do is often limited to high-level intuition. For instance, why does SFT sometimes increase hallucinations? And why does DPO, if run for too long, cause the probability of the "chosen" response to actually decrease?

Most existing research looks at these algorithms through the lens of loss functions or reinforcement learning theory. This paper instead focuses on learning dynamics: the step-by-step evolution of the model’s parameters and predictions. By understanding the "mechanical" influence of a single gradient update, the authors aim to explain phenomena like model degeneration and repetitive outputs.

Methods: Deconstructing Influence

The authors derive a per-step influence decomposition for LLMs. They break down the change in a model's prediction into three interacting components:

  1. The Adaptation Vector ($A^t$): How the model's current confidence levels constrain future changes.
  2. The Empirical Neural Tangent Kernel ($K^t$): A measure of "similarity" between different inputs. If two prompts are "similar" in the eyes of the NTK, updating on one will strongly affect the other.
  3. The Residual Term ($G^t$): The "energy" and direction of the update, determined by the specific loss function (e.g., SFT vs. DPO).
Influence decomposition and learning dynamics on a simplified MNIST task.

The Squeezing Effect in DPO

A major contribution of this work is the characterization of the "Squeezing Effect." In DPO, the model receives a negative gradient on "rejected" responses. If these responses are already unlikely (located in a "valley" of the probability distribution), the negative update doesn't just lower their probability; it "squeezes" that probability mass into the most likely tokens. This often reinforces the model's prior biases or leads to repetitive, "degenerate" text.

Comparison of probability shifts in SFT vs. Off-policy DPO, highlighting the squeezing effect.

To fix this, the authors propose the "Extend" method: before running DPO, perform SFT on both the chosen and rejected responses. This pulls the rejected responses out of the "valleys," making the subsequent DPO updates more stable and less prone to the squeezing effect.

Key Insights and Ideas

  • Hallucinations as "Pull-up": The authors suggest that SFT can amplify hallucinations because learning a fact for Question A "pulls up" the probability of similar-looking phrases for Question B, even if those phrases are factually incorrect for B.
  • The DPO Confidence Paradox: In off-policy DPO, the model often becomes less confident in the preferred response over time. The framework explains this as a side effect of the negative pressure on rejected responses being "squeezed" into the wrong parts of the distribution.
  • On-policy is Naturally Safer: The analysis explains why on-policy DPO (sampling from the model itself) performs better: it ensures the model isn't applying massive negative gradients to "valley" regions, thereby avoiding the worst of the squeezing effect.

Datasets and Key Results

The researchers validated their framework and the "Extend" method using a simplified MNIST warm-up, the Anthropic-HH dataset, and UltraFeedback across various model scales (Pythia and Qwen1.5 families).

The proposed "Extend" method—a remarkably simple change to the training pipeline—yielded significant gains:

  • Win Rates: When evaluated by GPT-3.5-Turbo and Claude-3-Haiku, the "Extend" method consistently beat the standard SFT+DPO baseline.
  • Performance Peak: On the Anthropic-HH dataset, the "Extend" method achieved a ~69% win rate against the baseline after 4 epochs of DPO.
  • Stability: The method successfully mitigated the "degeneration" typically seen in long DPO runs, maintaining higher confidence in preferred outputs.

Limitations and Future Work

The authors acknowledge a few key areas for further study:

  • NTK Stability: The framework assumes the Neural Tangent Kernel remains relatively stable during finetuning. While they provide empirical evidence for this, it may not hold in all extreme training scenarios.
  • Token-wise vs. Sequence-wise: The current analysis focuses on the influence at the token level. Extending this to capture the full auto-regressive complexity of long-form generation is a noted next step.
  • Interaction of Pressures: The paper primarily focuses on the negative gradient's squeezing effect; the complex "tug-of-war" between simultaneous positive and negative updates in DPO warrants more granular mathematical modeling.

Summary updated

Abstract

Learning dynamics, which describes how the learning of specific training examples influences the model's predictions on other examples, gives us a powerful tool for understanding the behavior of deep learning systems. We study the learning dynamics of large language models during different types of finetuning, by analyzing the step-wise decomposition of how influence accumulates among different potential responses. Our framework allows a uniform interpretation of many interesting observations about the training of popular algorithms for both instruction tuning and preference tuning. In particular, we propose a hypothetical explanation of why specific types of hallucination are strengthened after finetuning, e.g., the model might use phrases or facts in the response for question B to answer question A, or the model might keep repeating similar simple phrases when generating responses. We also extend our framework and highlight a unique ``squeezing effect'' to explain a previously observed phenomenon in off-policy direct preference optimization (DPO), where running DPO for too long makes even the desired outputs less likely. This framework also provides insights into where the benefits of on-policy DPO and other variants come from. The analysis not only provides a novel perspective of understanding LLM's finetuning but also inspires a simple, effective method to improve alignment performance.
BibTeX
@inproceedings{ren2025learning,
  author = {Yi Ren and Danica J. Sutherland},
  title = {{Learning Dynamics of LLM Finetuning}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=tPNHOoZFl9},
}
Figure