On Reward Maximization and Distribution Matching for Fine-Tuning Language Models

Tomasz Korbak, Hady Elsahar, Germán Kruszewski, Marc DymetmanICLR 2022 · OpenReview

TL;DR

This paper bridges the gap between two popular methods for fine-tuning language models: Reward Maximization (RM) and Distribution Matching (DM). By showing that RM techniques like KL-control can be viewed through a DM lens, the authors introduce a novel approach that leverages the strengths of both, leading to more stable and efficient fine-tuning with improved control over generated text.

What Problem Does This Paper Address?

Large pre-trained language models (LLMs) are powerful but often need to be adapted for specific downstream tasks. Two main paradigms exist for this fine-tuning: Reward Maximization (RM), which uses reinforcement learning to maximize a reward signal for desired outputs, and Distribution Matching (DM), which aims to make the model's output distribution match a target distribution. While RM is good at nudging models, it can lead to "catastrophic forgetting" of the original model's capabilities. DM, on the other hand, can better preserve the original distribution but has a less developed set of optimization tools. This paper addresses the challenge of effectively fine-tuning LLMs to satisfy specific preferences without sacrificing their general capabilities, aiming to combine the benefits of both RM and DM.

Methods

The core contribution of this paper is to reveal the deep connections between RM and DM. They show that KL-control, a technique often framed within RM, can be reinterpreted as a form of DM. This insight allows them to import a key concept from RM – baselines – into DM methods. Baselines are used in reinforcement learning to reduce the variance of gradient estimates, leading to more stable and efficient training. The authors propose and empirically validate adding a baseline to Distributional Policy Gradients (DPG), a prominent DM algorithm. This new method, termed GDC++, aims to improve the stability and sample efficiency of DM-based fine-tuning.

Datasets

The experiments were conducted using pre-trained GPT-2 small models. For training the language models used in experiments, the authors utilized:

  • Python150 dataset: A collection of 150k Python source code files.
  • WikiBio dataset: Used for fine-tuning some of the models.

The evaluation tasks involved generating text with controlled attributes such as topic, sentiment, and gender.

Key Insights and Ideas

  • RM and DM are not mutually exclusive: The paper demonstrates that techniques like KL-control, traditionally seen as RM, can be understood as a form of Distribution Matching. This reframing opens doors for cross-pollination of ideas between the two paradigms.
  • Baselines are crucial for DM stability: By importing the concept of baselines from RM into DM (specifically DPG), the authors show a significant improvement in training stability and sample efficiency, especially when using smaller batch sizes. This addresses a key challenge in DM methods.
  • Controllable generation benefits from unified approaches: The paper highlights that methods that explicitly model target distributions (DM) are better suited for complex distributional constraints (e.g., controlling the proportion of female mentions in generated text) compared to pure RM.

Key Results

The proposed GDC++ method, which incorporates a baseline into the GDC framework, demonstrated superior performance across various controllable language generation tasks:

  • Improved Constraint Satisfaction: GDC++ achieved higher rates of satisfying desired constraints (e.g., topic, sentiment, gender) compared to existing methods.
  • Enhanced Stability: The addition of a baseline led to smoother training curves and more stable performance, particularly with smaller batch sizes.
  • Increased Sample Efficiency: GDC++ required fewer training samples to reach desired performance levels.
  • Reduced Divergence: The method maintained closer adherence to both the target distribution (p) and the original pre-trained model (α), mitigating catastrophic forgetting.
  • Specific Example: In the "amazing" word constraint task, GDC++ achieved a Ctrl. score of 0.69, significantly better than Reinforce (1.00) and Ziegler (0.82), while maintaining low KL divergences.

Limitations and Future Work

  • Optimal Baseline: While the proposed baseline is effective, the paper notes that it's not necessarily the optimal constant baseline for all scenarios, suggesting further research into optimal baseline selection.
  • Generalizing RL Techniques: The authors suggest that other variance reduction techniques from RL, such as learned value functions or trust-region methods, could be explored for DM.
  • EBM Characterization: Further analysis is needed to understand the relationship between explicit EBMs used in DPG and implicit EBMs that arise in KL-control, and to characterize the space of EBMs reachable via KL-control.

Summary updated

Abstract

The availability of large pre-trained models is changing the landscape of Machine Learning research and practice, moving from a "training from scratch" to a "fine-tuning'' paradigm. While in some applications the goal is to "nudge'' the pre-trained distribution towards preferred outputs, in others it is to steer it towards a different distribution over the sample space. Two main paradigms have emerged to tackle this challenge: Reward Maximization (RM) and, more recently, Distribution Matching (DM). RM applies standard Reinforcement Learning (RL) techniques, such as Policy Gradients, to gradually increase the reward signal. DM prescribes to first make explicit the target distribution that the model is fine-tuned to approximate. Here we explore the intimate connections between the two paradigms and show that methods such as KL-control developed in the RM paradigm can also be construed as belonging to DM. We further observe that while DM differs from RM, it can suffer from similar training difficulties, such as high gradient variance. We leverage connections between the two paradigms to import the concept of baseline into DM methods. We empirically validate the benefits of adding a baseline on an array of controllable language generation tasks such as constraining topic, sentiment, and gender distributions in texts sampled from a language model. We observe superior performance in terms of constraint satisfaction, stability, and sample efficiency.
BibTeX
@inproceedings{korbak2022reward,
  author = {Tomasz Korbak and Hady Elsahar and Germán Kruszewski and Marc Dymetman},
  title = {{On Reward Maximization and Distribution Matching for Fine-Tuning Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=8f95ajHrIFc},
}
Figure