TL;DR
This paper bridges the gap between two popular methods for fine-tuning language models: Reward Maximization (RM) and Distribution Matching (DM). By showing that RM techniques like KL-control can be viewed through a DM lens, the authors introduce a novel approach that leverages the strengths of both, leading to more stable and efficient fine-tuning with improved control over generated text.
What Problem Does This Paper Address?
Large pre-trained language models (LLMs) are powerful but often need to be adapted for specific downstream tasks. Two main paradigms exist for this fine-tuning: Reward Maximization (RM), which uses reinforcement learning to maximize a reward signal for desired outputs, and Distribution Matching (DM), which aims to make the model's output distribution match a target distribution. While RM is good at nudging models, it can lead to "catastrophic forgetting" of the original model's capabilities. DM, on the other hand, can better preserve the original distribution but has a less developed set of optimization tools. This paper addresses the challenge of effectively fine-tuning LLMs to satisfy specific preferences without sacrificing their general capabilities, aiming to combine the benefits of both RM and DM.
Methods
The core contribution of this paper is to reveal the deep connections between RM and DM. They show that KL-control, a technique often framed within RM, can be reinterpreted as a form of DM. This insight allows them to import a key concept from RM – baselines – into DM methods. Baselines are used in reinforcement learning to reduce the variance of gradient estimates, leading to more stable and efficient training. The authors propose and empirically validate adding a baseline to Distributional Policy Gradients (DPG), a prominent DM algorithm. This new method, termed GDC++, aims to improve the stability and sample efficiency of DM-based fine-tuning.
Datasets
The experiments were conducted using pre-trained GPT-2 small models. For training the language models used in experiments, the authors utilized:
- Python150 dataset: A collection of 150k Python source code files.
- WikiBio dataset: Used for fine-tuning some of the models.
The evaluation tasks involved generating text with controlled attributes such as topic, sentiment, and gender.
Key Insights and Ideas
- RM and DM are not mutually exclusive: The paper demonstrates that techniques like KL-control, traditionally seen as RM, can be understood as a form of Distribution Matching. This reframing opens doors for cross-pollination of ideas between the two paradigms.
- Baselines are crucial for DM stability: By importing the concept of baselines from RM into DM (specifically DPG), the authors show a significant improvement in training stability and sample efficiency, especially when using smaller batch sizes. This addresses a key challenge in DM methods.
- Controllable generation benefits from unified approaches: The paper highlights that methods that explicitly model target distributions (DM) are better suited for complex distributional constraints (e.g., controlling the proportion of female mentions in generated text) compared to pure RM.
Key Results
The proposed GDC++ method, which incorporates a baseline into the GDC framework, demonstrated superior performance across various controllable language generation tasks:
- Improved Constraint Satisfaction: GDC++ achieved higher rates of satisfying desired constraints (e.g., topic, sentiment, gender) compared to existing methods.
- Enhanced Stability: The addition of a baseline led to smoother training curves and more stable performance, particularly with smaller batch sizes.
- Increased Sample Efficiency: GDC++ required fewer training samples to reach desired performance levels.
- Reduced Divergence: The method maintained closer adherence to both the target distribution (p) and the original pre-trained model (α), mitigating catastrophic forgetting.
- Specific Example: In the "amazing" word constraint task, GDC++ achieved a Ctrl. score of 0.69, significantly better than Reinforce (1.00) and Ziegler (0.82), while maintaining low KL divergences.
Limitations and Future Work
- Optimal Baseline: While the proposed baseline is effective, the paper notes that it's not necessarily the optimal constant baseline for all scenarios, suggesting further research into optimal baseline selection.
- Generalizing RL Techniques: The authors suggest that other variance reduction techniques from RL, such as learned value functions or trust-region methods, could be explored for DM.
- EBM Characterization: Further analysis is needed to understand the relationship between explicit EBMs used in DPG and implicit EBMs that arise in KL-control, and to characterize the space of EBMs reachable via KL-control.
Summary updated