Abstract
Reinforcement learning from human feedback (RLHF) has been extensively employed to align large language models with user intent. However, proximal policy optimization (PPO) based RLHF is occasionally unstable requiring significant hyperparameter finetuning, and computationally expensive to maximize the estimated reward during alignment. Recently, direct preference optimization (DPO) is proposed to address those challenges. However, DPO relies on contrastive responses generated from human annotator and alternative LLM, instead of the policy model, limiting the effectiveness of the RLHF. In this paper, we addresses both challenges by systematically combining rejection sampling (RS) and DPO. Our proposed method, RS-DPO, initiates with the development of a supervised fine-tuned policy model (SFT). A varied set of k responses per prompt are sampled directly from the SFT model. RS-DPO identifies pairs of contrastive samples based on their reward distribution. Finally, we apply DPO with the contrastive samples to align the model to human preference. Our experiments indicate that our proposed method effectively fine-tunes LLMs with limited resource environments, leading to improved alignment with user intent. Furthermore, it outperforms existing methods, including RS, PPO, and DPO.
Introduction
Aligning Large Language Models (LLMs) with human intent typically relies on Reinforcement Learning from Human Feedback (RLHF). The industry standard, Proximal Policy Optimization (PPO), is notoriously difficult to use: it is hyperparameter-sensitive, unstable, and computationally expensive because it requires keeping multiple models (policy, value, reward, and reference) in GPU memory simultaneously.
Direct Preference Optimization (DPO) emerged as a simpler alternative that eliminates the need for a reinforcement learning loop. However, DPO usually trains on static, "off-policy" datasets—preferences often generated by humans or other models. This creates a distribution shift; the model isn't learning from its own mistakes or successes, which limits the final alignment performance. RS-DPO aims to bridge this gap by providing DPO with high-quality, model-generated preference pairs.
Methodology
The RS-DPO pipeline consists of four main stages:
- Supervised Fine-Tuning (SFT): A base model is trained on high-quality instruction-following data.
- Reward Model (RM) Training: A separate model is trained to score responses based on human preferences.
- Preference Data Generation via Rejection Sampling (PDGRS): This is the core innovation. For every prompt in the training set, the SFT model generates $k$ (e.g., 16) different responses. The Reward Model then scores these. The authors create pairs of "winning" and "losing" responses $(y_w, y_l)$ only if the reward gap between them exceeds a specific threshold ($\eta$).
- DPO Optimization: The model is fine-tuned using the DPO objective on this newly created synthetic dataset.
By sampling from the SFT model itself, the preference data is "on-policy" relative to the starting point of the alignment phase, providing a much stronger training signal than static datasets.
Experimental Setup
The researchers evaluated their method using several prominent open-source datasets:
- Open Assistant (OASST1): Used for the initial SFT stage.
- Anthropic/HH-RLHF: Specifically the "helpfulness" subset, used for reward modeling and generating preference pairs.
- WebGPT: A long-form QA dataset used to test the method's robustness.
Evaluation was conducted using MT-Bench (a multi-turn conversation benchmark judged by GPT-4) and AlpacaEval (simulated win-rate against a baseline).
Key Results
RS-DPO consistently outperformed SFT, PPO, and standard DPO across benchmarks. As shown in the performance comparison for the Anthropic dataset, RS-DPO achieved superior scores in both MT-Bench and AlpacaEval.
- MT-Bench: On the Anthropic dataset, RS-DPO achieved a score of 5.49, surpassing PPO (5.22) and standard DPO (5.26).
- AlpacaEval: RS-DPO reached a win rate of 79.67%, significantly higher than PPO’s 69.23%.
- Data Augmentation: The PDGRS process significantly increased the volume of training data. For the Anthropic dataset, it turned ~10k prompts into over 63k high-quality preference pairs, leading to better generalization.
Discussion
A key insight from the study is the importance of the Reward Gap. Setting a threshold ($\eta$) for the reward difference is crucial. If the gap is too small, the model tries to learn from noisy, indistinguishable pairs. If it's too large, the data becomes too sparse. A sweet spot (around 0.85) maximizes performance.
Furthermore, RS-DPO demonstrates robustness to reward model quality. It is more resilient to "weaker" reward models than PPO. Even when using a reward model trained on a smaller, less diverse dataset, RS-DPO maintained high performance, whereas PPO’s performance degraded significantly. Finally, the efficiency via offline sampling allows for training on 1-2 moderate GPUs where PPO would be unfeasible, as sampling occurs once at the start rather than during the training loop.
Conclusion
RS-DPO offers a resource-efficient and stable alternative for LLM alignment. By combining rejection sampling with DPO, it leverages on-policy synthetic data to achieve state-of-the-art performance. Future work will explore scaling to larger models and iterative training cycles to further refine the alignment process.
Summary updated