TL;DR
This paper introduces Uncertainty-enhanced Preference Optimization (UPO), a novel framework to improve the self-evolution of Large Language Models (LLMs) by intelligently filtering noisy preference data. UPO leverages an uncertainty estimator to identify reliable feedback, leading to more robust and effective LLM alignment, as demonstrated by significant performance gains on various NLP and reasoning benchmarks.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly trained using preference optimization, a process that aligns models with human feedback. However, this iterative process often suffers from "noisy preference data" generated by the current policy and reward models, which can hinder performance. Existing methods for mitigating this noise, like simple filtering based on reward scores, can fall prey to confirmation bias. The paper tackles the challenge of making LLM self-evolution more robust and effective by developing a method to discern and utilize high-quality preference signals while downplaying unreliable ones.
Methods
The core of UPO is an Uncertainty-enhanced Preference Optimization (UPO) framework. Key components include:
- Estimator Model with MC Dropout: A novel estimator model, built upon a Bayesian Neural Network (BNN) approach using Monte Carlo (MC) dropout, is introduced. This model performs pairwise uncertainty estimation on preference data, providing a measure of confidence in the feedback.
- Uncertainty-Enhanced Self-Evolution Algorithm: This algorithm uses the uncertainty estimates to split preference data into reliable and unreliable pairs. It then integrates this uncertainty information into the Direct Preference Optimization (DPO) objective, encouraging the LLM to learn from both high-reward and high-certainty feedback.
- Iterative Preference Optimization: The framework operates iteratively, where the LLM policy generates new preference data, which is then filtered by the estimator and used to further train the policy.
Datasets
- UltraFeedback: Used for initial stage training and as a source of prompts in iterative stages.
- AlpacaEval 2.0: A benchmark for evaluating LLM performance on general NLP tasks.
- MT-Bench: Another benchmark for evaluating LLM capabilities across various NLP aspects.
- GSM8K: A dataset for evaluating mathematical reasoning.
- MATH: A dataset for evaluating mathematical reasoning.
- Math-Step-DPO-10K: A dataset specifically designed for fine-grained mathematical reasoning tasks.
- UltraChat200K: Used as a prompt set in the iterative stages.
Key Insights and Ideas
- Uncertainty is Key to Reliable Feedback: The paper strongly suggests that simply relying on reward scores is insufficient. Quantifying the uncertainty of the feedback is crucial for identifying truly informative preference signals.
- Self-Evolution Can Be Noisy: The iterative nature of LLM training, while powerful, can amplify noise if not carefully managed. UPO directly addresses this by actively filtering out unreliable data.
- Beyond Reward: Certainty Matters: UPO demonstrates that LLMs can benefit from learning not just what is preferred, but also how certain that preference is, leading to more robust and higher-quality outputs.
- Iterative Improvement Outperforms Merging: The results show that iterative evolution (UPO) is more effective than simply merging all generated data at the end (UPO-Merge), highlighting the benefit of continuous refinement.
Key Results
- AlpacaEval 2.0: UPO achieved the best win rate against GPT-4, improving by 7.20% over SFT and 3.92% over DPO after three iterations.
- MT-Bench: UPO obtained the highest score, improving the LLM policy from 6.79% to 7.02%.
- GSM8K & MATH: UPO consistently achieved substantial improvements, reaching 88.9% accuracy on GSM8K and 56.3% on MATH.
- Ablation Studies: Removing the uncertainty estimator or the uncertainty-aware weighting significantly degraded performance, underscoring the importance of UPO's core components.
- Noise Reduction: UPO demonstrated a significant reduction in noise rate (over 20%, 10%, and 3% on different datasets) compared to other sampling strategies.
Limitations and Future Work
- Mathematics Reasoning Data: The paper notes that for mathematics reasoning, the performance improvement from UPO-Merge was less pronounced than in NLP tasks. This is attributed to the high reliance on cleaned preference data, and the uncertainty estimation might still introduce some noise into fine-grained feedback.
- Future Work: The authors aim to further improve the overall performance and adapt the framework to Proximal Policy Optimization (PPO) and other LLMs. They also suggest exploring more sophisticated uncertainty estimation techniques and potentially integrating them with other alignment strategies.
Summary updated