Self-Evolutionary Large Language Models Through Uncertainty-Enhanced Preference Optimization

Jianing Wang, Yang Zhou, Xiaocheng Zhang, Mengjiao Bao, Peng Yan

TL;DR

This paper introduces Uncertainty-enhanced Preference Optimization (UPO), a novel framework to improve the self-evolution of Large Language Models (LLMs) by intelligently filtering noisy preference data. UPO leverages an uncertainty estimator to identify reliable feedback, leading to more robust and effective LLM alignment, as demonstrated by significant performance gains on various NLP and reasoning benchmarks.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly trained using preference optimization, a process that aligns models with human feedback. However, this iterative process often suffers from "noisy preference data" generated by the current policy and reward models, which can hinder performance. Existing methods for mitigating this noise, like simple filtering based on reward scores, can fall prey to confirmation bias. The paper tackles the challenge of making LLM self-evolution more robust and effective by developing a method to discern and utilize high-quality preference signals while downplaying unreliable ones.

Methods

The core of UPO is an Uncertainty-enhanced Preference Optimization (UPO) framework. Key components include:

  • Estimator Model with MC Dropout: A novel estimator model, built upon a Bayesian Neural Network (BNN) approach using Monte Carlo (MC) dropout, is introduced. This model performs pairwise uncertainty estimation on preference data, providing a measure of confidence in the feedback.
  • Uncertainty-Enhanced Self-Evolution Algorithm: This algorithm uses the uncertainty estimates to split preference data into reliable and unreliable pairs. It then integrates this uncertainty information into the Direct Preference Optimization (DPO) objective, encouraging the LLM to learn from both high-reward and high-certainty feedback.
  • Iterative Preference Optimization: The framework operates iteratively, where the LLM policy generates new preference data, which is then filtered by the estimator and used to further train the policy.

Datasets

  • UltraFeedback: Used for initial stage training and as a source of prompts in iterative stages.
  • AlpacaEval 2.0: A benchmark for evaluating LLM performance on general NLP tasks.
  • MT-Bench: Another benchmark for evaluating LLM capabilities across various NLP aspects.
  • GSM8K: A dataset for evaluating mathematical reasoning.
  • MATH: A dataset for evaluating mathematical reasoning.
  • Math-Step-DPO-10K: A dataset specifically designed for fine-grained mathematical reasoning tasks.
  • UltraChat200K: Used as a prompt set in the iterative stages.

Key Insights and Ideas

  • Uncertainty is Key to Reliable Feedback: The paper strongly suggests that simply relying on reward scores is insufficient. Quantifying the uncertainty of the feedback is crucial for identifying truly informative preference signals.
  • Self-Evolution Can Be Noisy: The iterative nature of LLM training, while powerful, can amplify noise if not carefully managed. UPO directly addresses this by actively filtering out unreliable data.
  • Beyond Reward: Certainty Matters: UPO demonstrates that LLMs can benefit from learning not just what is preferred, but also how certain that preference is, leading to more robust and higher-quality outputs.
  • Iterative Improvement Outperforms Merging: The results show that iterative evolution (UPO) is more effective than simply merging all generated data at the end (UPO-Merge), highlighting the benefit of continuous refinement.

Key Results

  • AlpacaEval 2.0: UPO achieved the best win rate against GPT-4, improving by 7.20% over SFT and 3.92% over DPO after three iterations.
  • MT-Bench: UPO obtained the highest score, improving the LLM policy from 6.79% to 7.02%.
  • GSM8K & MATH: UPO consistently achieved substantial improvements, reaching 88.9% accuracy on GSM8K and 56.3% on MATH.
  • Ablation Studies: Removing the uncertainty estimator or the uncertainty-aware weighting significantly degraded performance, underscoring the importance of UPO's core components.
  • Noise Reduction: UPO demonstrated a significant reduction in noise rate (over 20%, 10%, and 3% on different datasets) compared to other sampling strategies.

Limitations and Future Work

  • Mathematics Reasoning Data: The paper notes that for mathematics reasoning, the performance improvement from UPO-Merge was less pronounced than in NLP tasks. This is attributed to the high reliance on cleaned preference data, and the uncertainty estimation might still introduce some noise into fine-grained feedback.
  • Future Work: The authors aim to further improve the overall performance and adapt the framework to Proximal Policy Optimization (PPO) and other LLMs. They also suggest exploring more sophisticated uncertainty estimation techniques and potentially integrating them with other alignment strategies.

Summary updated

Abstract

Iterative preference optimization has recently become one of the de-facto training paradigms for large language models (LLMs), but the performance is still underwhelming due to too much noisy preference data yielded in the loop. To combat this issue, we present an \textbf{U}ncertainty-enhanced \textbf{P}reference \textbf{O}ptimization (UPO) framework to make the LLM self-evolve with reliable feedback. The key idea is mitigating the noisy preference data derived from the current policy and reward models by performing pair-wise uncertainty estimation and judiciously reliable feedback sampling. To reach this goal, we thus introduce an estimator model, which incorporates Monte Carlo (MC) dropout in Bayesian neural network (BNN) to perform uncertainty estimation for the preference data derived from the LLM policy. Compared to the existing methods that directly filter generated responses based on the reward score, the estimator focuses on the model uncertainty in a pair-wise manner and effectively bypasses the confirmation bias problem of the reward model. Additionally, we also propose an uncertainty-enhanced self-evolution algorithm to improve the robustness of preference optimization and encourage the LLM to generate responses with both high reward and certainty. Extensive experiments over multiple benchmarks demonstrate that our framework substantially alleviates the noisy problem and improves the performance of iterative preference optimization.
BibTeX
@inproceedings{wang2025self,
  author = {Jianing Wang and Yang Zhou and Xiaocheng Zhang and Mengjiao Bao and Peng Yan},
  title = {{Self-Evolutionary Large Language Models Through Uncertainty-Enhanced Preference Optimization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i24.34724},
}
Figure