Indirect Online Preference Optimization via Reinforcement Learning

En Wang, Xingyu Lin, Du Su, Chenfu Bao, Zhonghou Lv, Funing Yang, Yuanbo Xu, Wenbin Liu

Abstract

Large Language Model (LLM) alignment typically relies on Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO), both of which suffer from high annotation costs and distribution bias during iterative self-improvement. We present Indirect Online Preference Optimization (IOPO), a framework that reformulates alignment as a two-player zero-sum game between a protagonist and an adversarial model. By seeking a Nash Equilibrium, IOPO mitigates distribution drift and significantly accelerates iteration cycles. Experimental results across multiple datasets demonstrate that IOPO consistently outperforms state-of-the-art methods like DPO and IPO in both automated metrics and human evaluation.

Introduction: The Challenge of Distribution Bias

Aligning Large Language Models (LLMs) with human moral and ethical standards is a critical step in model deployment. Traditional methods like RLHF and DPO require massive amounts of high-quality, human-annotated preference data. To scale this process, researchers often employ "iterative online" strategies where the model generates its own data and scores it.

However, this approach introduces a significant distribution bias: as the model iteratively learns from its own outputs, its internal preference distribution drifts away from actual human preferences. This modeling error between the simulator (the model) and the real world (humanity) causes alignment quality to degrade over time. IOPO addresses this by treating distribution bias as an adversarial perturbation to be actively defeated.

Methodology: IOPO and Adversarial Alignment

IOPO frames preference optimization as a two-player zero-sum game. The framework consists of two primary agents:

  1. The Protagonist: The agent tasked with aligning to human preferences.
  2. The Adversary: An agent that challenges the protagonist by identifying biases or "holes" in its alignment.
The IOPO architecture utilizes a protagonist-adversary framework to calculate distribution shifts and optimize policies via clipped importance sampling.

Key Innovations

  • Nash Equilibrium Training: Instead of learning from static data, the protagonist is trained to reach a Nash Equilibrium against the adversary. This ensures the model is robust against the types of distribution shifts that typically plague self-improving models.
  • Clipped Importance Sampling (IS): To maintain training stability and computational efficiency, IOPO employs a clipped IS weight. This mechanism penalizes extreme policy deviations, ensuring the zero-sum game converges reliably with linear complexity.
  • Inverse Reward Design: By optimizing the reward functions of both agents simultaneously, the system prevents the protagonist from converging toward the biased rewards often found in standard self-rewarding models.

Experimental Evaluation

The researchers evaluated IOPO against three diverse real-world datasets:

  • safe world view: 44k single-turn comparisons regarding world-view demands.
  • distilabel-capybara-dpo: 7.56k multi-turn comparisons covering specialized domains like math, medicine, and CS.
  • orpo-dpo-mix-40k: 44.2k comparisons focused on rejecting toxic or illegal queries.

Results and Discussion

IOPO consistently outperformed state-of-the-art methods, including DPO, IPO, and KTO-PAIR, across all tested datasets.

Margin reward distributions across three datasets show IOPO (blue) consistently achieving higher rewards and tighter alignment compared to baseline methods.

Key Performance Metrics

  • Win Rates: In head-to-head evaluations, IOPO achieved a 50.4% win rate against DPO on the safe world view test set.
  • Pairwise Accuracy: IOPO reached a pairwise accuracy of 0.9659 on the safe world view dataset, surpassing Iter-DPO (0.961) and standard DPO (0.9286).
  • Human Evaluation: In real-world online tests, IOPO received the highest mean score from human evaluators (2.176) compared to SFT, DPO, and Iter-DPO.
  • Operational Efficiency: Perhaps most significantly, IOPO reduced the time required for model iteration cycles from months to a single week by minimizing the need for continuous human-in-the-loop annotation.

Conclusion and Future Work

Indirect Online Preference Optimization (IOPO) provides a robust solution to the distribution drift problem in LLM alignment. By leveraging adversarial training and clipped importance sampling, it achieves superior alignment with significantly reduced human annotation costs. While the framework requires careful tuning of hyper-parameters like the clipping epsilon ($\epsilon$) and KL-divergence penalty ($\beta$), its efficiency gains represent a major step forward for production-level LLM development. Future research will explore scaling this adversarial mechanism to complex multi-turn reasoning tasks and even larger model architectures.

Summary updated

Abstract

Human preference alignment (HPA) aims to ensure Large Language Models (LLMs) responding appropriately to meet human moral and ethical requirements. Existing methods, such as RLHF and DPO, rely heavily on high-quality human annotation, which restrict the efficiency of iterative online model refinement. To address the inefficiencies of human annotation acquisition, iterated online strategy advocates the use of fine-tuned LLMs to self-generate preference data. However, this approach is prone to distribution bias, because of differences between human and model annotations, as well as modeling errors between simulators and real-world contexts. To mitigate the impact of distribution bias, we adopt the principles of adversarial training, framing a zero-sum two-player game with a protagonist agent and an adversarial agent. With the adversarial agent challenging the alignment of protagonist agent, we continuously refine the protagonist’s performance. By utilizing min-max equilibrium and Nash equilibrium strategies, we propose Indirect Online Preference Optimization (IOPO) mechanism that enables the protagonist agent to converge without bias while maintaining linear computational complexity. Extensive experiments across three real-world datasets demonstrate that IOPO outperforms state-of-the-art alignment methods in both offline and online scenarios, evidenced by standard alignment metrics and human evaluations. This innovation reduces the time required for model iterations from months to one week, alleviates distribution shifts, and significantly cuts annotation costs.
BibTeX
@inproceedings{wang2025indirect,
  author = {En Wang and Xingyu Lin and Du Su and Chenfu Bao and Zhonghou Lv and Funing Yang and Yuanbo Xu and Wenbin Liu},
  title = {{Indirect Online Preference Optimization via Reinforcement Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/61},
}
Figure