TSO: Self-Training with Scaled Preference Optimization

Kaihui Chen, Hao Yi, Qingyang Li, Tianyu Qi, Yulan Hu, Fuzheng Zhang, Yong LiuarXiv 2024 · OpenReview

Abstract

Enhancing the conformity of large language models (LLMs) to human preferences remains an ongoing research challenge. Recently, offline approaches such as Direct Preference Optimization (DPO) have gained prominence as attractive options due to offering effective improvement in simple, efficient, and stable without interactions with reward models. However, these offline preference optimization methods highly rely on the quality of pairwise preference samples. Meanwhile, numerous iterative methods require additional training of reward models to select positive and negative samples from the model's own generated responses for preference learning. Furthermore, as LLMs' capabilities advance, it is quite challenging to continuously construct high-quality positive and negative preference instances from the model's outputs due to the lack of diversity. To tackle these challenges, we propose TSO, or Self-Training with Scaled Preference Optimization, a framework for preference optimization that conducts self-training preference learning without training an additional reward model. TSO enhances the diversity of responses by constructing a model matrix and incorporating human preference responses. Furthermore, TSO introduces corrections for model preference errors through human and AI feedback. Finally, TSO adopts iterative and dual clip reward strategies to update the reference model and its responses, adaptively adjusting preference data and balancing the optimization process. Experimental results demonstrate that TSO outperforms existing mainstream methods on various alignment evaluation benchmarks, providing practical insight into preference data construction and model training strategies in the alignment domain.

Introduction

Aligning Large Language Models (LLMs) with human expectations typically relies on Reinforcement Learning from Human Feedback (RLHF). While Direct Preference Optimization (DPO) simplified this by treating the LLM itself as a reward model, it faces significant hurdles. First, as models improve, generating distinct "negative" responses becomes difficult, reducing the learning signal. Second, standard DPO often suffers from optimization imbalance where negative samples dominate, causing rewards for positive samples to stagnate. Finally, self-rewarding models often exhibit evaluation bias on out-of-distribution (OOD) samples.

Methodology: The TSO Framework

TSO addresses these issues through a multi-stage self-training pipeline that eliminates the need for an external reward model.

The TSO framework overview, illustrating the model matrix construction, evaluation correction, and iterative mini-batch training.

The framework consists of three primary components:

  1. Model Matrix Instructions Construction: TSO utilizes a "matrix" of models across different scales (e.g., 13B to 175B). High-quality positive responses are drawn from larger models and human datasets, while diverse negative responses are sampled from smaller or older versions to ensure a distinct distributional gap.
  2. Evaluation Correction: To mitigate evaluation bias, a base model fine-tuned on human-scored data (LLM-as-a-Judge) re-evaluates generated pairs, correcting preference labels for OOD samples.
  3. Mini-Batches Iterative DPO & Dual Clip Reward: TSO updates the reference model after each mini-batch rather than once per epoch. It introduces a Dual Clip Reward Loss ($\mathcal{L}_{dual-clip}$) with separate margins for positive and negative rewards, preventing the "dominant negative" problem and stabilizing optimization.

Experimental Results

The TSO framework was evaluated using the HH-RLHF and Reddit TL;DR datasets, alongside a custom collection of 30,000 prompts (TSO-D). Performance was measured across several major benchmarks, including AlignBench, MT-Bench, AlpacaEval-v2, and Arena-Hard.

Comparison of TSO against baseline methods across various alignment benchmarks.

As shown in the results, TSO-3 (the third iteration) significantly outperforms standard DPO and other preference optimization methods. Notably, it achieved a 15.4% increase in length-controlled win rate on Arena-Hard and an 11.96% improvement on AlpacaEval-v2 compared to the base model.

Conclusion

TSO provides a robust framework for self-training preference optimization. By leveraging a model matrix for data diversity and a dual-clip loss for balanced optimization, it achieves state-of-the-art alignment performance without external reward models. Future work will focus on reducing the computational cost of maintaining the model matrix and further refining automated feedback loops.

Summary updated

Abstract

Enhancing the conformity of large language models (LLMs) to human preferences remains an ongoing research challenge. Recently, offline approaches such as Direct Preference Optimization (DPO) have gained prominence as attractive options due to offering effective improvement in simple, efficient, and stable without interactions with reward models. However, these offline preference optimization methods highly rely on the quality of pairwise preference samples. Meanwhile, numerous iterative methods require additional training of reward models to select positive and negative samples from the model's own generated responses for preference learning. Furthermore, as LLMs' capabilities advance, it is quite challenging to continuously construct high-quality positive and negative preference instances from the model's outputs due to the lack of diversity. To tackle these challenges, we propose TSO, or Self-Training with Scaled Preference Optimization, a framework for preference optimization that conducts self-training preference learning without training an additional reward model. TSO enhances the diversity of responses by constructing a model matrix and incorporating human preference responses. Furthermore, TSO introduces corrections for model preference errors through human and AI feedback. Finally, TSO adopts iterative and dual clip reward strategies to update the reference model and its responses, adaptively adjusting preference data and balancing the optimization process. Experimental results demonstrate that TSO outperforms existing mainstream methods on various alignment evaluation benchmarks, providing practical insight into preference data construction and model training strategies in the alignment domain.
BibTeX
@misc{chen2024tso,
  author = {Kaihui Chen and Hao Yi and Qingyang Li and Tianyu Qi and Yulan Hu and Fuzheng Zhang and Yong Liu},
  title = {{TSO: Self-Training with Scaled Preference Optimization}},
  year = {2024},
  eprint = {2409.02118},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.02118},
}
Figure