Abstract
Enhancing the conformity of large language models (LLMs) to human preferences remains an ongoing research challenge. Recently, offline approaches such as Direct Preference Optimization (DPO) have gained prominence as attractive options due to offering effective improvement in simple, efficient, and stable without interactions with reward models. However, these offline preference optimization methods highly rely on the quality of pairwise preference samples. Meanwhile, numerous iterative methods require additional training of reward models to select positive and negative samples from the model's own generated responses for preference learning. Furthermore, as LLMs' capabilities advance, it is quite challenging to continuously construct high-quality positive and negative preference instances from the model's outputs due to the lack of diversity. To tackle these challenges, we propose TSO, or Self-Training with Scaled Preference Optimization, a framework for preference optimization that conducts self-training preference learning without training an additional reward model. TSO enhances the diversity of responses by constructing a model matrix and incorporating human preference responses. Furthermore, TSO introduces corrections for model preference errors through human and AI feedback. Finally, TSO adopts iterative and dual clip reward strategies to update the reference model and its responses, adaptively adjusting preference data and balancing the optimization process. Experimental results demonstrate that TSO outperforms existing mainstream methods on various alignment evaluation benchmarks, providing practical insight into preference data construction and model training strategies in the alignment domain.
Introduction
Aligning Large Language Models (LLMs) with human expectations typically relies on Reinforcement Learning from Human Feedback (RLHF). While Direct Preference Optimization (DPO) simplified this by treating the LLM itself as a reward model, it faces significant hurdles. First, as models improve, generating distinct "negative" responses becomes difficult, reducing the learning signal. Second, standard DPO often suffers from optimization imbalance where negative samples dominate, causing rewards for positive samples to stagnate. Finally, self-rewarding models often exhibit evaluation bias on out-of-distribution (OOD) samples.
Methodology: The TSO Framework
TSO addresses these issues through a multi-stage self-training pipeline that eliminates the need for an external reward model.
The framework consists of three primary components:
- Model Matrix Instructions Construction: TSO utilizes a "matrix" of models across different scales (e.g., 13B to 175B). High-quality positive responses are drawn from larger models and human datasets, while diverse negative responses are sampled from smaller or older versions to ensure a distinct distributional gap.
- Evaluation Correction: To mitigate evaluation bias, a base model fine-tuned on human-scored data (LLM-as-a-Judge) re-evaluates generated pairs, correcting preference labels for OOD samples.
- Mini-Batches Iterative DPO & Dual Clip Reward: TSO updates the reference model after each mini-batch rather than once per epoch. It introduces a Dual Clip Reward Loss ($\mathcal{L}_{dual-clip}$) with separate margins for positive and negative rewards, preventing the "dominant negative" problem and stabilizing optimization.
Experimental Results
The TSO framework was evaluated using the HH-RLHF and Reddit TL;DR datasets, alongside a custom collection of 30,000 prompts (TSO-D). Performance was measured across several major benchmarks, including AlignBench, MT-Bench, AlpacaEval-v2, and Arena-Hard.
As shown in the results, TSO-3 (the third iteration) significantly outperforms standard DPO and other preference optimization methods. Notably, it achieved a 15.4% increase in length-controlled win rate on Arena-Hard and an 11.96% improvement on AlpacaEval-v2 compared to the base model.
Conclusion
TSO provides a robust framework for self-training preference optimization. By leveraging a model matrix for data diversity and a dual-clip loss for balanced optimization, it achieves state-of-the-art alignment performance without external reward models. Future work will focus on reducing the computational cost of maintaining the model matrix and further refining automated feedback loops.
Summary updated