SPO: Self Preference Optimization with Self Regularization

Yuhao Sun, Yifan Zhang, Quandong Wang, Qinzhuo Wu, Wei Liu, Jian LuanEMNLP 2025

TL;DR

This paper introduces SPO, a novel method for aligning large language models (LLMs) with human preferences. SPO improves upon existing techniques by using a new loss function (SiLU) that prevents "over-optimization," a common issue where models become too specialized and perform worse overall. SPO achieves this while being simpler and more efficient than previous methods.

What Problem Does This Paper Address?

Large language models (LLMs) have become incredibly capable, but aligning them with human preferences (making them helpful, truthful, and harmless) is a crucial step. This alignment is often achieved through preference optimization, where models learn from human judgments about which responses are better.

Direct Preference Optimization (DPO) is a popular method that simplifies this process by directly optimizing the LLM policy without needing a separate reward model. However, DPO and similar "reference-free" methods like SimPO and CPO can suffer from over-optimization. This means the model might become too good at satisfying the specific preference signals it's trained on, leading to a degradation in its overall performance, consistency, or diversity. Imagine a student who only studies for the exact questions they expect on a test, only to struggle with slightly different ones – that's a form of over-optimization. The paper aims to solve this by developing a preference optimization method that is both effective and resistant to over-optimization.

Methods

The core innovation of SPO lies in its loss function. Instead of the standard logsigmoid function used in DPO and SimPO, SPO employs the SiLU (Sigmoid Linear Unit) function.

Here's why this is important:

  • SiLU's Finite Minimum: The logsigmoid function approaches negative infinity as its input grows, which can lead to the model excessively amplifying the probability ratio between chosen and rejected samples. This is a key driver of over-optimization. SiLU, on the other hand, has a finite minimum value. This property acts as a self-regularizer, preventing the model from pushing the probability ratios too extreme.
  • Theoretical Guarantee: The authors theoretically prove that the SPO loss function is an upper bound of the DPO loss. This means that by optimizing SPO, they are implicitly optimizing the DPO objective, ensuring that the benefits of DPO are retained.
  • Reference-Free: Like SimPO and CPO, SPO does not require a separate reference model, making it more memory and computationally efficient than DPO.
  • Length Normalization: SPO also incorporates length normalization, similar to SimPO, to address potential biases introduced by longer sequences.

Datasets

The paper uses publicly available datasets for training and evaluation:

  • UltraFeedback: Used for training the preference optimization models.
  • AlpacaEval 2: An automatic evaluation framework for assessing LLM performance, specifically measuring length-controlled win rates.
  • MT-Bench: Another automatic evaluation benchmark that assesses conversational abilities across various categories.

The paper does not introduce new benchmarks.

Key Insights and Ideas

  • The Loss Function is Key: The paper strongly emphasizes that the choice of loss function is a critical factor in mitigating over-optimization in preference learning.
  • SiLU as a Natural Regularizer: The SiLU function's inherent properties, particularly its finite minimum, provide a form of self-regularization that effectively combats over-optimization without needing explicit regularization terms like SFT weights (though they explored this too).
  • Theoretical Equivalence: The proof that SPO's loss is an upper bound of DPO's loss is a significant theoretical contribution, providing confidence that SPO is not sacrificing DPO's core benefits.
  • Efficiency Gains: By being reference-free, SPO offers practical advantages in terms of memory and computational cost compared to DPO.

Key Results

SPO demonstrates strong performance across multiple benchmarks and base models:

  • AlpacaEval 2: SPO achieved a 7% improvement over SimPO in length-controlled win rate on AlpacaEval 2 when applied to the Gemma model. It also outperformed DPO by up to 9.6% on this metric.
  • MT-Bench: SPO showed superior performance on MT-Bench compared to DPO and SimPO.
  • Robustness: The paper shows that SPO's performance advantage increases with larger base models, indicating its robustness and generality.
  • Ablation Studies: Experiments confirmed that the SiLU loss function was crucial for SPO's effectiveness, outperforming other activation functions like ReLU and GELU. The impact of SFT regularization was found to be less critical, suggesting SPO's self-regularization is sufficient.

Limitations and Future Work

  • Off-Policy Focus: The study primarily focuses on off-policy methods (like DPO and SimPO). The authors acknowledge that on-policy methods might behave differently.
  • Limited Benchmarks: The evaluation was conducted on only two model sizes and two public human feedback datasets.
  • Automated Evaluation Bias: The paper notes that recent studies suggest automated evaluations can have biases compared to human evaluations, and a more extensive comparative study would be beneficial.
  • Future Work: The authors suggest a more extensive comparative study of various preference optimization methods as a direction for future research.

Summary updated

BibTeX
@inproceedings{sun2025spo,
  author = {Yuhao Sun and Yifan Zhang and Quandong Wang and Qinzhuo Wu and Wei Liu and Jian Luan},
  title = {{SPO: Self Preference Optimization with Self Regularization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure