SePPO: Semi-Policy Preference Optimization for Diffusion Alignment

Daoan Zhang, Guangchen Lan, Dong-Jun Han, Wenlin Yao, Xiaoman Pan, Hongming Zhang, Mingxiao Li, Pengcheng Chen, Dong Yu, Christopher Brinton, Jiebo Luo

Abstract

Reinforcement learning from human feedback (RLHF) methods are emerging as a way to fine-tune diffusion models (DMs) for visual generation. However, commonly used on-policy strategies are limited by the generalization capability of the reward model, while off-policy approaches require large amounts of difficult-to-obtain paired human-annotated data, particularly in visual generation tasks. To address the limitations of both on- and off-policy RLHF, we propose a preference optimization method that aligns DMs with preferences without relying on reward models or paired human-annotated data. Specifically, we introduce a Semi-Policy Preference Optimization (SePPO) method. SePPO leverages previous checkpoints as reference models while using them to generate on-policy reference samples, which replace "losing images" in preference pairs. This approach allows us to optimize using only off-policy "winning images." Furthermore, we design a strategy for reference model selection that expands the exploration in the policy space. Notably, we do not simply treat reference samples as negative examples for learning. Instead, we design an anchor-based criterion to assess whether the reference samples are likely to be winning or losing images, allowing the model to selectively learn from the generated reference samples. This approach mitigates performance degradation caused by the uncertainty in reference sample quality. We validate SePPO across both text-to-image and text-to-video benchmarks. SePPO surpasses all previous approaches on the text-to-image benchmarks and also demonstrates outstanding performance on the text-to-video benchmarks.

Introduction

Aligning diffusion models with human intent typically involves two distinct paradigms: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). While SFT offers stability, it often fails to generalize to novel prompts. Conversely, RL-based methods like Direct Preference Optimization (DPO) offer better generalization but are notoriously difficult to stabilize.

Standard RL alignment for visual tasks (e.g., Diffusion-DPO) requires massive datasets of human-annotated "winning" and "losing" image pairs or a robust external Reward Model. However, these reward models are expensive to build and prone to "reward hacking," where models generate artifacts that score highly but appear unnatural to humans. SePPO (referred to as SSPO in technical diagrams) addresses these challenges by bridging the gap between SFT and RL without requiring external preference data or reward models.

Methodology

SePPO introduces a "self-improvement" framework through two core mechanisms: Random Checkpoint Replay (RCR) and Self-Sampling Regularization (SSR).

Comparison of SFT, DPO, and the proposed SePPO (SSPO) framework.

Random Checkpoint Replay (RCR)

Instead of relying solely on the current model iteration to generate "losing" samples (a standard self-play technique), SePPO uniformly samples from all historical checkpoints. This strategy prevents the model from overfitting to recent errors and provides a more diverse and stable training signal, maintaining a broader understanding of the data distribution.

Self-Sampling Regularization (SSR)

In traditional self-play, model outputs are assumed to be "losers" compared to ground truth. However, as a model improves, it may generate high-quality samples that should not be penalized. SSR dynamically evaluates the quality of generated samples. If a sample is identified as high-quality (a "winning" sample), the algorithm switches from a DPO loss to an SFT loss. This anchor-based criterion ensures the model selectively learns from generated samples, acting like DPO when outputs are poor and transitioning to SFT when they are high-quality.

Experimental Setup

The researchers validated SePPO across multiple modalities and base models, including Stable Diffusion 1.5, SDXL, and AnimateDiff:

  • Text-to-Image: Training utilized the "winning" images from the Pick-a-Pic v2 dataset. Evaluation was conducted on Parti-prompt, HPSv2, and the Pick-a-Pic validation set.
  • Text-to-Video: Training used the MagicTime dataset, with evaluation on the ChronoMagic-Bench-150 benchmark.

Key Results

SePPO consistently outperformed state-of-the-art methods such as Diffusion-DPO, SPO, and SPIN-Diffusion across several metrics.

PickScore performance across training steps for different reference selection strategies.
  • Superior Alignment: On the Pick-a-Pic validation set, SePPO achieved a PickScore of 21.57, surpassing SPO (21.43). When applied to SDXL, it reached a PickScore of 23.24, a significant jump from the base model's 21.95.
  • Video Generation: In text-to-video tasks, SePPO reduced the Fréchet Video Distance (FVD) from 1608.41 to 1300.97, indicating a substantial improvement in video quality and temporal consistency.
  • Efficiency: The method is comparable to existing approaches in GPU memory and time costs but requires fewer hyperparameters, simplifying deployment.

Conclusion and Future Work

SePPO demonstrates that a model's "past selves" are valuable assets for alignment. By replaying historical outputs and using a smart switching mechanism, it achieves high-quality alignment without expensive human labels. Future work will focus on deeper theoretical understanding of policy space exploration in diffusion models and exploring optimization directly in pixel space rather than latent space to further sharpen model preferences.

Summary updated

Abstract

Reinforcement learning from human feedback (RLHF) methods are emerging as a way to fine-tune diffusion models (DMs) for visual generation. However, commonly used on-policy strategies are limited by the generalization capability of the reward model, while off-policy approaches require large amounts of difficult-to-obtain paired human-annotated data, particularly in visual generation tasks. To address the limitations of both on- and off-policy RLHF, we propose a preference optimization method that aligns DMs with preferences without relying on reward models or paired human-annotated data. Specifically, we introduce a Semi-Policy Preference Optimization (SePPO) method. SePPO leverages previous checkpoints as reference models while using them to generate on-policy reference samples, which replace "losing images" in preference pairs. This approach allows us to optimize using only off-policy "winning images." Furthermore, we design a strategy for reference model selection that expands the exploration in the policy space. Notably, we do not simply treat reference samples as negative examples for learning. Instead, we design an anchor-based criterion to assess whether the reference samples are likely to be winning or losing images, allowing the model to selectively learn from the generated reference samples. This approach mitigates performance degradation caused by the uncertainty in reference sample quality. We validate SePPO across both text-to-image and text-to-video benchmarks. SePPO surpasses all previous approaches on the text-to-image benchmarks and also demonstrates outstanding performance on the text-to-video benchmarks. Code will be released in https://github.com/DwanZhang-AI/SePPO.
BibTeX
@misc{zhang2024seppo,
  author = {Daoan Zhang and Guangchen Lan and Dong-Jun Han and Wenlin Yao and Xiaoman Pan and Hongming Zhang and Mingxiao Li and Pengcheng Chen and Dong Yu and Christopher Brinton and Jiebo Luo},
  title = {{SePPO: Semi-Policy Preference Optimization for Diffusion Alignment}},
  year = {2024},
  eprint = {2410.05255},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.05255},
}
Figure