SePPO: Semi-Policy Preference Optimization for Diffusion Alignment

Daoan Zhang, Guangchen Lan, Dong-Jun Han, Wenlin Yao, Xiaoman Pan, Hongming Zhang, Mingxiao Li, pengcheng chen, Dong Yu, Christopher Brinton, Jiebo LuoSubmission to ICLR 2025 · Withdrawn

TL;DR

Aligning diffusion models with human preferences typically requires either error-prone reward models (on-policy RLHF) or costly human-annotated preference pairs (off-policy DPO). Semi-Policy Preference Optimization (SePPO) eliminates both requirements by pairing high-quality demonstration images with synthetic negative images generated on-policy from randomly sampled past checkpoints. To prevent degrading performance when generated samples are accidentally better than the target, SePPO introduces an Anchor-based Adaptive Flipper (AAF) that dynamically determines whether to reject or embrace generated samples.

What Problem Does This Paper Address?

Fine-tuning diffusion models (DMs) for visual generation using human feedback currently faces a dilemma:

  • On-policy methods (e.g., DDPO): Rely on learned reward models (RMs) or vision-language models. RMs are hard to train, fail to capture fine-grained visual quality, and are prone to reward hacking, which degrades output realism.
  • Off-policy methods (e.g., Diffusion-DPO): Eliminate reward models by optimizing directly on preference pairs $(x^w, x^l)$, but they depend heavily on extensive, expensive human annotations.
  • Self-improvement methods (e.g., SPIN-Diffusion, DITTO): Generate "losing" candidates using earlier checkpoints. However, these methods blindly assume that earlier or current checkpoints always generate inferior images. If an earlier checkpoint accidentally produces a high-quality sample, penalizing it misguides training.

The authors ask: Can we align visual diffusion models without reward models or paired human annotations, while ensuring the model doesn't penalize accidentally high-quality generated images?

Methods

SePPO operates in a semi-policy setup: positive samples ($x^w$) are taken off-policy from supervised fine-tuning (SFT) data, while negative samples ($x^\text{ref}$) are generated on-policy by reference policies.

                  ┌──────────────────────┐
                  │ Off-policy Winning   │──> Positive Anchor (xʷ)
                  │ Demonstrations (SFT) │
                  └──────────────────────┘
                             │
                             ▼
  Checkpoints [0, ..., k-1] ──> Sample Reference Policy (θ_ref)
                             │
                             ▼
                  Generate Reference Sample (xʳᵉᶠ)
                             │
                             ▼
           Anchor-based Adaptive Flipper (AAF)
   [Checks: Does θ_ref predict noise on xʷ better than θ_k?]
                             │
              ┌──────────────┴──────────────┐
       sign = +1 (Normal DPO)         sign = -1 (Flip Loss)
       Treat xʳᵉᶠ as "loser"          Learn from *both* xʷ and xʳᵉᶠ
  1. Random Reference Sampling ($\text{ref} \sim \mathcal{U}(0, k-1)$):
    Standard DPO fixes the reference model to the initial checkpoint ($\text{ref} = 0$), which rapidly overfits and limits exploration. Updating to only the most recent checkpoint ($\text{ref} = k - 1$, like SPIN) causes training instability. SePPO uniformly samples a reference model from all previously saved checkpoints, stabilizing training and broadening policy exploration.

  2. Anchor-based Adaptive Flipper (AAF):
    Rather than unconditionally treating $x^\text{ref}$ as a negative ("losing") sample, SePPO uses the known winning image $x^w$ as an anchor. By comparing the noise prediction errors of the current model ($\theta$) versus the reference model ($\theta_\text{ref}$) on $x^w$: $$\text{sign} = \text{sgn}\left(\|\epsilon_\text{ref}(x_t^w) - \epsilon^w\|^2 - \|\epsilon_\theta(x_t^w) - \epsilon^w\|^2\right)$$

  3. If the current model predicts noise better on the anchor ($\text{sign} = +1$), $x^\text{ref}$ is treated as a standard negative sample.
  4. If the reference model is actually better on the anchor ($\text{sign} = -1$), $x^\text{ref}$ is likely high quality too. The loss adaptively flips signs so the model learns from both $x^w$ and $x^\text{ref}$, avoiding penalizing good outputs.

Datasets

  • Training:
  • Text-to-Image: Pick-a-Pic v2 (filtered to 851,293 pairs across 58,960 prompts). SePPO uses only the winning images ($\text{SFT}^w$) or a random selection of single images ($\text{SePPO}^r$).
  • Text-to-Video: MagicTime training set (time-lapse videos).
  • Evaluation:
  • Pick-a-Pic v2 validation split (in-distribution).
  • HPSv2 and Parti-prompts (out-of-distribution evaluation for text-to-image).
  • ChronoMagic-Bench-150 (for text-to-video evaluation).
  • No new benchmarks were introduced in this work.

Key Insights and Ideas

  • Earlier checkpoints are not strictly worse: Generative quality fluctuates. Treating self-generated images blindly as negative pairs introduces noisy gradients that stall learning or cause gradient updates to collapse toward zero when generated samples are close to positive samples.
  • Denoising error as a quality proxy: The authors prove theoretically (Theorem 4.1) that a lower mean squared error on noise prediction directly bounds the KL divergence between the recovered image and ground truth. Thus, comparing noise prediction on a known good sample serves as a reliable discriminator without an explicit reward model.
  • Random reference selection acts as regularization: Sampling across past checkpoints prevents the policy from oscillating or collapsing into non-global optima, outperforming both static-initial and immediate-predecessor reference choices.

Key Results

  • Text-to-Image (Stable Diffusion 1.5):
  • On Pick-a-Pic validation, $\text{SePPO}^w$ achieves a 21.57 PickScore, 27.20 HPSv2, and 0.615 ImageReward, outperforming SFT, DDPO, D3PO, Diffusion-DPO, and SPIN-Diffusion.
  • Notably, $\text{SePPO}^r$ (trained on randomly selected images from the dataset, including losers) still outperforms $\text{SFT}^w$ on three key metrics.
  • Demonstrates superior out-of-distribution generalization on HPSv2 and Parti-prompts (e.g., ImageReward of 0.70 vs. 0.54 for SPIN-Diffusion on HPS).
  • Text-to-Video (AnimateDiff):
  • On ChronoMagic-Bench-150, SePPO drops FVD from 1608.41 (raw) and 1415.68 (SFT) down to 1300.97, while improving FID (115.32), LPIPS (0.61), SSIM (0.20), and PSNR (9.36).
  • Ablation on AAF: Removing AAF drops PickScore from 21.57 to 20.88 and ImageReward from 0.615 to 0.366, confirming the critical importance of filtering "insufficiently negative" samples.

Limitations and Future Work

  • Theoretical exploration bounds: A formal theoretical framework for reference-model-constrained policy exploration specifically tailored to diffusion processes remains an open question.
  • Latent vs. Pixel space: Current preference loss calculations occur exclusively in latent space; the authors suggest that incorporating pixel-space evaluation prior to VAE encoding could yield further quality gains.

Summary updated

Abstract

Reinforcement learning from human feedback (RLHF) methods are emerging as a way to fine-tune diffusion models (DMs) for visual generation. However, commonly used on-policy strategies are limited by the generalization capability of the reward model, while off-policy approaches require large amounts of difficult-to-obtain paired human-annotated data, particularly in visual generation tasks. To address the limitations of both on- and off-policy RLHF, we propose a preference optimization method that aligns DMs with preferences without relying on reward models or paired human-annotated data. Specifically, we introduce a Semi-Policy Preference Optimization (SePPO) method. SePPO leverages previous checkpoints as reference models while using them to generate on-policy reference samples, which replace “losing images” in preference pairs. This approach allows us to optimize using only off-policy “winning images”. Furthermore, we design a strategy for reference model selection that expands the exploration in the policy space. Notably, we do not simply treat reference samples as negative examples for learning. Instead, we design an anchor-based criterion to assess whether the reference samples are likely to be winning or losing images, allowing the model to selectively learn from the generated reference samples. This approach mitigates performance degradation caused by the uncertainty in reference sample quality. We validate SePPO across both text-to-image and text-to-video benchmarks. SePPO surpasses all previous approaches on the text-to-image benchmarks and also demonstrates outstanding performance on the text-to-video benchmarks.
BibTeX
@misc{zhang2024seppo,
  author = {Daoan Zhang and Guangchen Lan and Dong-Jun Han and Wenlin Yao and Xiaoman Pan and Hongming Zhang and Mingxiao Li and pengcheng chen and Dong Yu and Christopher Brinton and Jiebo Luo},
  title = {{SePPO: Semi-Policy Preference Optimization for Diffusion Alignment}},
  year = {2024},
  eprint = {2410.05255},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.05255},
}

Loading related content...

Figure