TL;DR
Aligning diffusion models with human preferences typically requires either error-prone reward models (on-policy RLHF) or costly human-annotated preference pairs (off-policy DPO). Semi-Policy Preference Optimization (SePPO) eliminates both requirements by pairing high-quality demonstration images with synthetic negative images generated on-policy from randomly sampled past checkpoints. To prevent degrading performance when generated samples are accidentally better than the target, SePPO introduces an Anchor-based Adaptive Flipper (AAF) that dynamically determines whether to reject or embrace generated samples.
What Problem Does This Paper Address?
Fine-tuning diffusion models (DMs) for visual generation using human feedback currently faces a dilemma:
- On-policy methods (e.g., DDPO): Rely on learned reward models (RMs) or vision-language models. RMs are hard to train, fail to capture fine-grained visual quality, and are prone to reward hacking, which degrades output realism.
- Off-policy methods (e.g., Diffusion-DPO): Eliminate reward models by optimizing directly on preference pairs $(x^w, x^l)$, but they depend heavily on extensive, expensive human annotations.
- Self-improvement methods (e.g., SPIN-Diffusion, DITTO): Generate "losing" candidates using earlier checkpoints. However, these methods blindly assume that earlier or current checkpoints always generate inferior images. If an earlier checkpoint accidentally produces a high-quality sample, penalizing it misguides training.
The authors ask: Can we align visual diffusion models without reward models or paired human annotations, while ensuring the model doesn't penalize accidentally high-quality generated images?
Methods
SePPO operates in a semi-policy setup: positive samples ($x^w$) are taken off-policy from supervised fine-tuning (SFT) data, while negative samples ($x^\text{ref}$) are generated on-policy by reference policies.
┌──────────────────────┐
│ Off-policy Winning │──> Positive Anchor (xʷ)
│ Demonstrations (SFT) │
└──────────────────────┘
│
▼
Checkpoints [0, ..., k-1] ──> Sample Reference Policy (θ_ref)
│
▼
Generate Reference Sample (xʳᵉᶠ)
│
▼
Anchor-based Adaptive Flipper (AAF)
[Checks: Does θ_ref predict noise on xʷ better than θ_k?]
│
┌──────────────┴──────────────┐
sign = +1 (Normal DPO) sign = -1 (Flip Loss)
Treat xʳᵉᶠ as "loser" Learn from *both* xʷ and xʳᵉᶠ
-
Random Reference Sampling ($\text{ref} \sim \mathcal{U}(0, k-1)$):
Standard DPO fixes the reference model to the initial checkpoint ($\text{ref} = 0$), which rapidly overfits and limits exploration. Updating to only the most recent checkpoint ($\text{ref} = k - 1$, like SPIN) causes training instability. SePPO uniformly samples a reference model from all previously saved checkpoints, stabilizing training and broadening policy exploration. -
Anchor-based Adaptive Flipper (AAF):
Rather than unconditionally treating $x^\text{ref}$ as a negative ("losing") sample, SePPO uses the known winning image $x^w$ as an anchor. By comparing the noise prediction errors of the current model ($\theta$) versus the reference model ($\theta_\text{ref}$) on $x^w$: $$\text{sign} = \text{sgn}\left(\|\epsilon_\text{ref}(x_t^w) - \epsilon^w\|^2 - \|\epsilon_\theta(x_t^w) - \epsilon^w\|^2\right)$$ - If the current model predicts noise better on the anchor ($\text{sign} = +1$), $x^\text{ref}$ is treated as a standard negative sample.
- If the reference model is actually better on the anchor ($\text{sign} = -1$), $x^\text{ref}$ is likely high quality too. The loss adaptively flips signs so the model learns from both $x^w$ and $x^\text{ref}$, avoiding penalizing good outputs.
Datasets
- Training:
- Text-to-Image: Pick-a-Pic v2 (filtered to 851,293 pairs across 58,960 prompts). SePPO uses only the winning images ($\text{SFT}^w$) or a random selection of single images ($\text{SePPO}^r$).
- Text-to-Video: MagicTime training set (time-lapse videos).
- Evaluation:
- Pick-a-Pic v2 validation split (in-distribution).
- HPSv2 and Parti-prompts (out-of-distribution evaluation for text-to-image).
- ChronoMagic-Bench-150 (for text-to-video evaluation).
- No new benchmarks were introduced in this work.
Key Insights and Ideas
- Earlier checkpoints are not strictly worse: Generative quality fluctuates. Treating self-generated images blindly as negative pairs introduces noisy gradients that stall learning or cause gradient updates to collapse toward zero when generated samples are close to positive samples.
- Denoising error as a quality proxy: The authors prove theoretically (Theorem 4.1) that a lower mean squared error on noise prediction directly bounds the KL divergence between the recovered image and ground truth. Thus, comparing noise prediction on a known good sample serves as a reliable discriminator without an explicit reward model.
- Random reference selection acts as regularization: Sampling across past checkpoints prevents the policy from oscillating or collapsing into non-global optima, outperforming both static-initial and immediate-predecessor reference choices.
Key Results
- Text-to-Image (Stable Diffusion 1.5):
- On Pick-a-Pic validation, $\text{SePPO}^w$ achieves a 21.57 PickScore, 27.20 HPSv2, and 0.615 ImageReward, outperforming SFT, DDPO, D3PO, Diffusion-DPO, and SPIN-Diffusion.
- Notably, $\text{SePPO}^r$ (trained on randomly selected images from the dataset, including losers) still outperforms $\text{SFT}^w$ on three key metrics.
- Demonstrates superior out-of-distribution generalization on HPSv2 and Parti-prompts (e.g., ImageReward of 0.70 vs. 0.54 for SPIN-Diffusion on HPS).
- Text-to-Video (AnimateDiff):
- On ChronoMagic-Bench-150, SePPO drops FVD from 1608.41 (raw) and 1415.68 (SFT) down to 1300.97, while improving FID (115.32), LPIPS (0.61), SSIM (0.20), and PSNR (9.36).
- Ablation on AAF: Removing AAF drops PickScore from 21.57 to 20.88 and ImageReward from 0.615 to 0.366, confirming the critical importance of filtering "insufficiently negative" samples.
Limitations and Future Work
- Theoretical exploration bounds: A formal theoretical framework for reference-model-constrained policy exploration specifically tailored to diffusion processes remains an open question.
- Latent vs. Pixel space: Current preference loss calculations occur exclusively in latent space; the authors suggest that incorporating pixel-space evaluation prior to VAE encoding could yield further quality gains.
Summary updated