Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Reward

Zhiwei Jia, Yuesong Nan, Huixi Zhao, Gengdai Liu

Abstract

Recent research has shown that fine-tuning diffusion models (DMs) with arbitrary rewards, including non-differentiable ones, is feasible with reinforcement learning (RL) techniques, enabling flexible model alignment. However, applying existing RL methods to timestep-distilled DMs is challenging for ultra-fast (-step) image generation. Our analysis suggests several limitations of policy-based RL methods such as PPO or DPO toward this goal. Based on the insights, we propose fine-tuning DMs with learned differentiable surrogate rewards. Our method, named LaSRO, learns surrogate reward models in the latent space of SDXL to convert arbitrary rewards into differentiable ones for efficient reward gradient guidance. LaSRO leverages pre-trained latent DMs for reward modeling and specifically targets image generation steps for reward optimization, enhancing generalizability and efficiency. LaSRO is effective and stable for improving ultra-fast image generation with different reward objectives, outperforming popular RL methods including PPO and DPO. We further show LaSRO's connection to value-based RL, providing theoretical insights.

Introduction

Standard diffusion models are computationally intensive, often requiring 50 or more denoising steps to generate a single image. To mitigate this, "step-distillation" techniques have produced ultra-fast models like SDXL-Turbo and Latent Consistency Models (LCMs) that function in just 1–2 steps. However, fine-tuning these distilled models to align with human preferences or specific aesthetic rewards using traditional Reinforcement Learning (RL) is notoriously difficult. This paper introduces LaSRO (Latent-Space Surrogate Reward Optimization), a method designed to overcome the unique obstacles of few-step model alignment.

The Challenges of Few-Step RL

The authors identify three primary "RL Obstacles" inherent to ultra-fast models:

  1. Hard Exploration: Because these models use very few steps, they are nearly deterministic. The lack of injected noise during sampling limits the model's ability to explore different ways to satisfy a reward signal.
  2. Degenerated Objectives: Many RL loss functions assume a multi-step stochastic process. When compressed to 1 or 2 steps, the mathematical framework often fails to capture the full generation dynamics.
  3. Non-Smooth Mappings: The transition from a noisy latent to a clean image in a 2-step model is mathematically "sharp." Small weight updates can lead to massive, unstable swings in the output image, making gradient estimation unreliable.

Methodology: LaSRO

LaSRO addresses these issues by learning a differentiable surrogate reward model in the latent space. Instead of estimating gradients through a "black-box" reward, LaSRO provides stable, direct gradient guidance.

Latent-Space Backbone

The method utilizes the pre-trained UNet encoder from SDXL as the foundation for its reward model. Operating in the latent space is more memory-efficient than pixel-space models and better captures the specific features of diffusion latents.

Two-Stage Pipeline

  • Pre-training: The surrogate reward model is trained using a Bradley-Terry preference model to predict which generated images better satisfy a target reward.
  • Fine-tuning: The process alternates between updating the Diffusion Model using the surrogate's gradients and "online adapting" the surrogate to ensure it remains accurate as the model's output distribution shifts.
  • Off-policy Exploration: To solve the exploration problem, the system samples multiple initial noise vectors and first-step outputs, creating a diverse replay buffer of experiences.

Key Results and Evaluation

LaSRO consistently outperforms existing RL baselines like DDPO and Diffusion-DPO across multiple benchmarks, including MJHQ-30K and T2I-CompBench.

Quantitative Performance

On the MJHQ-30K benchmark, LaSRO improved the Image Reward of a 2-step SSD-1B model from 0.781 to 0.998, significantly higher than RLCM's 0.830. It also successfully fine-tuned 1-step models (SDXL-Turbo), raising their Image Reward from 0.817 to 0.904.

Qualitative Improvements

Images generated after LaSRO fine-tuning exhibit superior anatomical detail, more vibrant colors, and stricter adherence to complex prompts compared to baseline distilled models.

High-quality image generation of a car at sunset using a fine-tuned 2-step model.
Single-step generation of an astronaut in a jungle, demonstrating improved prompt adherence and visual quality.

Conclusion

By bypassing noisy policy gradient estimation in favor of a differentiable surrogate, LaSRO provides a stable and theoretically grounded approach to fine-tuning ultra-fast diffusion models. The authors demonstrate that the surrogate reward essentially acts as a "Value Function" (Critic) in a value-based RL framework. While the two-stage process adds some overhead, the gains in alignment and image quality represent a significant advancement for real-time generative AI.

Summary updated

Abstract

Recent research has shown that fine-tuning diffusion models (DMs) with arbitrary rewards, including non-differentiable ones, is feasible with reinforcement learning (RL) techniques, enabling flexible model alignment. However, applying existing RL methods to timestep-distilled DMs is challenging for ultra-fast (-step) image generation. Our analysis suggests several limitations of policy-based RL methods such as PPO or DPO toward this goal. Based on the insights, we propose fine-tuning DMs with learned differentiable surrogate rewards. Our method, named LaSRO, learns surrogate reward models in the latent space of SDXL to convert arbitrary rewards into differentiable ones for efficient reward gradient guidance. LaSRO leverages pre-trained latent DMs for reward modeling and specifically targets image generation steps for reward optimization, enhancing generalizability and efficiency. LaSRO is effective and stable for improving ultra-fast image generation with different reward objectives, outperforming popular RL methods including PPO and DPO. We further show LaSRO's connection to value-based RL, providing theoretical insights. See our webpage at https://sites.google.com/view/lasro.
BibTeX
@inproceedings{jia2025reward,
  author = {Zhiwei Jia and Yuesong Nan and Huixi Zhao and Gengdai Liu},
  title = {{Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Reward}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=X69Q0ed5rT},
}
Figure