DiffExp: Efficient Exploration in Reward Fine-tuning for Text-to-Image Diffusion Models

Daewon Chae, June Suk Choi, Jinkyu Kim, Kimin Lee

Introduction: The Exploration Challenge in Diffusion Fine-tuning

Fine-tuning text-to-image diffusion models to maximize specific rewards—such as aesthetic scores or human preference—is fundamentally a Reinforcement Learning (RL) problem. In this setup, the model generates images online, receives a reward signal, and updates its weights accordingly.

However, a significant bottleneck exists: the exploration-exploitation trade-off. If a base model is initially poor at a specific complex prompt (e.g., "a dolphin riding a bike"), it is unlikely to generate the necessary elements (the bike) by chance. Without these elements, the reward model provides no positive reinforcement, causing the fine-tuning process to stall or collapse into trivial solutions, such as simply improving the image quality of the dolphin without ever addressing the "riding a bike" component.

The DiffExp Framework

To address this, DiffExp (DiffusionExplore) introduces two lightweight exploration strategies designed to help models discover high-reward samples more efficiently during the fine-tuning phase.

1. Dynamic CFG Scale Scheduling

Classifier-Free Guidance (CFG) typically balances image quality and diversity using a fixed scale. DiffExp employs a "low-then-high" schedule. During the early stages of the denoising process (high $t$), a low CFG scale is used to encourage structural diversity and "wandering" in the latent space. In the later stages (low $t$), the scale is increased to ensure the final output is high-quality and well-defined.

2. Random Prompt Weighting

To ensure the model attends to neglected parts of a prompt, DiffExp randomly selects a word within the text prompt and boosts its embedding weight (by a factor of 1.0 to 1.2) during sampling. This "nudges" the model toward generating specific elements it might otherwise ignore, providing the reward model with the necessary signals to reinforce complex compositions.

Experimental Results

Sample Efficiency

One of the primary advantages of DiffExp is its sample efficiency. By generating more diverse and relevant training samples, the method reaches target reward levels using approximately 20% fewer reward queries than standard baselines like DDPO or AlignProp.

Reward improvement vs. number of queries. DiffExp achieves significantly higher rewards with fewer samples compared to the DDPO baseline.

Qualitative Superiority and Generalization

DiffExp excels at "impossible" prompts where baselines typically fail. For instance, on the task of generating a dolphin riding a bike, standard DDPO often fails to produce a bicycle at all. In contrast, DiffExp successfully generates the composition and refines it over time.

Qualitative results for the prompt 'a dolphin riding a bike'. DiffExp successfully learns the complex composition where standard fine-tuning methods often fail.

Furthermore, testing on unseen prompts using the ImageReward metric shows that DiffExp-enhanced models significantly outperform baselines. This suggests that the model learns robust, generalizable features rather than simply "reward hacking" the specific training set.

Conclusion

DiffExp demonstrates that the primary hurdle in reward fine-tuning for diffusion models is not the optimization algorithm, but the lack of diverse, high-reward samples. By utilizing dynamic CFG scheduling and random prompt weighting, DiffExp provides a simple yet effective way to unlock the potential of reward-based training, even for complex and challenging compositions.

Summary updated

Abstract

Fine-tuning text-to-image diffusion models to maximize rewards has proven effective for enhancing model performance. However, reward fine-tuning methods often suffer from slow convergence due to online sample generation. Therefore, obtaining diverse samples with strong reward signals is crucial for improving sample efficiency and overall performance. In this work, we introduce DiffExp, a simple yet effective exploration strategy for reward fine-tuning of text-to-image models. Our approach employs two key strategies: (a) dynamically adjusting the scale of classifier-free guidance to enhance sample diversity, and (b) randomly weighting phrases of the text prompt to exploit high-quality reward signals. We demonstrate that these strategies significantly enhance exploration during online sample generation, improving the sample efficiency of recent reward fine-tuning methods, such as DDPO and AlignProp.
BibTeX
@inproceedings{chae2025diffexp,
  author = {Daewon Chae and June Suk Choi and Jinkyu Kim and Kimin Lee},
  title = {{DiffExp: Efficient Exploration in Reward Fine-tuning for Text-to-Image Diffusion Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=f97CTNR9K6},
}
Figure