Introduction: The Exploration Challenge in Diffusion Fine-tuning
Fine-tuning text-to-image diffusion models to maximize specific rewards—such as aesthetic scores or human preference—is fundamentally a Reinforcement Learning (RL) problem. In this setup, the model generates images online, receives a reward signal, and updates its weights accordingly.
However, a significant bottleneck exists: the exploration-exploitation trade-off. If a base model is initially poor at a specific complex prompt (e.g., "a dolphin riding a bike"), it is unlikely to generate the necessary elements (the bike) by chance. Without these elements, the reward model provides no positive reinforcement, causing the fine-tuning process to stall or collapse into trivial solutions, such as simply improving the image quality of the dolphin without ever addressing the "riding a bike" component.
The DiffExp Framework
To address this, DiffExp (DiffusionExplore) introduces two lightweight exploration strategies designed to help models discover high-reward samples more efficiently during the fine-tuning phase.
1. Dynamic CFG Scale Scheduling
Classifier-Free Guidance (CFG) typically balances image quality and diversity using a fixed scale. DiffExp employs a "low-then-high" schedule. During the early stages of the denoising process (high $t$), a low CFG scale is used to encourage structural diversity and "wandering" in the latent space. In the later stages (low $t$), the scale is increased to ensure the final output is high-quality and well-defined.
2. Random Prompt Weighting
To ensure the model attends to neglected parts of a prompt, DiffExp randomly selects a word within the text prompt and boosts its embedding weight (by a factor of 1.0 to 1.2) during sampling. This "nudges" the model toward generating specific elements it might otherwise ignore, providing the reward model with the necessary signals to reinforce complex compositions.
Experimental Results
Sample Efficiency
One of the primary advantages of DiffExp is its sample efficiency. By generating more diverse and relevant training samples, the method reaches target reward levels using approximately 20% fewer reward queries than standard baselines like DDPO or AlignProp.
Qualitative Superiority and Generalization
DiffExp excels at "impossible" prompts where baselines typically fail. For instance, on the task of generating a dolphin riding a bike, standard DDPO often fails to produce a bicycle at all. In contrast, DiffExp successfully generates the composition and refines it over time.
Furthermore, testing on unseen prompts using the ImageReward metric shows that DiffExp-enhanced models significantly outperform baselines. This suggests that the model learns robust, generalizable features rather than simply "reward hacking" the specific training set.
Conclusion
DiffExp demonstrates that the primary hurdle in reward fine-tuning for diffusion models is not the optimization algorithm, but the lack of diverse, high-reward samples. By utilizing dynamic CFG scheduling and random prompt weighting, DiffExp provides a simple yet effective way to unlock the potential of reward-based training, even for complex and challenging compositions.
Summary updated