TL;DR
This paper introduces DiffExp, a novel exploration strategy to accelerate reward fine-tuning for text-to-image diffusion models. By dynamically adjusting classifier-free guidance (CFG) and randomly weighting prompt phrases, DiffExp significantly improves sample efficiency and the quality of generated images, especially for challenging prompts.
What Problem Does This Paper Address?
Text-to-image diffusion models are powerful, but fine-tuning them to align with specific rewards (like aesthetic quality or image-text alignment) can be slow. This is because these methods often rely on generating many samples online to learn from reward signals. If the generated samples don't consistently provide strong reward signals, the model converges slowly or gets stuck in suboptimal solutions. This paper tackles the challenge of improving the sample efficiency of reward fine-tuning, making the process faster and more effective, particularly for complex prompts that are difficult for existing methods to handle.
Methods
DiffExp introduces two core strategies to enhance exploration during reward fine-tuning:
- Dynamic CFG Scale Scheduling: Classifier-free guidance (CFG) is a key parameter in diffusion models that balances sample quality and diversity. Typically, it's kept constant. DiffExp proposes to start with a low CFG scale to encourage diverse samples and then gradually increase it. This dynamic adjustment helps discover a wider range of outputs while maintaining good image quality.
- Random Prompt Weighting: This technique involves randomly selecting a word (token) within the text prompt and slightly increasing its weight in the prompt embedding. This emphasis encourages the model to generate images that more strongly represent that specific word, leading to more diverse and potentially higher-rewarding outputs.
These two strategies are integrated into existing reward fine-tuning frameworks like DDPO (policy gradient) and AlignProp (direct backpropagation).
Datasets
The paper uses existing datasets and prompts for training and evaluation. Specific datasets mentioned include:
- Animal names and activities: Used for training prompts with Aesthetic and PickScore reward functions.
- DrawBench: A benchmark of challenging prompts used to evaluate the model's ability to handle complex generation tasks.
- Parti prompts: Used as unseen evaluation prompts to test generalization.
The paper does not introduce new datasets but leverages existing ones to demonstrate the effectiveness of DiffExp.
Key Insights and Ideas
- Exploration is Key for Reward Fine-tuning: The core insight is that simply maximizing rewards isn't enough; the model needs to explore effectively to find those high-rewarding samples in the first place. Slow convergence in existing methods is a direct consequence of insufficient exploration.
- Dynamic CFG is More Than Just a Trade-off: Instead of a fixed trade-off, dynamically adjusting CFG allows for a more nuanced approach, balancing diversity early on and quality later, leading to better overall learning.
- Prompt Manipulation for Targeted Diversity: Randomly emphasizing parts of a prompt is a surprisingly simple yet effective way to steer generation towards more diverse and potentially better-aligned outputs, especially for prompts that are known to be tricky.
- Generalization is Improved: DiffExp doesn't just help with the prompts it's trained on; it also improves the model's ability to generalize to unseen prompts, suggesting a more robust learning process.
Key Results
- Improved Sample Efficiency: DiffExp consistently reduced the number of samples required to achieve a target reward score by approximately 20% compared to baselines like DDPO and AlignProp.
- Higher Reward Scores: The method achieved higher reward scores across various reward functions (Aesthetic, PickScore, ImageReward) and fine-tuning methods.
- Enhanced Image Quality: Qualitative results show that DiffExp-enhanced models generate images with comparable or better aesthetic quality and significantly improved image-text alignment, especially for challenging prompts.
- Effective on SDXL: The approach demonstrated strong performance when applied to the state-of-the-art SDXL model, indicating its broad applicability.
Limitations and Future Work
- Hyperparameter Sensitivity: While the paper explores the impact of the
t_thresparameter for CFG scheduling, it's likely that other hyperparameters related to prompt weighting and CFG scheduling might require careful tuning for optimal performance across different tasks. - Computational Cost: While sample efficiency is improved, the exploration strategies themselves might add a slight computational overhead during the fine-tuning process.
- Future Work: The authors suggest further exploring more advanced prompt manipulation techniques and investigating the interplay of exploration strategies with different diffusion model architectures and reward functions. They also highlight the potential for DiffExp to be applied to other generative model fine-tuning tasks beyond text-to-image.
Summary updated