Abstract
Diffusion models are prone to inheriting training data biases and suffering from mode collapse, particularly at high guidance scales. This paper introduces a general reinforcement learning (RL) framework to fine-tune diffusion models for enhanced diversity. By utilizing a novel set-based Diversity Reward and marginal utility for individual image feedback, the framework aligns model outputs with small, unbiased reference sets. Experimental results demonstrate significant improvements in both visual diversity and demographic fairness across multiple benchmarks without sacrificing image quality.
Introduction
While modern diffusion models like Stable Diffusion have revolutionized image generation, they often struggle with "mode collapse" and the amplification of societal biases present in their training data. For instance, prompts for certain professions frequently yield stereotypical demographic results, and increasing classifier-free guidance (CFG) scales—intended to improve prompt faithfulness—often leads to a significant reduction in the variety of generated outputs.
This research proposes a universal reinforcement learning (RL) approach to "teach" models diversity. Unlike previous methods that rely on manual prompt engineering or extensive retraining, this framework fine-tunes models using a small set of unbiased reference images to guide the generation process toward a more representative distribution.
Methodology
The core of the proposed approach is the formulation of the diffusion process as a Multi-step Decision-making Problem (MDP), optimized via policy gradients.
The RL Framework
The fine-tuning process involves generating a batch of images, calculating a collective diversity reward, and then updating the model parameters. To ensure efficiency, the authors employ Low-Rank Adaptation (LoRA), updating only a small fraction of the model's weights.
Diversity Reward and Marginal Utility
A key challenge in optimizing for diversity is that it is a property of a set, not an individual image. The authors introduce a Diversity Reward ($r_D$) that compares a set of generated images ($X_g$) against a reference set ($X_r$). Two primary formulations are explored:
- Maximum Mean Discrepancy (MMD): A kernel-based distance measure between distributions.
- Gaussian Process Mutual Information (GP-MI): A measure of the information coverage provided by the generated set.
To provide the "dense" feedback required for effective RL, the framework calculates an Individual Reward for each image using the concept of marginal utility. This measures how much a specific image contributes to the overall diversity of the set, effectively identifying which samples are most beneficial for expanding the model's output range.
Experimental Evaluation
The framework was evaluated on several benchmarks:
- ImageNet (128x128): For class-conditional generation.
- CIFAR-10/100: For class-balancing comparisons.
- CelebA: To source reference sets for debiasing face generation in Stable Diffusion (e.g., balancing gender or age attributes).
Results
The RL fine-tuning approach yielded significant improvements across metrics:
- Diversity Metrics: On ImageNet, the "Recall" metric (measuring distribution coverage) increased from 36.15 to 47.66 at a guidance scale of 4.0.
- Image Quality: Notably, image quality (measured by FID) often improved alongside diversity, suggesting that the model becomes more robust rather than just more varied.
- Bias Mitigation: In text-to-image tasks, the model successfully shifted from biased distributions (e.g., predominantly male programmers) to balanced 50/50 splits using as few as 50 reference images.
Conclusion and Future Work
This work demonstrates that RL is a powerful tool for addressing the inherent biases and collapse issues in diffusion models. By shifting from instance-based to set-based rewards, the framework provides a scalable way to ensure generative AI produces more diverse and equitable content. Future research will focus on reducing the computational overhead of RL fine-tuning and automating the identification of biased attributes to eliminate the need for manually curated reference sets.
Summary updated