Training Diffusion Models Towards Diverse Image Generation with Reinforcement Learning

Zichen Miao, Jiang Wang, Ze Wang, Zhengyuan Yang, Lijuan Wang, Qiang Qiu, Zicheng Liu

Abstract

Diffusion models are prone to inheriting training data biases and suffering from mode collapse, particularly at high guidance scales. This paper introduces a general reinforcement learning (RL) framework to fine-tune diffusion models for enhanced diversity. By utilizing a novel set-based Diversity Reward and marginal utility for individual image feedback, the framework aligns model outputs with small, unbiased reference sets. Experimental results demonstrate significant improvements in both visual diversity and demographic fairness across multiple benchmarks without sacrificing image quality.

Introduction

While modern diffusion models like Stable Diffusion have revolutionized image generation, they often struggle with "mode collapse" and the amplification of societal biases present in their training data. For instance, prompts for certain professions frequently yield stereotypical demographic results, and increasing classifier-free guidance (CFG) scales—intended to improve prompt faithfulness—often leads to a significant reduction in the variety of generated outputs.

Overview of RL-FT for Diversity, showing how it corrects gender bias and increases visual variety.

This research proposes a universal reinforcement learning (RL) approach to "teach" models diversity. Unlike previous methods that rely on manual prompt engineering or extensive retraining, this framework fine-tunes models using a small set of unbiased reference images to guide the generation process toward a more representative distribution.

Methodology

The core of the proposed approach is the formulation of the diffusion process as a Multi-step Decision-making Problem (MDP), optimized via policy gradients.

The RL Framework

The fine-tuning process involves generating a batch of images, calculating a collective diversity reward, and then updating the model parameters. To ensure efficiency, the authors employ Low-Rank Adaptation (LoRA), updating only a small fraction of the model's weights.

Schematic of the RL fine-tuning loop, incorporating diversity and individual rewards.

Diversity Reward and Marginal Utility

A key challenge in optimizing for diversity is that it is a property of a set, not an individual image. The authors introduce a Diversity Reward ($r_D$) that compares a set of generated images ($X_g$) against a reference set ($X_r$). Two primary formulations are explored:

  • Maximum Mean Discrepancy (MMD): A kernel-based distance measure between distributions.
  • Gaussian Process Mutual Information (GP-MI): A measure of the information coverage provided by the generated set.

To provide the "dense" feedback required for effective RL, the framework calculates an Individual Reward for each image using the concept of marginal utility. This measures how much a specific image contributes to the overall diversity of the set, effectively identifying which samples are most beneficial for expanding the model's output range.

Experimental Evaluation

The framework was evaluated on several benchmarks:

  • ImageNet (128x128): For class-conditional generation.
  • CIFAR-10/100: For class-balancing comparisons.
  • CelebA: To source reference sets for debiasing face generation in Stable Diffusion (e.g., balancing gender or age attributes).

Results

The RL fine-tuning approach yielded significant improvements across metrics:

  • Diversity Metrics: On ImageNet, the "Recall" metric (measuring distribution coverage) increased from 36.15 to 47.66 at a guidance scale of 4.0.
  • Image Quality: Notably, image quality (measured by FID) often improved alongside diversity, suggesting that the model becomes more robust rather than just more varied.
  • Bias Mitigation: In text-to-image tasks, the model successfully shifted from biased distributions (e.g., predominantly male programmers) to balanced 50/50 splits using as few as 50 reference images.

Conclusion and Future Work

This work demonstrates that RL is a powerful tool for addressing the inherent biases and collapse issues in diffusion models. By shifting from instance-based to set-based rewards, the framework provides a scalable way to ensure generative AI produces more diverse and equitable content. Future research will focus on reducing the computational overhead of RL fine-tuning and automating the identification of biased attributes to eliminate the need for manually curated reference sets.

Summary updated

Abstract

Diffusion models have demonstrated unprecedented capabilities in image generation. Yet, they incorporate and amplify the data bias (e.g., gender, age) from the original training set, limiting the diversity of generated images. In this paper, we propose a diversity-oriented fine-tuning method using reinforcement learning (RL) for diffusion models under the guidance of an image-set-based reward function. Specifically, the proposed reward function, denoted as Diversity Reward, utilizes a set of generated images to evaluate the coverage of the current generative distribution w.r.t. the reference distribution, represented by a set of unbiased images. Built on top of the probabilistic method of distribution discrepancy estimation, Diversity Reward can measure the relative distribution gap with a small set of images efficiently. We further formulate the diffusion process as a multi-step decision-making problem (MDP) and apply policy gradient methods to fine-tune diffusion models by maximizing the Diversity Reward. The proposed rewards are validated on a post-sampling selection task, where a subset of the most diverse images are selected based on Diversity Reward values. We also show the effectiveness of our RL fine-tuning framework on enhancing the diversity of image generation with different types of diffusion models, including class-conditional models and text-conditional models, e.g., StableDiffusion.
BibTeX
@inproceedings{miao2024training,
  author = {Zichen Miao and Jiang Wang and Ze Wang and Zhengyuan Yang and Lijuan Wang and Qiang Qiu and Zicheng Liu},
  title = {{Training Diffusion Models Towards Diverse Image Generation with Reinforcement Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01031},
}
Figure