Abstract
Reinforcement Learning (RL) has become a cornerstone for aligning generative models with human preferences. However, in text-to-image (T2I) generation, standard RL methods often succumb to "reward hacking," where generators exploit the biases of scalar-based reward models to produce low-quality images that technically score high. We introduce Adv-GRPO, a framework that mitigates reward hacking through adversarial reward training and dense visual signals from vision foundation models (VFMs) like DINOv2. By treating the reward model as a dynamic discriminator and utilizing rich feature distributions, Adv-GRPO achieves superior alignment and aesthetic quality, outperforming baselines like Stable Diffusion 3 (SD3) in human evaluations.
Introduction
The primary challenge in aligning T2I models via RL is the reliance on static preference models (e.g., PickScore). These models provide a single scalar value that often fails to capture the complexity of human visual perception. Consequently, generators learn to "game" these metrics, leading to artifacts like oversaturation or blurred backgrounds. Traditional regularization techniques, such as KL-divergence constraints, often hinder the model's ability to reach peak performance. Adv-GRPO addresses this by shifting the paradigm from scalar rewards to an "image-as-reward" approach, leveraging adversarial training to maintain a robust and evolving reward signal.
Methodology
Adv-GRPO is built upon the Group Relative Policy Optimization (GRPO) algorithm and incorporates three key innovations to enhance stability and quality:
- Adversarial Reward Training: Instead of a fixed reward model, Adv-GRPO employs a discriminator that is iteratively updated alongside the generator. This setup, reminiscent of Generative Adversarial Networks (GANs), forces the reward model to distinguish between high-quality reference images and generated samples, preventing the generator from finding easy shortcuts.
- Dense Visual Signals: The framework utilizes Visual Foundation Models (VFMs) like DINOv2 or SigLIP to extract both global and local patch-level features. This provides the generator with a holistic understanding of image composition and texture, far exceeding the information provided by a simple scalar score.
- Multi-Reward Optimization: For specialized tasks like Optical Character Recognition (OCR), the model combines task-specific rewards with visual similarity metrics. This ensures that the model can render accurate text without compromising the overall aesthetic fidelity of the image.
Experimental Setup and Datasets
The researchers evaluated Adv-GRPO using several benchmarks:
- PickScore & OCR Prompt Sets: For general preference and text-rendering training.
- GenEval: To assess compositional abilities.
- Reference Anchors: High-quality images generated by Qwen-Image served as the "ground truth" distribution for the adversarial discriminator.
- Style Customization: Small datasets of anime and sci-fi images were used to demonstrate the model's ability to adapt to specific visual domains via RL.
Results and Discussion
Adv-GRPO significantly outperforms existing baselines in human preference tests. When compared to the base SD3 model, the framework achieved a 72.4% win rate in aesthetics. In comparisons with Flow-GRPO using the PickScore reward, Adv-GRPO maintained a 70.0% win rate in image quality.
A critical success of the adversarial approach is its ability to maintain visual clarity in complex tasks. For instance, in OCR tasks, while traditional RL might blur the background to focus on text accuracy, Adv-GRPO preserves high aesthetic fidelity (maintaining an OCR accuracy of 0.91). This demonstrates that the adversarial signal effectively prevents the "stagnation" or "hacking" typical of static reward models.
Conclusion
Adv-GRPO represents a significant step forward in the alignment of text-to-image models. By moving away from static scalar rewards and embracing dynamic, adversarial feedback from vision foundation models, the framework effectively eliminates the common pitfalls of reward hacking. Future research will focus on scaling this adversarial training to even larger architectures and exploring a broader range of foundation models as reward signals.
Summary updated