The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generation

Weijia Mao, Hao Chen, Zhenheng Yang, Mike Zheng Shou

Abstract

Reinforcement Learning (RL) has become a cornerstone for aligning generative models with human preferences. However, in text-to-image (T2I) generation, standard RL methods often succumb to "reward hacking," where generators exploit the biases of scalar-based reward models to produce low-quality images that technically score high. We introduce Adv-GRPO, a framework that mitigates reward hacking through adversarial reward training and dense visual signals from vision foundation models (VFMs) like DINOv2. By treating the reward model as a dynamic discriminator and utilizing rich feature distributions, Adv-GRPO achieves superior alignment and aesthetic quality, outperforming baselines like Stable Diffusion 3 (SD3) in human evaluations.

Introduction

The primary challenge in aligning T2I models via RL is the reliance on static preference models (e.g., PickScore). These models provide a single scalar value that often fails to capture the complexity of human visual perception. Consequently, generators learn to "game" these metrics, leading to artifacts like oversaturation or blurred backgrounds. Traditional regularization techniques, such as KL-divergence constraints, often hinder the model's ability to reach peak performance. Adv-GRPO addresses this by shifting the paradigm from scalar rewards to an "image-as-reward" approach, leveraging adversarial training to maintain a robust and evolving reward signal.

Methodology

Adv-GRPO is built upon the Group Relative Policy Optimization (GRPO) algorithm and incorporates three key innovations to enhance stability and quality:

  1. Adversarial Reward Training: Instead of a fixed reward model, Adv-GRPO employs a discriminator that is iteratively updated alongside the generator. This setup, reminiscent of Generative Adversarial Networks (GANs), forces the reward model to distinguish between high-quality reference images and generated samples, preventing the generator from finding easy shortcuts.
  2. Dense Visual Signals: The framework utilizes Visual Foundation Models (VFMs) like DINOv2 or SigLIP to extract both global and local patch-level features. This provides the generator with a holistic understanding of image composition and texture, far exceeding the information provided by a simple scalar score.
  3. Multi-Reward Optimization: For specialized tasks like Optical Character Recognition (OCR), the model combines task-specific rewards with visual similarity metrics. This ensures that the model can render accurate text without compromising the overall aesthetic fidelity of the image.
Overview of the Adv-GRPO framework, highlighting its three core components: alleviating reward hacking, utilizing VFMs as rewards, and RL-based distribution transfer.

Experimental Setup and Datasets

The researchers evaluated Adv-GRPO using several benchmarks:

  • PickScore & OCR Prompt Sets: For general preference and text-rendering training.
  • GenEval: To assess compositional abilities.
  • Reference Anchors: High-quality images generated by Qwen-Image served as the "ground truth" distribution for the adversarial discriminator.
  • Style Customization: Small datasets of anime and sci-fi images were used to demonstrate the model's ability to adapt to specific visual domains via RL.

Results and Discussion

Adv-GRPO significantly outperforms existing baselines in human preference tests. When compared to the base SD3 model, the framework achieved a 72.4% win rate in aesthetics. In comparisons with Flow-GRPO using the PickScore reward, Adv-GRPO maintained a 70.0% win rate in image quality.

A critical success of the adversarial approach is its ability to maintain visual clarity in complex tasks. For instance, in OCR tasks, while traditional RL might blur the background to focus on text accuracy, Adv-GRPO preserves high aesthetic fidelity (maintaining an OCR accuracy of 0.91). This demonstrates that the adversarial signal effectively prevents the "stagnation" or "hacking" typical of static reward models.

Quantitative win rates and visual comparisons between Adv-GRPO and Flow-GRPO, demonstrating the prevention of reward hacking in quality and OCR tasks.

Conclusion

Adv-GRPO represents a significant step forward in the alignment of text-to-image models. By moving away from static scalar rewards and embracing dynamic, adversarial feedback from vision foundation models, the framework effectively eliminates the common pitfalls of reward hacking. Future research will focus on scaling this adversarial training to even larger architectures and exploring a broader range of foundation models as reward signals.

Summary updated

BibTeX
@inproceedings{mao2026image,
  author = {Weijia Mao and Hao Chen and Zhenheng Yang and Mike Zheng Shou},
  title = {{The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=5HOTpLhVmD},
}
Figure