Fake it till You Make it: Reward Modeling as Discriminative Prediction

Runtao Liu, Jiahao Zhan, Yingqing He, Chen Wei, Alan Yuille, Qifeng ChenSubmission to ICLR 2026 · Rejected

TL;DR

Training reward models for visual generative models typically requires hundreds of thousands of expensive human pairwise annotations or complex multi-metric engineering. GAN-RM revives the core insight of GAN discriminators: it trains a reward model simply by distinguishing between standard model outputs and just a few hundred unpaired, high-quality images (termed Preference Proxy Data). Despite using orders of magnitude less data, GAN-RM matches or outperforms models trained on over one million human preference labels across inference-time sample selection, supervised fine-tuning (SFT), and direct preference optimization (DPO).

What Problem Does This Paper Address?

Reinforcement learning from human feedback (RLHF) and preference alignment are crucial for steering visual models away from artifacts, safety violations, and poor aesthetics. However, building reliable reward models in computer vision is notoriously painful:

  1. Extreme Data Costs: Leading preference models (e.g., PickScore, ImageReward) rely on hundreds of thousands to millions of crowdsourced preference pairs.
  2. Domain Mismatch & Overfitting: Datasets annotated on one generator’s outputs do not transfer cleanly to another, and optimizing against a single preference reward often leads to "reward hacking" (improving one metric while cratering others like image-text alignment).
  3. Engineering Overhead: Multi-dimensional evaluations (such as those in video generation) require designing complex ensembles of dozens of heuristic vision models.

GAN-RM addresses this bottleneck by asking: Can we learn an effective visual reward model without any paired preference labels or complex quality heuristics?

Methods

[ Unpaired High-Quality Proxy Data (e.g., 500 images) ]  --> Label: 1
                                                                 \   Binary Cross-Entropy
                                                                  --> [ GAN-RM (Frozen CLIP + MLP) ]
                                                                 /
[ Raw Generator Output Samples ] ------------------------------> Label: 0
  • Preference Proxy Data (PPD): Instead of paired comparisons, the user provides a small set ($N \approx 500$) of unpaired, representative target samples that capture the target style, high aesthetic quality, or safety criteria.
  • Discriminative Reward Architecture: GAN-RM leverages a frozen vision backbone (CLIP-Vision for images, ViCLIP for video) attached to a lightweight Multi-Layer Perceptron (MLP) called the Reward Projection Layer (RPL). The RPL outputs a confidence score via softmax indicating the probability that a sample belongs to the PPD distribution rather than the generator's raw output.
  • Rank-Based Bootstrapping: To help GAN-RM generalize across the generative model's output space, the authors generate candidate images for additional prompts and score them using the initial GAN-RM. High- and low-scoring samples are assigned soft pseudo-labels decayed by rank ($y = e^{-\alpha \cdot r}$) and merged into the training set.
  • Downstream Optimization Modes:
  • Best-of-$N$ Inference Scaling: Generates $K$ candidates per prompt and selects the top-scoring output without fine-tuning.
  • SFT & DPO: Uses GAN-RM to filter top samples for supervised fine-tuning or pairs top ($x^h$) and bottom ($x^l$) samples for Direct Preference Optimization.
  • Iterative Multi-Round DPO: The generator is updated via DPO, and new samples from the updated generator serve as harder negative examples to retrain GAN-RM in subsequent rounds.

Datasets

  • Image Quality Experiments:
  • PPD: 500 images sampled from JourneyDB. Prompts sampled from JourneyDB for synthetic generation.
  • Advanced Models (Flux): 500 photography samples from Photozilla.
  • Safety Alignment Experiments:
  • PPD: 15,690 safe images and unsafe prompts from CoProV2 (with an ablation testing down to 2,000 samples). Evaluated on MS-COCO.
  • Video Alignment Experiments:
  • PPD: 500 high-quality clips from Artgrid. 5,000 prompts sampled from VidProm.
  • Baselines / Comparison Datasets: Pick-a-Pic (1M human preference pairs used by DiffusionDPO and PickScore), ImageReward (137k comparisons), and HPS v2.

Key Insights and Ideas

  • "Fake it till you make it" works with foundation encoders: GAN discriminators historically struggled with stability and mode collapse when trained from scratch. However, building a discriminative head on top of frozen, highly generalized vision foundation models (CLIP) stabilizes training completely, even when trained on as few as 250–500 target images.
  • Mitigating Reward Hacking: Models fine-tuned directly on single existing reward models (e.g., PickScore or ImageReward) overfit to those specific metrics, dropping sharply in CLIPScore or FID. Because GAN-RM models the target distribution holistically as a density ratio, it achieves balanced improvements across FID, ImageReward, PickScore, and CLIPScore simultaneously.
  • Monotonic Test-Time Scaling: When sorting generated samples by GAN-RM score, higher-ranked samples monotonically track improvements in FID, PickScore, ImageReward, and HPS. Sampling more candidates ($K$) yields log-linear FID improvements.
  • Co-evolution via Multi-Turn DPO: Traditional DPO uses static datasets. Because GAN-RM continuously discriminates against the current generator policy, it can be updated alongside the policy across multiple training rounds to prevent reward saturation.

Key Results

  • Text-to-Image Quality (SD 1.5 & SDXL):
  • On Stable Diffusion 1.5, Ours-DPO (trained on 0.5k unpaired images) achieved an FID of 63.61, outperforming DiffusionDPO (FID 68.15)—which required 1 million human-labeled pairs—and the base model (FID 72.06).
  • User studies showed evaluators preferred images aligned with GAN-RM over base SD 1.5 74.4% to 25.6%. Human agreement with GAN-RM rankings reached 70.5%.
  • Safety Alignment: On SD 1.5, GAN-RM dropped the Inappropriate Probability (IP) metric from 42 to 18 (and from 51 to 17 on SDXL) while retaining text-image alignment and avoiding catastrophic distribution drift.
  • Video Generation (VideoCrafter2): Fine-tuning VideoCrafter2 with GAN-RM dropped FVD from 1021.77 to 983.28 and boosted VBench score to 81.38, performing comparably to VideoDPO without requiring its compute-heavy ensemble of expert models.
  • Generalization to Modern Architectures: Applied as a Best-of-10 selector on FLUX, GAN-RM lowered FID from 145.3 to 140.9 while improving ImageReward and PickScore.

Limitations and Future Work

  • Sensitivity to Proxy Data Quality: Ablations show that contaminating the PPD with lower-quality samples or using low prompt diversity severely degrades performance. Curating a high-quality, diverse set is mandatory.
  • Extreme Low-Data Boundaries: Performance remains robust down to ~250 samples, but drops when the proxy dataset falls below 50 samples.
  • Integration with Newer RL Objectives: The paper demonstrated GAN-RM on SFT and standard DPO; the authors note that combining GAN-RM with recent algorithmic variants like Step-aware Preference Optimization (SPO) or Negative Preference Optimization (NPO) remains future work.

Summary updated

Abstract

An effective reward model plays a pivotal role in reinforcement learning for post-training enhancement of visual generative models. However, current approaches of reward modeling suffer from implementation complexity due to their reliance on extensive human-annotated preference data or meticulously engineered quality dimensions that are often incomplete and engineering-intensive. Inspired by adversarial training in generative adversarial networks (GANs), this paper proposes GAN-RM, an efficient reward modeling framework that eliminates manual preference annotation and explicit quality dimension engineering. Our method trains the reward model through discrimination between a small set of representative, unpaired target samples(denoted as Preference Proxy Data) and model-generated ordinary outputs, requiring only a few hundred target samples. Comprehensive experiments demonstrate our GAN-RM's effectiveness across multiple key applications including test-time scaling implemented as Best-of-N sample filtering, post-training approaches like Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO).
BibTeX
@misc{liu2025fake,
  author = {Runtao Liu and Jiahao Zhan and Yingqing He and Chen Wei and Alan Yuille and Qifeng Chen},
  title = {{Fake it till You Make it: Reward Modeling as Discriminative Prediction}},
  year = {2025},
  eprint = {2506.13846},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.13846},
}
Figure