Unified Reward Model for Multimodal Understanding and Generation

Yibin Wang, Yuhang Zang, Hao Li, Cheng Jin, Jiaqi WangSubmission to ICLR 2026 · Withdrawn

TL;DR

Existing multimodal reward models are heavily siloed—specialized strictly for either image or video, and either understanding or generation. UnifiedReward breaks this fragmentation by training a single Vision-Language Model to act as a general critic across all four quadrants using both pairwise ranking and pointwise scoring. Leveraging cross-task synergies, the unified model outperforms task-specific reward models and enables high-precision Direct Preference Optimization (DPO) for both vision understanding and generative models.

What Problem Does This Paper Address?

Aligning vision models with human preferences via reinforcement learning or Direct Preference Optimization (DPO) relies on accurate reward signals. However, the current landscape is fragmented: models like PickScore and ImageReward evaluate text-to-image synthesis, VideoScore targets video generation, and LLaVA-Critic evaluates image comprehension.

This compartmentalization has two major drawbacks:

  1. Lack of generalizability: Practitioners must maintain completely different reward models for different modalities and visual tasks.
  2. Missed cross-task synergies: Evaluating visual generation inherently requires understanding semantic content and visual coherence, while evaluating video frames relies directly on static image critique. Training isolated critics wastes these natural interconnections.

Methods

The authors build UnifiedReward (based on LLaVA-OneVision 7B and Qwen2.5-VL) and deploy it in an end-to-end alignment pipeline:

  [Candidate Outputs] 
           │
           ▼
   (1) Pair Ranking      ──► Splits into Chosen (C) & Rejected (R)
           │
           ▼
   (2) Point Sifting     ──► Picks Max Score from C and Min Score from R
           │
           ▼
   [High-Margin DPO Pairs] ──► Optimizes VLMs & Diffusion Models
  • Unified Multi-Task Critic Training: The model is jointly fine-tuned across diverse tasks to predict either an absolute pointwise score (1–5 scale, with rationales when available) or a relative pairwise ranking ("Image 1 is better than Image 2") conditioned on task-specific instructions.
  • Two-Stage Preference Curation (Pair Ranking + Point Sifting): To generate clean DPO datasets from unaligned vision models:
  • Candidate Generation: The target model produces $N=10$ candidate responses or generations.
  • Pair Ranking: Candidates are paired, and UnifiedReward ranks them into preliminary "chosen" and "rejected" pools.
  • Point Sifting: UnifiedReward assigns absolute scores to all candidates; the top-scoring candidate from the chosen pool and lowest-scoring candidate from the rejected pool are selected. This widens the quality gap and removes noisy ties.
  • Direct Preference Optimization (DPO): The filtered pairs are used to fine-tune diffusion models (SDXL-Turbo, T2V-Turbo) for visual generation and VLMs (LLaVA-OneVision, LLaVA-Video) for visual understanding.

Datasets

The authors curate an aggregated ~236K preference dataset spanning:

  • Image Generation: EvalMuse (35.7K pairs/points), HPD (25.6K pairs), Open-Image-Preferences (7.4K pairs).
  • Image Understanding: LLaVA-Critic (50K pairs/points).
  • Video Generation: VideoDPO (10K pairs), LiFT-HRA (20K points), VideoFeedback (36.6K points).
  • Video Understanding: ShareGPTVideo-DPO (51K pairs/points).

Evaluation Benchmarks: - Reward Assessment: VLRewardBench (image understanding), ShareGPTVideo test set (video understanding), GenAI-Bench (image & video generation), and VideoGen-RewardBench. - Downstream Alignment Benchmarks: LLaVABench, WildVision, MathVista, MMBench, MME, etc. (for image VQA); MSRVTT, MSVD, TGIF, Video-MME, MLVU (for video VQA); Partiprompt, HPSv2, and VBench (for generation).

Key Insights and Ideas

  • Mutual Reinforcement Over Task Interference: Jointly training across modalities does not lead to negative transfer. Instead, multi-task assessment provides massive gains: adding image understanding and generation data boosted video understanding accuracy from 74.2% to 84.0%.
  • Point Sifting Amplifies Preference Margins: Preference alignment degrades when chosen and rejected pairs are nearly indistinguishable. Combining coarse pairwise classification with fine-grained pointwise scoring ensures that the DPO loss operates on pairs with genuine contrast.
  • Unified Beats Specialized: Despite evaluating across four distinct tasks, UnifiedReward beats purpose-built critics (such as VideoScore for video generation or LLaVA-Critic for multimodal reasoning) on their own respective benchmarks.

Key Results

  • Reward Benchmarking:
  • Image Understanding (VLRewardBench): UnifiedReward achieved 66.5% macro accuracy, significantly surpassing LLaVA-Critic (46.6%), GPT-4o (62.4%), and Gemini-1.5-Pro (62.5%).
  • Video Generation (GenAI-Bench): Outperformed dedicated video reward models, achieving 60.7 tau / 77.2 diff vs. VideoScore (46.2 / 70.6) and VideoReward (50.2 / 73.3).
  • Post-DPO Model Enhancements:
  • Image Understanding: LLaVA-OneVision-7B aligned with UnifiedReward scored 101.4 on LLaVABench (up from 90.3 baseline, and beating LLaVA-Critic DPO at 100.3) and reduced hallucinations on MMHal (score improved from 3.19 to 4.01).
  • Video Understanding: LLaVA-Video-7B jumped from 52.8% to 65.0% accuracy on MSRVTT and 69.7% to 78.3% on MSVD.
  • Image Generation: SDXL-Turbo DPO improved its PickScore from 43.24 (baseline) and 54.32 (direct Pick-a-Pic training) to 63.32 using UnifiedReward's curated pairs.

Limitations and Future Work

  • Pair Data Asymmetry: High-quality pairwise ranking data for video generation remains scarce relative to image generation, leaving video ranking reliant on multi-task transfer.
  • Compute Overhead During Curation: The two-stage data generation protocol requires sampling 10 outputs per prompt followed by both pairwise ranking passes and pointwise inference, making offline data filtering computationally expensive.
  • Annotation Biases: Inheriting diverse datasets means potentially absorbing contradictory rating scales or idiosyncratic evaluator biases across sources.

Summary updated

Abstract

Recent advances in human preference alignment have significantly enhanced multimodal generation and understanding. A key approach is training reward models to guide preference optimization. However, existing reward models are often task-specific, limiting their adaptability across diverse visual applications. We also argue that a reward model jointly learning to assess multiple vision tasks may foster a synergistic effect, where improved image understanding enhances image generation assessment, and refined image evaluation benefits video assessment through better frame analysis. To this end, this paper proposes UNIFIEDREWARD, the first unified reward model for multimodal understanding and generation assessment, enabling both pairwise ranking and pointwise scoring, which can be employed for vision model preference alignment. Specifically, (1) we first develop UNIFIEDREWARD on our constructed large-scale human preference dataset, including both image and video generation/understanding tasks. (2) Then, it is utilized to automatically construct high-quality preference pair data based on the vision models, fine-gradually filtering their outputs through pair ranking and point sifting. (3) Finally, these data are used for their preference alignment through Direct Preference Optimization (DPO). Experimental results demonstrate that joint learning to assess diverse visual tasks can lead to substantial mutual benefits, and we apply our pipeline to both image and video understanding/generation tasks, significantly improving the performance in each domain.
BibTeX
@misc{wang2025unified,
  author = {Yibin Wang and Yuhang Zang and Hao Li and Cheng Jin and Jiaqi Wang},
  title = {{Unified Reward Model for Multimodal Understanding and Generation}},
  year = {2025},
  eprint = {2503.05236},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.05236},
}

Loading related content...

Figure