VideoScore2: Think before You Score in Generative Video Evaluation

Xuan He, Dongfu Jiang, Ping Nie, Minghao Liu, Zhengxuan Jiang, Mingyi Su, Wentao Ma, Junru Lin, Chun Ye, Yi Lu, Keming Wu, Benjamin Schneider, Quy Duc Do, Zhuofeng Li, Yiming Jia, Yuxuan Zhang, Guo Cheng, Haozhe Wang, Wangchunshu Zhou, Qunshu Lin, Yuanxing Zhang, Ge Zhang, Wenhao Huang, Wenhu ChenSubmission to ICLR 2026 · Withdrawn

TL;DR

VideoScore2 introduces a "thinking-before-scoring" paradigm for evaluating AI-generated video. Rather than producing opaque scalar ratings, it generates multi-dimensional Chain-of-Thought (CoT) rationales across visual quality, text alignment, and physical consistency before outputting calibrated scores. Trained via cold-start Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO), it sets new state-of-the-art results on both in-domain and out-of-domain benchmarks while serving as an effective reward model for Best-of-$N$ video generation.

What Problem Does This Paper Address?

Evaluating text-to-video (T2V) models is notoriously difficult. Unlike images or text, videos require assessing fine-grained visual fidelity (motion blur, flicker, resolution), semantic prompt adherence (sequential actions, scene attributes), and physical plausibility (gravity, collision, anatomical coherence).

Existing video reward models and automated evaluators present two major issues:

  1. Lack of interpretability: Most models collapse video assessment into a single black-box score without explanatory rationale, offering zero transparency or diagnostic feedback.
  2. Poor generalization: Standard models trained purely with supervised fine-tuning (SFT) often overfit to in-domain distributions and degrade sharply on out-of-domain (OOD) benchmarks.

Methods

VideoScore2 builds on top of Qwen2.5-VL-7B-Instruct using a two-stage post-training pipeline:

[Prompt + Frames] 
       │
       ▼
[Cold-Start SFT] ──► Teaches formatting & baseline reasoning
       │
       ▼
[GRPO Reinforcement Learning] ──► Aligns scores & strengthens reasoning robustness
       │
       ▼
[Output: <think> rationale </think> + Multi-dimensional Scores]
  • Cold-Start SFT: Fine-tunes the base Vision-Language Model (VLM) on structured (instruction, video, rationale, score) pairs to instill strict schema adherence and baseline analytical reasoning.
  • Reinforcement Learning via GRPO: Using the Video-R1 framework, the model is trained with Group Relative Policy Optimization. The reward function combines:
  • Accuracy Reward ($R_{acc}$): Strictly penalizes errors. Exact matches receive 1.0; minor deviations ($\pm 1$ point on a 1–5 scale) receive partial credit (0.1–0.7); deviations $\ge 2$ points receive 0.
  • Format Reward ($R_{fmt}$): Ensures reasoning is enclosed within <think> tags (primarily active when bootstrapping without SFT).
  • Soft-Float Inference: During inference at 2 fps, instead of naive discrete argmax decoding, VideoScore2 computes continuous float scores weighted by token probabilities ($\tilde{y} = \arg\max_s p(s) \times \frac{p(s)}{\sum_{j=1}^5 p(j)}$), capturing finer-grained quality differences.

Datasets

VideoFeedback2 (Training & Curation)

The authors curated a dataset consisting of 2,933 unique prompts, 27,168 videos, and 81,504 dimension scores with rationales:

  • Prompt Sources: Real user queries from VidProM, captions from Koala-36M, and manually designed challenging prompts focusing on multi-action sequences, OCR text rendering, and camera motion.
  • Generators: Sampled across 22 T2V models categorized into four quality tiers—from early systems (ModelScope, Text2Video-Zero) to modern generative models (Kling-1.6, Sora, Wanx-2.1).
  • Rationale Elicitation: 15 trained human annotators provided 1–5 scores and diagnostic notes. These notes were expanded into detailed rationales using Claude-4-Sonnet (with thinking enabled) and reconciled with human ratings via GPT-5-mini.

Evaluation Benchmarks

  • VideoScore-Bench-v2 (In-Domain): 500 held-out samples evaluated across visual quality, text alignment, and physical consistency.
  • Out-of-Domain Benchmarks:
  • Pairwise Preference: VideoGenReward-Bench (4,691 videos, 25k pairs) and T2VQA-DB (1,822 pairs).
  • Point-Score: MJ-Bench-Video (2,170 videos) and VideoPhy2-test (3,396 videos).

Key Insights and Ideas

  • Thinking improves scoring robustness: SFT with CoT rationales dramatically outperforms SFT without rationales on point-score benchmarks (39.81 vs. 32.17 in-domain accuracy). Explicit reasoning forces the model to verify visual evidence before committing to a score.
  • SFT and RL are complementary: Initializing RL directly from the base VLM leads to format drift and lower overall performance (36.70 vs. 44.53). Cold-start SFT establishes the reasoning template, allowing subsequent RL to focus purely on score calibration.
  • Soft float conversion bridges preference and regression: Generating discrete score tokens often leads to tie-breaking issues in pairwise comparisons. Soft probability normalization smooths predictions into continuous values, yielding much stronger preference ranking without degrading point-score accuracy.

Key Results

  • In-Domain (VideoScore-Bench-v2): VideoScore2 achieved 44.35% exact accuracy (+5.94 over the previous best, VideoScore1.1), 90.78% relaxed accuracy ($\pm 1$), and a PLCC of 60.37 (+8.32 over baselines).
  • Out-of-Domain Generalization: VideoScore2 attained an average score of 50.37 across 4 OOD benchmarks, outperforming all image- and video-based reward models (e.g., Q-Insight at 46.05, VideoReward at 44.73).
  • Controllable Generation via Best-of-$N$ (BoN): Using VideoScore2 as a reward model with $N=5$ on 500 prompts across 6 base generators (e.g., ModelScope, AnimateDiff, LaVie) consistently boosted quality scores on VBench compared to random sampling.

Limitations and Future Work

  • Physical Commonsense Remains Hard: Despite outperforming baselines, performance on physical consistency lagged behind visual quality and text alignment across all evaluations, indicating that current VLMs still lack robust internal "world models."
  • Inference Overhead: Generating detailed chain-of-thought tokens before producing numerical scores introduces latency, making it computationally heavier for ultra-fast or real-time reward-guided decoding.
  • Future Directions: Integrating specialized physics-grounded world models into the video evaluator backbone to improve physical commonsense understanding.

Summary updated

Abstract

Recent advances in text-to-video generation have produced increasingly realistic and diverse content, yet evaluating such videos remains a fundamental challenge due to their multi-faceted nature encompassing visual quality, semantic alignment, and physical consistency. Existing evaluators and reward models are limited to single opaque scores, lack interpretability, or provide only coarse analysis, making them insufficient for capturing the comprehensive nature of video quality assessment. We present **VideoScore2**, a *multi-dimensional*, *interpretable*, and *human-aligned* framework that explicitly evaluates visual quality, text-to-video alignment, and physical/common-sense consistency while producing detailed chain-of-thought rationales. Our model is trained on a large-scale dataset **VideoFeedback2** containing 27,168 human-annotated videos with both scores and reasoning traces across three dimensions, using a two-stage pipeline of supervised fine-tuning followed by reinforcement learning with Group Relative Policy Optimization (GRPO) to enhance analytical robustness. Extensive experiments demonstrate that VideoScore2 achieves superior performance with 44.35 (+5.94) accuracy on our in-domain benchmark VideoScore-Bench-v2 and 50.37 (+4.32) average performance across four out-of-domain benchmarks (VideoGenReward-Bench, VideoPhy2, etc), while providing interpretable assessments that bridge the gap between evaluation and controllable generation through effective reward modeling for Best-of-N sampling.
BibTeX
@misc{he2025videoscore,
  author = {Xuan He and Dongfu Jiang and Ping Nie and Minghao Liu and Zhengxuan Jiang and Mingyi Su and Wentao Ma and Junru Lin and Chun Ye and Yi Lu and Keming Wu and Benjamin Schneider and Quy Duc Do and Zhuofeng Li and Yiming Jia and Yuxuan Zhang and Guo Cheng and Haozhe Wang and Wangchunshu Zhou and Qunshu Lin and Yuanxing Zhang and Ge Zhang and Wenhao Huang and Wenhu Chen},
  title = {{VideoScore2: Think before You Score in Generative Video Evaluation}},
  year = {2025},
  eprint = {2509.22799},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.22799},
}
Figure