EVA-RSI: Recursive Self-Improvement of Video Evaluators for Judging Temporal Quality

Yu Yuan, Ruizhou Ding, Shaoheng Fang, Stanley H. Chan, Zhuo ChenarXiv 2026 · arXiv

TL;DR

EVA-RSI is a multi-agent framework that allows video evaluation models to self-improve their judgment of temporal quality (like motion and physics) without needing massive human-labeled datasets. By training on a "proxy exam" of synthetically altered videos and using a Nash bargaining rule to balance competing quality dimensions, the system produces evaluators that significantly outperform existing models and serve as superior rewards for training video generators.

What Problem Does This Paper Address?

As video generation models like Sora and Kling advance, the bottleneck is shifting from generation to evaluation. Current Vision-Language Models (VLMs) are surprisingly bad at judging temporal consistency. For instance, a state-of-the-art model might give the same "physics plausibility" score to a normal video and a version where a person suddenly freezes mid-motion.

This "blind spot" is dangerous when these models are used as reward functions for Reinforcement Learning (RL). If the evaluator can't distinguish between a moving boat and a static one, the generator learns that it can maximize its reward by simply producing high-quality still images, effectively "killing" the motion in generated videos.

Recursive Self-Improvement (RSI) has worked wonders in domains like coding and math because those fields have explicit verifiers (e.g., a compiler or a unit test). Video evaluation has no such ground truth, and it involves balancing conflicting goals: improving a model's sensitivity to "smoothness" might accidentally make it worse at judging "dynamics."

Figure 1. A reward that misjudges motion trains generators that do not move. Top: physics ratings for a clip and for a copy in which the person freezes. Bottom: Wan2.1 after 50 RL updates with VideoReward (visual and motion quality, no penalties) and with EVA-RSI as the reward.

Methods

The authors introduce EVA-RSI, a framework that treats model improvement as a multi-agent game.

  1. The Multi-Agent Team: The system employs specialized LLM agents in four roles:
  2. Proposers: One agent per quality dimension (e.g., physics, smoothness, dynamics) analyzes the current model's errors and suggests training recipes.
  3. Leader: Coordinates these proposals to maximize overall progress.
  4. Engineers: Implement the suggested changes and train new candidate models.
  5. Reviewer: A final check to ensure no "cheating" (like training on test data) occurred.
  6. The Proxy Exam: To replace a missing verifier, the authors build a fixed exam using construction-defined targets. They take "clean" videos and inject defects: reversing them, dropping frames, or freezing objects. The "correct" answer is always the original, clean clip. This provides a reliable, automated signal for improvement.
  7. Nash Bargaining Acceptance: To handle the trade-offs between dimensions, they use a rule based on Nash bargaining theory. A new candidate model only becomes the "champion" if it improves the average score across main dimensions without letting any single dimension drop below a certain "floor." This ensures balanced progress rather than over-optimizing for one trait at the expense of others.
Figure 2. The EVA-RSI system. ①A team of LLM agents proposes, implements and trains candidate evaluators from the current champion. ②A frozen verifier scores each candidate on an exam with fixed targets, and a Nash bargaining rule accepts or rejects it. An accepted candidate becomes the champion of the next round; every outcome enters a shared memory that the proposers read. Humans design the verifier and redesign the exam between stages.

Datasets

  • Training/Development: A constructed exam with ~5.8k questions covering nine dimensions (physics, consistency, smoothness, quality, dynamics, etc.). It uses injected defects and optical flow pairs.
  • Hidden Test Bank: 3,450 questions featuring defect types never seen during training (e.g., teleporting objects, texture swaps) to test generalization.
  • Public Benchmarks: The models were evaluated on 12 external benchmarks, including VideoPhy-2, VideoScore-Bench, and VBench, to compare against 21 other state-of-the-art models.

Key Insights and Ideas

  • Synthetic Supervision Works: You don't need millions of human labels to teach a model about motion. Simple edits like frame-dropping provide enough signal for a model to learn the underlying concept of "smoothness."
  • Generalization to the Unseen: Remarkably, models trained to detect simple "frozen" frames became better at detecting complex, unseen defects like "teleporting objects."
  • The Human in the Loop: While the agents do the heavy lifting, humans act as "curriculum designers," refining the exam between stages when the model saturates on simpler tasks.
  • Nash Equilibrium for AI Training: Using a formal bargaining rule is a robust way to manage the "Pareto front" of model capabilities, preventing the common "catastrophic forgetting" seen in multi-task learning.

Key Results

Starting from Qwen3-VL-8B, EVA-RSI achieved dramatic improvements:

  • Smoothness scores on the hidden test bank jumped from 0.461 to 0.888.
  • On public benchmarks, EVA-RSI achieved the highest observed scores for dynamics among 21 tested models, outperforming even much larger proprietary models like Gemini 1.5 Pro and GPT-4o on specific temporal metrics.
  • Downstream Impact: When used as an RL reward for the Wan2.1 generator, EVA-RSI produced videos with significantly more motion. In blind human studies, users preferred EVA-RSI-trained videos over those trained with standard rewards by a wide margin.
Figure 3. RSI improves the evaluator over rounds. (a) Normalized score of each champion on the hidden test bank. (b) Scored candidates (grey) and champions (blue) on the development banks, which change at the start of stage 2. (c) Mean rank score on 12 public benchmarks, against all evaluators except VideoReward (Appendix D). The development banks determine the final champion.

Limitations and Future Work

The primary limitation is the proxy exam itself. While effective, synthetic edits are simpler than the messy, interacting failures found in real-world generated videos. The authors note that construction-defined targets describe individual edits, but they don't yet capture complex failures where defects interact with scene intent.

Future work aims to build richer exams directly from generator failures and to further validate these automated targets against human perceptual judgments across a wider variety of video architectures.

Summary updated

Abstract

Video evaluators play an important role in video generation, yet they miss motion defects, limiting the quality of generated videos. Recursive self-improvement (RSI) has made strong progress in mathematics and coding, where explicit verifiers provide reliable feedback on a single goal. Video evaluation poses two difficulties for RSI: it has no verifier, and it must judge many dimensions at once, such as physics, smoothness and dynamics, where a gain on one can cost another. We introduce EVA-RSI, a multi-agent framework for exploring RSI under these two constraints. We build a fixed proxy exam with construction-defined targets by applying controlled changes to videos, such as temporal reversal and frame dropping. These edits supply supervision without human annotation. A small set of human-labeled real videos anchors the exam to human judgment. Each dimension has an LLM agent that proposes improvements. A leader agent coordinates the proposals, and a Nash bargaining rule selects updates that balance progress across dimensions. Humans design the exam, review candidate changes and revise the curriculum between stages. Starting from Qwen3-VL-8B, EVA-RSI improves motion evaluation, with the largest gain on smoothness, and the gains carry over to defect types never seen during the run. On the dynamics subsets of public benchmarks, our evaluator achieves the highest observed scores among 21 tested models. As an RL reward, it increases generated motion and earns human preference. A data-selection pipeline incorporating it also increases motion.
BibTeX
@misc{yuan2026eva,
  author = {Yu Yuan and Ruizhou Ding and Shaoheng Fang and Stanley H. Chan and Zhuo Chen},
  title = {{EVA-RSI: Recursive Self-Improvement of Video Evaluators for Judging Temporal Quality}},
  year = {2026},
  eprint = {2611.23890},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23890},
}

Loading related content...

Figure