TL;DR
EVA-RSI is a multi-agent framework that allows video evaluation models to self-improve their judgment of temporal quality (like motion and physics) without needing massive human-labeled datasets. By training on a "proxy exam" of synthetically altered videos and using a Nash bargaining rule to balance competing quality dimensions, the system produces evaluators that significantly outperform existing models and serve as superior rewards for training video generators.
What Problem Does This Paper Address?
As video generation models like Sora and Kling advance, the bottleneck is shifting from generation to evaluation. Current Vision-Language Models (VLMs) are surprisingly bad at judging temporal consistency. For instance, a state-of-the-art model might give the same "physics plausibility" score to a normal video and a version where a person suddenly freezes mid-motion.
This "blind spot" is dangerous when these models are used as reward functions for Reinforcement Learning (RL). If the evaluator can't distinguish between a moving boat and a static one, the generator learns that it can maximize its reward by simply producing high-quality still images, effectively "killing" the motion in generated videos.
Recursive Self-Improvement (RSI) has worked wonders in domains like coding and math because those fields have explicit verifiers (e.g., a compiler or a unit test). Video evaluation has no such ground truth, and it involves balancing conflicting goals: improving a model's sensitivity to "smoothness" might accidentally make it worse at judging "dynamics."
Methods
The authors introduce EVA-RSI, a framework that treats model improvement as a multi-agent game.
- The Multi-Agent Team: The system employs specialized LLM agents in four roles:
- Proposers: One agent per quality dimension (e.g., physics, smoothness, dynamics) analyzes the current model's errors and suggests training recipes.
- Leader: Coordinates these proposals to maximize overall progress.
- Engineers: Implement the suggested changes and train new candidate models.
- Reviewer: A final check to ensure no "cheating" (like training on test data) occurred.
- The Proxy Exam: To replace a missing verifier, the authors build a fixed exam using construction-defined targets. They take "clean" videos and inject defects: reversing them, dropping frames, or freezing objects. The "correct" answer is always the original, clean clip. This provides a reliable, automated signal for improvement.
- Nash Bargaining Acceptance: To handle the trade-offs between dimensions, they use a rule based on Nash bargaining theory. A new candidate model only becomes the "champion" if it improves the average score across main dimensions without letting any single dimension drop below a certain "floor." This ensures balanced progress rather than over-optimizing for one trait at the expense of others.
Datasets
- Training/Development: A constructed exam with ~5.8k questions covering nine dimensions (physics, consistency, smoothness, quality, dynamics, etc.). It uses injected defects and optical flow pairs.
- Hidden Test Bank: 3,450 questions featuring defect types never seen during training (e.g., teleporting objects, texture swaps) to test generalization.
- Public Benchmarks: The models were evaluated on 12 external benchmarks, including VideoPhy-2, VideoScore-Bench, and VBench, to compare against 21 other state-of-the-art models.
Key Insights and Ideas
- Synthetic Supervision Works: You don't need millions of human labels to teach a model about motion. Simple edits like frame-dropping provide enough signal for a model to learn the underlying concept of "smoothness."
- Generalization to the Unseen: Remarkably, models trained to detect simple "frozen" frames became better at detecting complex, unseen defects like "teleporting objects."
- The Human in the Loop: While the agents do the heavy lifting, humans act as "curriculum designers," refining the exam between stages when the model saturates on simpler tasks.
- Nash Equilibrium for AI Training: Using a formal bargaining rule is a robust way to manage the "Pareto front" of model capabilities, preventing the common "catastrophic forgetting" seen in multi-task learning.
Key Results
Starting from Qwen3-VL-8B, EVA-RSI achieved dramatic improvements:
- Smoothness scores on the hidden test bank jumped from 0.461 to 0.888.
- On public benchmarks, EVA-RSI achieved the highest observed scores for dynamics among 21 tested models, outperforming even much larger proprietary models like Gemini 1.5 Pro and GPT-4o on specific temporal metrics.
- Downstream Impact: When used as an RL reward for the Wan2.1 generator, EVA-RSI produced videos with significantly more motion. In blind human studies, users preferred EVA-RSI-trained videos over those trained with standard rewards by a wide margin.
Limitations and Future Work
The primary limitation is the proxy exam itself. While effective, synthetic edits are simpler than the messy, interacting failures found in real-world generated videos. The authors note that construction-defined targets describe individual edits, but they don't yet capture complex failures where defects interact with scene intent.
Future work aims to build richer exams directly from generator failures and to further validate these automated targets against human perceptual judgments across a wider variety of video architectures.
Summary updated