TL;DR
GraftOOD introduces a novel way to prevent "motion collapse" in video generation—a phenomenon where models produce high-quality but static videos when optimized for aesthetic rewards. By "grafting" external, high-motion video exemplars directly into the reinforcement learning loop via a shared velocity-regression objective, the authors successfully balance visual beauty with realistic, dynamic movement.
What Problem Does This Paper Address?
Reinforcement Learning from Human Feedback (RLHF) has revolutionized generative models, but it faces a unique hurdle in video: motion collapse.
When we optimize a video model using proxy rewards (like aesthetic or quality scores), the model often learns to "cheat." It discovers that the easiest way to get a high score is to generate a beautiful, crystal-clear image that barely moves. As the reward score goes up, the actual "Dynamic Degree" (the amount of motion) often plummets. This is a form of reward exploitation where the model prioritizes frame-level prettiness over temporal realism.
Methods
The authors propose GraftOOD (Grafting Out-of-Distribution exemplars). Instead of relying solely on the model's own generated "rollouts" for training, GraftOOD replaces a portion of these with high-quality external video exemplars.
Key technical highlights include:
- Shared Velocity-Regression: Unlike standard methods that might use a separate supervised loss, GraftOOD uses a single objective. It treats external exemplars as having "theoretically maximal reward" and integrates them into the same mathematical framework used for online reward optimization.
- Advantage-Weighted Rectification: It builds on FlowAWR, using "advantages" (how much better a sample is than the average) to guide updates.
- Pointwise Optimal Field: The paper mathematically characterizes the training target as a weighted combination of the model's current path and the path toward the high-quality exemplars.
Datasets
The authors introduced GraftSet-Video, a specialized dataset designed to provide the "external supervision" needed to fight motion collapse.
- Composition: 9,588 video exemplars paired with 2,397 prompts.
- Sources: A mix of public reward-model resources and high-end outputs from state-of-the-art generators like Seedance and Kling.
- Evaluation: The models were tested on the VidProM (300 prompts) and VBench suites to measure aesthetics, consistency, and dynamics.
Key Insights and Ideas
- Rewards Aren't Enough: Even adding a specific "motion reward" (like one based on optical flow) mitigates but does not stop motion collapse. Sequence-level supervision (showing the model actual moving videos) is a necessary complement to scalar rewards.
- The "Grafting" Balance: There is a fundamental trade-off between maximizing a proxy reward and maintaining dynamics. As shown in the paper's ablations, increasing the ratio of external exemplars boosts motion but can slightly lower aesthetic scores.
- OOD as a Reference: By using "Out-of-Distribution" data (videos the model didn't make itself), the training process is anchored to realistic motion patterns that the model might never have discovered through self-exploration alone.
Key Results
Testing on the Wan2.1-1.3B model, GraftOOD showed dramatic improvements:
- Dynamic Degree (DD): Jumped from a stagnant 4.86 (using standard reward-only optimization) to a much more lifelike 44.41.
- Metric Leadership: GraftOOD led in 9 out of 10 metrics compared to other exemplar-augmented baselines (FlashOOD and NFTOOD).
- Qualitative Improvement: Visual comparisons show that while reward-only models produce "slideshows," GraftOOD produces videos with coherent, significant movement.
Limitations and Future Work
- Exemplar Sources: Currently, all exemplars in GraftSet-Video are AI-generated. The authors note that using real-world video recordings could further improve physical consistency, though this requires high-quality video-text pairing.
- Motion Quality vs. Quantity: A high "Dynamic Degree" means there is a lot of movement, but it doesn't always mean the movement is correct. In some cases (like the FlashOOD baseline), high dynamics can actually manifest as "deformation" or warping rather than natural motion.
- Scaling: The study was limited to the 1.3B parameter scale; future work is needed to see how these "grafting" techniques behave with much larger models or in multi-modal (audio-video) settings.
Summary updated