GraftOOD: Grafting Out-of-Distribution Exemplars into Advantage-Weighted Rectification for Video Reward Optimization

Zheming Fu, Ruizhe He, Wenbin Ning, Xiaoxiao Ma, Xuan Yang, Lianrui Mu, Haoyang Huang, Chang Liu, Lei Wang, Siming FuarXiv 2026 · arXiv

TL;DR

GraftOOD introduces a novel way to prevent "motion collapse" in video generation—a phenomenon where models produce high-quality but static videos when optimized for aesthetic rewards. By "grafting" external, high-motion video exemplars directly into the reinforcement learning loop via a shared velocity-regression objective, the authors successfully balance visual beauty with realistic, dynamic movement.

What Problem Does This Paper Address?

Reinforcement Learning from Human Feedback (RLHF) has revolutionized generative models, but it faces a unique hurdle in video: motion collapse.

When we optimize a video model using proxy rewards (like aesthetic or quality scores), the model often learns to "cheat." It discovers that the easiest way to get a high score is to generate a beautiful, crystal-clear image that barely moves. As the reward score goes up, the actual "Dynamic Degree" (the amount of motion) often plummets. This is a form of reward exploitation where the model prioritizes frame-level prettiness over temporal realism.

Motion collapse: As HPSv3 (aesthetic reward) increases, Dynamic Degree (motion) decreases.

Methods

The authors propose GraftOOD (Grafting Out-of-Distribution exemplars). Instead of relying solely on the model's own generated "rollouts" for training, GraftOOD replaces a portion of these with high-quality external video exemplars.

Key technical highlights include:

  • Shared Velocity-Regression: Unlike standard methods that might use a separate supervised loss, GraftOOD uses a single objective. It treats external exemplars as having "theoretically maximal reward" and integrates them into the same mathematical framework used for online reward optimization.
  • Advantage-Weighted Rectification: It builds on FlowAWR, using "advantages" (how much better a sample is than the average) to guide updates.
  • Pointwise Optimal Field: The paper mathematically characterizes the training target as a weighted combination of the model's current path and the path toward the high-quality exemplars.

Datasets

The authors introduced GraftSet-Video, a specialized dataset designed to provide the "external supervision" needed to fight motion collapse.

  • Composition: 9,588 video exemplars paired with 2,397 prompts.
  • Sources: A mix of public reward-model resources and high-end outputs from state-of-the-art generators like Seedance and Kling.
  • Evaluation: The models were tested on the VidProM (300 prompts) and VBench suites to measure aesthetics, consistency, and dynamics.

Key Insights and Ideas

  • Rewards Aren't Enough: Even adding a specific "motion reward" (like one based on optical flow) mitigates but does not stop motion collapse. Sequence-level supervision (showing the model actual moving videos) is a necessary complement to scalar rewards.
  • The "Grafting" Balance: There is a fundamental trade-off between maximizing a proxy reward and maintaining dynamics. As shown in the paper's ablations, increasing the ratio of external exemplars boosts motion but can slightly lower aesthetic scores.
  • OOD as a Reference: By using "Out-of-Distribution" data (videos the model didn't make itself), the training process is anchored to realistic motion patterns that the model might never have discovered through self-exploration alone.

Key Results

Testing on the Wan2.1-1.3B model, GraftOOD showed dramatic improvements:

  • Dynamic Degree (DD): Jumped from a stagnant 4.86 (using standard reward-only optimization) to a much more lifelike 44.41.
  • Metric Leadership: GraftOOD led in 9 out of 10 metrics compared to other exemplar-augmented baselines (FlashOOD and NFTOOD).
  • Qualitative Improvement: Visual comparisons show that while reward-only models produce "slideshows," GraftOOD produces videos with coherent, significant movement.
Figure 1: Reward-only optimization versus GraftOOD. Same-prompt sequences from reward- only optimization (left) and GraftOOD (right), with frames in temporal order.

Limitations and Future Work

  • Exemplar Sources: Currently, all exemplars in GraftSet-Video are AI-generated. The authors note that using real-world video recordings could further improve physical consistency, though this requires high-quality video-text pairing.
  • Motion Quality vs. Quantity: A high "Dynamic Degree" means there is a lot of movement, but it doesn't always mean the movement is correct. In some cases (like the FlashOOD baseline), high dynamics can actually manifest as "deformation" or warping rather than natural motion.
  • Scaling: The study was limited to the 1.3B parameter scale; future work is needed to see how these "grafting" techniques behave with much larger models or in multi-modal (audio-video) settings.

Summary updated

Abstract

Video reward optimization can increase proxy rewards while reducing video dynamics, a mismatch we identify as \emph{motion collapse}, including under a video-level motion-quality reward. We propose \textbf{GraftOOD}, which replaces part of the online rollouts in selected prompt groups with out-of-distribution video exemplars. A shared velocity-regression objective combines group-relative reward rectification with sequence-level exemplar supervision, requiring neither exemplar generation trajectories nor an additional supervised-loss coefficient. We characterize its pointwise optimal field as a source-posterior-weighted combination of the two source-specific target fields. We construct \textbf{GraftSet-Video}, containing 9,588 format-matched exemplars for 2,397 prompts, to supply external supervision. On Wan2.1-1.3B, GraftOOD raises Dynamic Degree from 4.86 to 44.41 relative to reward-only FlowAWR, while HPSv3 decreases from 8.10 to 7.32 and VisionReward remains close. At the main setting, GraftOOD leads nine of ten metrics among three exemplar-augmented methods. Ablations of exemplar frequency and within-group count reveal dynamics--reward trade-offs rather than uniform gains from increasing exemplar usage. These results support external exemplar supervision as a complement to scalar reward feedback, without equating higher measured dynamics with coherent, visually plausible motion in the generated videos. Code is available at \href{https://github.com/Fuzm7/FlowAWR}{GraftOOD}.
BibTeX
@misc{fu2026graftood,
  author = {Zheming Fu and Ruizhe He and Wenbin Ning and Xiaoxiao Ma and Xuan Yang and Lianrui Mu and Haoyang Huang and Chang Liu and Lei Wang and Siming Fu},
  title = {{GraftOOD: Grafting Out-of-Distribution Exemplars into Advantage-Weighted Rectification for Video Reward Optimization}},
  year = {2026},
  eprint = {2610.13470},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.13470},
}
Figure