TL;DR
Video diffusion models (VDMs) are highly prone to replicating exact content and motion from their training data rather than generating novel videos. This paper systematically exposes this memorization behavior, reveals how standard evaluation metrics like Fréchet Video Distance (FVD) inadvertently reward plagiarism, and proposes a new evaluation framework alongside training protocols to encourage true generation over replication.
What Problem Does This Paper Address?
While the generative AI community has widely acknowledged data replication in image diffusion models, the phenomenon remains understudied in the video domain. Video generation is inherently more complex, operating in higher dimensions with severe data scarcity.
The authors address a critical vulnerability: because video datasets are relatively small, video diffusion models frequently memorize and output near-identical copies of training samples. Compounding this issue, standard evaluation metrics like FVD measure how closely the generated distribution matches the training distribution. Consequently, a model that simply copies and pastes its training data receives an excellent (low) FVD score, masking its lack of actual creative capacity.
The paper distinguishes between two types of replication based on the generation context:
- Content + Motion Replication (Unconditional): The model generates a 1:1 copy of both the visual elements and the dynamics from pure noise.
- Motion Replication (Conditional): Given a starting frame, the model fails to understand the underlying physics or action, instead regurgitating a specific motion trajectory it memorized during training.
Methods
To detect and quantify replication, the authors introduced two novel evaluation strategies:
- VSSCD Score (Video Self-Supervised Copy Detector): The authors adapted SSCD (a state-of-the-art image copy descriptor) for video. They extract frame-level features from both real and generated videos, concatenate them, and use cosine similarity to identify the highest training-set match for any generated video.
- The Integrated FVD-VSSCD Curve: To fix the loophole in FVD evaluation, the authors proposed filtering out generated videos that exceed a certain replication threshold (VSSCD > 0.6) and recalculating the FVD score on the remaining, truly novel videos.
- Robustness via Frame Augmentation: To detect motion replication in conditional models (video prediction), the authors applied minor semantic-preserving transformations (flipping, cropping, rotating, occluding) to the conditioning first frame. If the model truly understands motion, its performance should remain stable; if it has merely memorized a trajectory tied to that exact pixel layout, performance degrades sharply.
Datasets
The models were trained and evaluated using three standard video datasets:
- UCF-101: A dataset of 101 human action classes, frequently used to train smaller-scale or baseline video models.
- Kinetics-400: A large-scale high-quality dataset of diverse human actions.
- WebVid-10M: A massive dataset of 10 million text-video pairs used for large-scale text-to-video (T2V) models.
Key Insights and Ideas
- Video Models are Data-Hungry Plagiarists: An image diffusion model can generate unique, non-replicated samples when trained on as few as 1,000 images. However, a video diffusion model trained on 1,000 videos still heavily replicates its training data. Video requires significantly more data to learn generalized concepts.
- FVD Rewards Copying: Quantitative experiments proved that higher replication rates directly correlate with artificially "better" FVD scores.
- The T2I Backbone Shield: Models built on top of pre-trained Text-to-Image (T2I) backbones (like Stable Diffusion) are far less prone to content replication because the spatial representations are already well-generalized.
- Fine-Tune Temporal Layers Only: When data is scarce, freezing the spatial weights of a T2I backbone and fine-tuning only the temporal layers allows the model to learn motion dynamics without collapsing into content memorization.
Key Results
- Unconditional Models: Popular models like VIDM, RaMViD, and LVDM trained on UCF-101 exhibited high average top VSSCD scores (0.63 to 0.74), confirming substantial replication of training videos.
- Conditional Overfitting: When evaluating video prediction models, introducing minor alterations like flipping or cropping to the first frame caused the FVD score to degrade significantly (e.g., from 667.64 on original frames to 942.02 on flipped frames for UCF-101). This proves the models memorized trajectories rather than understanding the action.
- Mitigation Success: Training a model from scratch on UCF-101 yielded a high replication VSSCD score of 0.74. However, using a pre-trained T2I backbone and fine-tuning only the temporal layers dropped the replication score drastically to 0.30 (for temporal-only fine-tuning), demonstrating a successful strategy for generating unique content.
Limitations and Future Work
A primary limitation acknowledged by the authors is the limited public availability of proprietary, state-of-the-art video diffusion models, forcing them to rely on samples provided on official project websites for certain evaluations.
For future work, the authors plan to explore the phenomenon of motion replication across varying content—specifically, investigating whether models blindly apply memorized biometric motion patterns (like a specific human gait) onto entirely new characters or environments, which poses unique privacy and security risks.
Summary updated