Frame by Familiar Frame: Understanding Replication in Video Diffusion Models

Aimon Rahman, Malsha V. Perera, Vishal M. Patel

TL;DR

Video diffusion models (VDMs) are highly prone to replicating exact content and motion from their training data rather than generating novel videos. This paper systematically exposes this memorization behavior, reveals how standard evaluation metrics like Fréchet Video Distance (FVD) inadvertently reward plagiarism, and proposes a new evaluation framework alongside training protocols to encourage true generation over replication.

What Problem Does This Paper Address?

While the generative AI community has widely acknowledged data replication in image diffusion models, the phenomenon remains understudied in the video domain. Video generation is inherently more complex, operating in higher dimensions with severe data scarcity.

The authors address a critical vulnerability: because video datasets are relatively small, video diffusion models frequently memorize and output near-identical copies of training samples. Compounding this issue, standard evaluation metrics like FVD measure how closely the generated distribution matches the training distribution. Consequently, a model that simply copies and pastes its training data receives an excellent (low) FVD score, masking its lack of actual creative capacity.

Figure 1. Diffusion-based video synthesis models can sometimes replicate training data by assembling memorized foreground and back- ground elements. We demonstrate this trend across multiple diffusion models trained on diverse datasets. Such occurrences prompt inquiries regarding data memorization and the ownership of videos produced by diffusion methods. Bottom row: Videos sourced from the RaMViD [18], VIDM [28], and LVDM [14] project websites. Top row: The most similar counterparts from the training dataset.

The paper distinguishes between two types of replication based on the generation context:

  1. Content + Motion Replication (Unconditional): The model generates a 1:1 copy of both the visual elements and the dynamics from pure noise.
  2. Motion Replication (Conditional): Given a starting frame, the model fails to understand the underlying physics or action, instead regurgitating a specific motion trajectory it memorized during training.
Figure 2. Definition of replication in video generation domain. Content and motion replication refers to the direct duplication of content and motion from the training dataset, essentially producing a 1:1 copy. On the other hand, motion replication assesses a video generation model’s inherent ability to create motion from an initial frame. This initial frame supplies the content context, but the true measure of a video generation network’s capability lies in its un- derstanding of the comprehensive content within that first frame. The critical question is whether the network genuinely compre- hends and generates subsequent motion, or if it merely replicates sequences it has learned from the training data.

Methods

To detect and quantify replication, the authors introduced two novel evaluation strategies:

  • VSSCD Score (Video Self-Supervised Copy Detector): The authors adapted SSCD (a state-of-the-art image copy descriptor) for video. They extract frame-level features from both real and generated videos, concatenate them, and use cosine similarity to identify the highest training-set match for any generated video.
  • The Integrated FVD-VSSCD Curve: To fix the loophole in FVD evaluation, the authors proposed filtering out generated videos that exceed a certain replication threshold (VSSCD > 0.6) and recalculating the FVD score on the remaining, truly novel videos.
  • Robustness via Frame Augmentation: To detect motion replication in conditional models (video prediction), the authors applied minor semantic-preserving transformations (flipping, cropping, rotating, occluding) to the conditioning first frame. If the model truly understands motion, its performance should remain stable; if it has merely memorized a trajectory tied to that exact pixel layout, performance degrades sharply.

Datasets

The models were trained and evaluated using three standard video datasets:

  • UCF-101: A dataset of 101 human action classes, frequently used to train smaller-scale or baseline video models.
  • Kinetics-400: A large-scale high-quality dataset of diverse human actions.
  • WebVid-10M: A massive dataset of 10 million text-video pairs used for large-scale text-to-video (T2V) models.

Key Insights and Ideas

  • Video Models are Data-Hungry Plagiarists: An image diffusion model can generate unique, non-replicated samples when trained on as few as 1,000 images. However, a video diffusion model trained on 1,000 videos still heavily replicates its training data. Video requires significantly more data to learn generalized concepts.
  • FVD Rewards Copying: Quantitative experiments proved that higher replication rates directly correlate with artificially "better" FVD scores.
  • The T2I Backbone Shield: Models built on top of pre-trained Text-to-Image (T2I) backbones (like Stable Diffusion) are far less prone to content replication because the spatial representations are already well-generalized.
  • Fine-Tune Temporal Layers Only: When data is scarce, freezing the spatial weights of a T2I backbone and fine-tuning only the temporal layers allows the model to learn motion dynamics without collapsing into content memorization.
Motion replication drops significantly when fine-tuning only the temporal layers of a T2I backbone

Key Results

  • Unconditional Models: Popular models like VIDM, RaMViD, and LVDM trained on UCF-101 exhibited high average top VSSCD scores (0.63 to 0.74), confirming substantial replication of training videos.
  • Conditional Overfitting: When evaluating video prediction models, introducing minor alterations like flipping or cropping to the first frame caused the FVD score to degrade significantly (e.g., from 667.64 on original frames to 942.02 on flipped frames for UCF-101). This proves the models memorized trajectories rather than understanding the action.
  • Mitigation Success: Training a model from scratch on UCF-101 yielded a high replication VSSCD score of 0.74. However, using a pre-trained T2I backbone and fine-tuning only the temporal layers dropped the replication score drastically to 0.30 (for temporal-only fine-tuning), demonstrating a successful strategy for generating unique content.
Figure 7. A representation of the FVD-VSSCD Curve. The y-axis represents the FVD score, which measures the quality of video frames. The x-axis indicates the percentage of non-replicated sam- ples, which reflects the proportion of the generated videos that have been filtered out based on a certain threshold of the VSSCD score (0.6).

Limitations and Future Work

A primary limitation acknowledged by the authors is the limited public availability of proprietary, state-of-the-art video diffusion models, forcing them to rely on samples provided on official project websites for certain evaluations.

For future work, the authors plan to explore the phenomenon of motion replication across varying content—specifically, investigating whether models blindly apply memorized biometric motion patterns (like a specific human gait) onto entirely new characters or environments, which poses unique privacy and security risks.

Summary updated

Abstract

Building on the momentum of image generation diffusion models, there is an increasing interest in video-based diffusion models. However, video generation poses greater challenges due to its higher-dimensional nature, the scarcity of training data, and the complex spatiotemporal relationships involved. Image generation models, due to their extensive data requirements, have already strained computational resources to their limits. There have been instances of these models reproducing elements from the training samples, leading to concerns and even legal disputes over sample replication. Video diffusion models, which operate with even more constrained datasets and are tasked with generating both spatial and temporal content, may be more prone to replicating samples from their training sets. Compounding the issue, these models are often evaluated using metrics that inadvertently reward replication. In our paper, we present a systematic investigation into the phenomenon of sample replication in video diffusion models. We scrutinize various recent diffusion models for video synthesis, assessing their tendency to replicate spatial and temporal content in both unconditional and conditional generation scenarios. Our study identifies strategies that are less likely to lead to replication. Furthermore, we propose new evaluation strategies that take replication into account, offering a more accurate measure of a model's ability to generate the original content.
BibTeX
@inproceedings{rahman2025frame,
  author = {Aimon Rahman and Malsha V. Perera and Vishal M. Patel},
  title = {{Frame by Familiar Frame: Understanding Replication in Video Diffusion Models}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure