Fréchet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos

Jiahe Liu, Youran Qu, Qi Yan, Xiaohui Zeng, Lele Wang, Renjie Liao

Abstract

Recent advancements in video generative models have highlighted a significant gap in evaluation metrics: the ability to assess temporal and motion consistency effectively. While existing metrics like FVD and FID-VID focus on frame quality or high-level semantic features, they often fail to capture the physical plausibility of motion. We introduce the Fréchet Video Motion Distance (FVMD), a metric that explicitly evaluates motion consistency by analyzing keypoint trajectories, velocities, and accelerations. Our results demonstrate that FVMD aligns more closely with human perception than traditional metrics and is highly sensitive to temporal artifacts.

Introduction

The rapid evolution of video generation models, such as Sora, has created a need for robust evaluation frameworks. Generating high-quality individual frames is no longer the sole challenge; ensuring that motion across those frames is physically consistent and free from temporal flickering is paramount. Current standard metrics, such as Fréchet Video Distance (FVD), rely on pre-trained action recognition models. These "black-box" features often conflate visual appearance with motion, making it difficult to isolate motion quality. Furthermore, some benchmarks inadvertently reward "smoothness," which can bias metrics toward static or low-motion videos. To address this, we propose FVMD, a physics-informed metric designed to measure whether the motion in a generated video follows consistent physical patterns.

Methodology

The FVMD pipeline transitions from generic deep-learning features to explicit physical properties of motion.

1. Keypoint Tracking

The system utilizes PIPs++, a state-of-the-art tracker, to follow a grid of points across video frames. PIPs++ is specifically chosen for its robustness against occlusions and its ability to track objects as they move in and out of the frame.

2. Motion Field Calculation

From the tracked trajectories, we derive first-order (velocity) and second-order (acceleration) differences. This approach is grounded in the principle that natural physical motion should not exhibit erratic spikes in acceleration—a common artifact in AI-generated videos known as "temporal flickering."

3. Feature Extraction and Comparison

These velocity and acceleration fields are converted into histograms. We utilize a dense 1D histogram (inspired by Histograms of Oriented Gradients, or HOG) where magnitudes are aggregated into angular bins. Finally, we calculate the Fréchet distance between the distribution of these motion features in generated videos and a reference set of real-world videos. This distance serves as a quantitative measure of the "motion signature" discrepancy between AI and reality.

The overall pipeline of FVMD, illustrating the transition from keypoint tracking to Fréchet distance calculation.

Experimental Results

We validated FVMD across several datasets, including high-intensity human motion (TikTok) and robotics (BAIR Video Pushing).

Human Alignment

A large-scale human study was conducted to compare FVMD against existing metrics. FVMD achieved a Pearson correlation of 0.8469 with human judgments of video quality, significantly outperforming FVD (0.6708) and FID-VID (0.3402). This indicates that FVMD captures the nuances of motion that humans find important, particularly in complex scenarios like dancing.

Comparison of different metrics against human ranking. FVMD correctly identifies the rank order of video quality where others fail.

Sensitivity and Robustness

Sensitivity analysis involved injecting various types of temporal noise (e.g., local frame swaps) into real videos. FVMD proved highly sensitive to these artifacts, whereas traditional metrics often yielded inconsistent results. Additionally, sanity checks confirmed that FVMD correctly converges to zero when comparing different subsets of the same real dataset, ensuring the metric is not biased by sample size.

Discussion and Conclusion

FVMD represents a shift toward more interpretable and physics-informed video evaluation. By focusing on the consistency of motion paths rather than just the absence of change, it avoids the "smoothness bias" of previous benchmarks. Furthermore, the motion features developed for FVMD can be integrated into unary Video Quality Assessment (VQA) models to improve their accuracy.

While FVMD is a significant advancement, it remains dependent on the accuracy of the underlying keypoint tracker and does not currently account for semantic correctness (e.g., anatomical plausibility). Future work will explore more comprehensive motion representations to address complex object interactions and physically implausible events.

Summary updated

Abstract

Significant advancements have been made in video generative models recently. Unlike image generation, video generation presents greater challenges, requiring not only generating high-quality frames but also ensuring temporal consistency across these frames. Despite the impressive progress, research on metrics for evaluating the quality of generated videos, especially concerning temporal and motion consistency, remains underexplored. To bridge this research gap, we propose Fr\'echet Video Motion Distance (FVMD) metric, which focuses on evaluating motion consistency in video generation. Specifically, we design explicit motion features based on key point tracking, and then measure the similarity between these features via the Fr\'echet distance. We conduct sensitivity analysis by injecting noise into real videos to verify the effectiveness of FVMD. Further, we carry out a large-scale human study, demonstrating that our metric effectively detects temporal noise and aligns better with human perceptions of generated video quality than existing metrics. Additionally, our motion features can consistently improve the performance of Video Quality Assessment (VQA) models, indicating that our approach is also applicable to unary video quality evaluation. Code is available at https://github.com/ljh0v0/FMD-frechet-motion-distance.
BibTeX
@inproceedings{liu2024frechet,
  author = {Jiahe Liu and Youran Qu and Qi Yan and Xiaohui Zeng and Lele Wang and Renjie Liao},
  title = {{Fréchet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos}},
  booktitle = {CVG @ ICML},
  year = {2024},
  url = {https://openreview.net/forum?id=tTZ2eAhK9D},
}
Figure