Abstract
Recent advancements in video generative models have highlighted a significant gap in evaluation metrics: the ability to assess temporal and motion consistency effectively. While existing metrics like FVD and FID-VID focus on frame quality or high-level semantic features, they often fail to capture the physical plausibility of motion. We introduce the Fréchet Video Motion Distance (FVMD), a metric that explicitly evaluates motion consistency by analyzing keypoint trajectories, velocities, and accelerations. Our results demonstrate that FVMD aligns more closely with human perception than traditional metrics and is highly sensitive to temporal artifacts.
Introduction
The rapid evolution of video generation models, such as Sora, has created a need for robust evaluation frameworks. Generating high-quality individual frames is no longer the sole challenge; ensuring that motion across those frames is physically consistent and free from temporal flickering is paramount. Current standard metrics, such as Fréchet Video Distance (FVD), rely on pre-trained action recognition models. These "black-box" features often conflate visual appearance with motion, making it difficult to isolate motion quality. Furthermore, some benchmarks inadvertently reward "smoothness," which can bias metrics toward static or low-motion videos. To address this, we propose FVMD, a physics-informed metric designed to measure whether the motion in a generated video follows consistent physical patterns.
Methodology
The FVMD pipeline transitions from generic deep-learning features to explicit physical properties of motion.
1. Keypoint Tracking
The system utilizes PIPs++, a state-of-the-art tracker, to follow a grid of points across video frames. PIPs++ is specifically chosen for its robustness against occlusions and its ability to track objects as they move in and out of the frame.
2. Motion Field Calculation
From the tracked trajectories, we derive first-order (velocity) and second-order (acceleration) differences. This approach is grounded in the principle that natural physical motion should not exhibit erratic spikes in acceleration—a common artifact in AI-generated videos known as "temporal flickering."
3. Feature Extraction and Comparison
These velocity and acceleration fields are converted into histograms. We utilize a dense 1D histogram (inspired by Histograms of Oriented Gradients, or HOG) where magnitudes are aggregated into angular bins. Finally, we calculate the Fréchet distance between the distribution of these motion features in generated videos and a reference set of real-world videos. This distance serves as a quantitative measure of the "motion signature" discrepancy between AI and reality.
Experimental Results
We validated FVMD across several datasets, including high-intensity human motion (TikTok) and robotics (BAIR Video Pushing).
Human Alignment
A large-scale human study was conducted to compare FVMD against existing metrics. FVMD achieved a Pearson correlation of 0.8469 with human judgments of video quality, significantly outperforming FVD (0.6708) and FID-VID (0.3402). This indicates that FVMD captures the nuances of motion that humans find important, particularly in complex scenarios like dancing.
Sensitivity and Robustness
Sensitivity analysis involved injecting various types of temporal noise (e.g., local frame swaps) into real videos. FVMD proved highly sensitive to these artifacts, whereas traditional metrics often yielded inconsistent results. Additionally, sanity checks confirmed that FVMD correctly converges to zero when comparing different subsets of the same real dataset, ensuring the metric is not biased by sample size.
Discussion and Conclusion
FVMD represents a shift toward more interpretable and physics-informed video evaluation. By focusing on the consistency of motion paths rather than just the absence of change, it avoids the "smoothness bias" of previous benchmarks. Furthermore, the motion features developed for FVMD can be integrated into unary Video Quality Assessment (VQA) models to improve their accuracy.
While FVMD is a significant advancement, it remains dependent on the accuracy of the underlying keypoint tracker and does not currently account for semantic correctness (e.g., anatomical plausibility). Future work will explore more comprehensive motion representations to address complex object interactions and physically implausible events.
Summary updated