Abstract
Video frame interpolation, the task of synthesizing new frames in between two or more given ones, is becoming an increasingly popular research target. However, the current evaluation of frame interpolation techniques is not ideal. Due to the plethora of test datasets available and inconsistent computation of error metrics, a coherent and fair comparison across papers is very challenging. Furthermore, new test sets have been proposed as part of method papers so they are unable to provide the in-depth evaluation of a dedicated benchmarking paper. Another severe downside is that these test sets violate the assumption of linearity when given two input frames, making it impossible to solve without an oracle. We hence strongly believe that the community would greatly benefit from a benchmarking paper, which is what we propose. Specifically, we present a benchmark which establishes consistent error metrics by utilizing a submission website that computes them, provides insights by analyzing the interpolation quality with respect to various per-pixel attributes such as the motion magnitude, contains a carefully designed test set adhering to the assumption of linearity by utilizing synthetic data, and evaluates the computational efficiency in a coherent manner.
Introduction
Video Frame Interpolation (VFI) is a cornerstone of modern video processing, essential for compression, display technology, and creative effects. However, the current state of VFI evaluation is fragmented. Researchers from TU Darmstadt and Adobe Research highlight three critical issues: inconsistent metric implementations (e.g., PSNR), the "oracle problem" where non-linear motion in test sets makes interpolation mathematically impossible from two frames, and a lack of granular performance analysis across different motion types and occlusions.
Methodology
To standardize evaluation, the authors introduced a centralized submission platform that computes metrics using uniform code and hardware. This ensures an "apples-to-apples" comparison across different research papers.
Key methodological contributions include:
- PSNR* Metric: A stabilized PSNR variant that weights every pixel in the dataset equally, preventing individual frames from disproportionately affecting the final score.
- Temporal Consistency (PSNR*$\\sigma$): A metric specifically for multi-frame interpolation that penalizes temporal artifacts like jitter.
- Linearity-Compliant Synthetic Data: By utilizing programmatically controlled motion of sprites over high-quality backgrounds, the authors created a dataset where motion is guaranteed to be linear, allowing for a pure test of interpolation algorithms without the need for an "oracle."
The Benchmarking Dataset
The proposed benchmark includes a new synthetic dataset of 666 nine-frame sequences at resolutions up to 4K. Unlike existing real-world datasets like Vimeo-90k or UCF101, this dataset provides perfect ground truth for motion magnitude, directional angle, and occlusion masks. This allows researchers to dissect model performance based on specific pixel-level attributes.
Analysis and Insights
The benchmark provides several key insights into the current state of VFI:
- The Linearity Factor: Rankings of state-of-the-art models change significantly when non-linear pixels are excluded, suggesting that current benchmarks may be rewarding models for their ability to guess non-linear paths rather than their interpolation logic.
- Directional Bias: Analysis shows that many models perform worse on vertical or diagonal motion, likely inheriting biases from horizontally-dominant training sets.
- The Occlusion Challenge: Performance drops significantly in "2-occ" regions—areas occluded in both input frames—where models must hallucinate missing content.
Results
The authors evaluated 21 representative VFI methods. Standout findings include the observation that ensembling techniques can artificially inflate rankings, leading to a new requirement for authors to disclose such methods. Additionally, the benchmark provides standardized runtime evaluations on NVIDIA A100 GPUs, highlighting the trade-offs between interpolation quality and computational efficiency.
Conclusion
This work establishes a rigorous "North Star" for the VFI community by standardizing metrics and providing a dataset that respects the fundamental assumptions of the task. While currently focused on rigid motion, future work will aim to incorporate non-rigid deformations and complex artifacts like motion blur, further refining the evaluation of video synthesis technologies.
Summary updated