A Perceptual Quality Metric for Video Frame Interpolation

Qiqi Hou, Abhijay Ghildyal, Feng Liu

Abstract

Research on video frame interpolation has made significant progress in recent years. However, existing methods mostly use off-the-shelf metrics to measure the quality of interpolation results with the exception of a few methods that employ user studies, which is time-consuming. As video frame interpolation results often exhibit unique artifacts, existing quality metrics sometimes are not consistent with human perception when measuring the interpolation results. Some recent deep learning-based perceptual quality metrics are shown more consistent with human judgments, but their performance on videos is compromised since they do not consider temporal information. In this paper, we present a dedicated perceptual quality metric for measuring video frame interpolation results. Our method learns perceptual features directly from videos instead of individual frames. It compares pyramid features extracted from video frames and employs Swin Transformer blocks-based spatio-temporal modules to extract spatio-temporal information. To train our metric, we collected a new video frame interpolation quality assessment dataset. Our experiments show that our dedicated quality metric outperforms state-of-the-art methods when measuring video frame interpolation results.

Introduction: The VFI Measurement Gap

Video Frame Interpolation (VFI) generates intermediate frames to create smooth slow-motion or higher frame rates. However, the process is notorious for producing unique artifacts that differ significantly from standard video compression or blur. VFI often results in structural deformities, such as twisted objects or "ghosting," where moving objects appear in multiple locations simultaneously.

Comparison of artifacts across different tasks. VFI results (rightmost) exhibit unique structural deformities compared to compression or deblurring.

Traditional metrics like PSNR and SSIM, which focus on pixel-level differences, often miss these structural warps. Even modern deep-learning metrics like LPIPS fall short because they operate on individual images, ignoring the temporal flow and continuity essential for a high-quality video experience. To address this, we developed VFIPS, the first dedicated perceptual quality metric specifically designed for VFI.

Methodology

VFIPS utilizes a learned perceptual video similarity architecture with several specialized components:

  • Feature Extraction Pyramid: The network extracts features using a five-level pyramid, allowing the model to analyze distortions across multiple scales, from fine textures to large structural shapes.
  • Spatio-Temporal (ST) Module: By employing Swin Transformer blocks, the model captures interactions between spatial regions and across time. This allows the metric to "watch" how artifacts evolve over a 12-frame sequence.
  • Siamese Training: The model is trained to predict human preference by comparing two different interpolated versions of the same scene against a ground-truth reference.
  • Architecture Optimization: The authors found that removing Layer Norm (LN) was critical, as standard normalization can erase the subtle feature differences needed to distinguish high-quality video from low-quality results.

The VFIPS Dataset

To train and validate the metric, we introduced the VFIPS Dataset, comprising 25,887 video triplets generated by 11 state-of-the-art VFI methods.

The hybrid annotation strategy: LPIPS was used for clear-cut cases, while human participants labeled the 'hard' cases where LPIPS scores were too close to be reliable.

Labeling this dataset required a hybrid strategy. For pairs where LPIPS scores differed significantly, automatic labels were used. For "hard" cases where LPIPS was uncertain, 41 human participants provided manual preferences to ensure the ground truth aligned with human perception.

Key Results and Insights

VFIPS consistently outperforms existing image and video quality metrics across multiple benchmarks:

  • Human Alignment: On the VFIPS validation set, the model achieved a 2AFC (Two Alternative Forced Choice) score of 0.830, beating LPIPS (0.808) and VMAF (0.805).
  • Correlation: On the BVI-VFI test set, it reached a Spearman Correlation (SROCC) of 0.794, a significant improvement over the next best video-specific metric, STRRED (0.614).
  • Temporal Consistency: The results confirm that for human viewers, temporal consistency is often more important than per-frame sharpness. VFIPS successfully captures this by analyzing motion windows rather than static frames.

Limitations and Future Work

While highly effective, VFIPS currently requires approximately 2500 MB of GPU memory to process a 12-frame sequence, making it difficult to integrate directly as a loss function during the training of new VFI models. Additionally, the model provides a global quality score but does not yet generate spatial error maps (heatmaps) to pinpoint specific distortion locations. Future work will focus on reducing network size and exploring the metric's potential as a perceptual loss function.

Summary updated

Abstract

Research on video frame interpolation has made significant progress in recent years. However, existing methods mostly use off-the-shelf metrics to measure the quality of interpolation results with the exception of a few methods that employ user studies, which is time-consuming. As video frame interpolation results often exhibit unique artifacts, existing quality metrics sometimes are not consistent with human perception when measuring the interpolation results. Some recent deep learning-based perceptual quality metrics are shown more consistent with human judgments, but their performance on videos is compromised since they do not consider temporal information. In this paper, we present a dedicated perceptual quality metric for measuring video frame interpolation results. Our method learns perceptual features directly from videos instead of individual frames. It compares pyramid features extracted from video frames and employs Swin Transformer blocks-based spatio-temporal modules to extract spatio-temporal information. To train our metric, we collected a new video frame interpolation quality assessment dataset. Our experiments show that our dedicated quality metric outperforms state-of-the-art methods when measuring video frame interpolation results. Our code and model are made publicly available at \url{https://github.com/hqqxyy/VFIPS}.
BibTeX
@inproceedings{hou2022perceptual,
  author = {Qiqi Hou and Abhijay Ghildyal and Feng Liu},
  title = {{A Perceptual Quality Metric for Video Frame Interpolation}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  doi = {10.1007/978-3-031-19784-0\_14},
}
Figure