FloLPIPS: A Bespoke Video Quality Metric for Frame Interpoation

Duolikun Danier, Fan Zhang, David R. Bull

TL;DR

This paper introduces FloLPIPS, a novel video quality metric specifically designed for evaluating frame interpolation. By incorporating temporal information through optical flow differences to weight spatial feature discrepancies, FloLPIPS significantly outperforms existing metrics in predicting human perception of interpolated video quality.

What Problem Does This Paper Address?

Video Frame Interpolation (VFI) is crucial for enhancing video processing applications, from compression to visual effects. However, accurately assessing the perceptual quality of interpolated videos remains a significant challenge. Existing metrics like PSNR, SSIM, and even the popular perceptual image quality metric LPIPS, struggle to correlate well with human judgment for VFI. This gap forces researchers to rely on time-consuming and expensive subjective quality assessments, hindering progress in VFI algorithm development. The core problem is the lack of a reliable, automated metric that can effectively capture the unique distortions introduced by VFI.

Methods

FloLPIPS builds upon the Learned Perceptual Image Patch Similarity (LPIPS) metric. LPIPS extracts features from deep convolutional neural networks (CNNs) trained for image classification and compares the feature maps of reference and distorted images. FloLPIPS enhances this by addressing two key limitations for VFI:

  1. Temporal Distortion: LPIPS only considers spatial distortions. FloLPIPS incorporates temporal information by computing optical flow between consecutive frames in both the reference and distorted videos.
  2. Spatial Aggregation: LPIPS uses a simple spatial average of feature differences. FloLPIPS re-designs this by using the difference between the reference and distorted optical flow maps to weight the feature difference maps. This means regions with greater motion discrepancies (and thus likely more noticeable interpolation artifacts) contribute more to the final score.

The optical flow estimation uses a pre-trained PWC-Net, and LPIPS features are extracted using a pre-trained AlexNet (first five layers). The entire FloLPIPS framework is designed to be differentiable, allowing it to potentially be used as a perceptual loss function for training VFI models.

Datasets

The primary dataset used for evaluation is the BVI-VFI database. This dataset is highlighted as the only publicly available subjective quality database specifically containing uncompressed video sequences with only VFI-induced distortions. It comprises 36 reference sequences and 180 distorted sequences generated by five different VFI methods, covering a range of interpolation artifacts.

Key Insights and Ideas

  • Temporal Context is Crucial for VFI Quality: The paper strongly suggests that purely spatial metrics are insufficient for VFI. Capturing how motion is represented and distorted over time is key to predicting perceptual quality.
  • Motion Discrepancy as a Weighting Signal: Using the difference in optical flow between reference and distorted videos as a weighting mechanism for feature differences is a novel approach. This intuitively makes sense: areas with significant changes in motion are where interpolation errors are most likely to be perceived.
  • Bespoke Metrics Drive Progress: The success of FloLPIPS demonstrates the value of developing specialized metrics for specific video processing tasks, rather than relying on generic image or video quality measures.
  • Differentiability Opens New Avenues: The fact that FloLPIPS is differentiable is a significant advantage, paving the way for its use in end-to-end optimization of VFI networks.

Key Results

  • Superior Performance: FloLPIPS significantly outperforms 12 other commonly used image and video quality assessment metrics on the BVI-VFI dataset. It achieves the best correlation with subjective ground truth (DMOS) across Pearson's Linear Correlation Coefficient (PLCC), Spearman's Rank-Order Correlation Coefficient (SROCC), and Root Mean Squared Error (RMSE).
  • Significant Improvement over LPIPS: FloLPIPS shows substantial gains over its predecessor, LPIPS, with improvements of +0.109 in PLCC and +0.084 in SROCC.
  • Statistical Significance: An F-test confirms that FloLPIPS's performance improvement over the top-performing metrics is statistically significant at the 95% confidence interval.
  • Moderate Runtime: FloLPIPS offers a relatively moderate runtime, making it practical for evaluation.

Limitations and Future Work

  • Optical Flow Estimator Choice: While PWC-Net is used, the paper notes that the choice of optical flow estimator can impact performance. Using more advanced estimators like GMFlow can yield slightly better SROCC, suggesting that further improvements are possible by integrating state-of-the-art flow techniques.
  • Potential for Optimization: The authors suggest that FloLPIPS's differentiability could be leveraged to use it as a perceptual loss function for optimizing VFI methods, potentially leading to further improvements in interpolation quality.
  • Code Availability: The authors have made their code publicly available, which is a significant contribution to the research community and facilitates further work.

Summary updated

Abstract

Video frame interpolation (VFI) serves as a useful tool for many video processing applications. Recently, it has also been applied in the video compression domain for enhancing both conventional video codecs and learning-based compression architectures. While there has been an increased focus on the development of enhanced frame interpolation algorithms in recent years, the perceptual quality assessment of interpolated content remains an open field of research. In this paper, we present a bespoke full reference video quality metric for VFI, FloLPIPS, that builds on the popular perceptual image quality metric, LPIPS, which captures the perceptual degradation in extracted image feature space. In order to enhance the performance of LPIPS for evaluating interpolated content, we re-designed its spatial feature aggregation step by using the temporal distortion (through comparing optical flows) to weight the feature difference maps. Evaluated on the BVI-VFI database, which contains 180 test sequences with various frame interpolation artefacts, FloLPIPS shows superior correlation performance (with statistical significance) with subjective ground truth over 12 popular quality assessors. To facilitate further research in VFI quality assessment, our code is publicly available at https://danier97.github.io/FloLPIPS.
BibTeX
@inproceedings{danier2022flolpips,
  author = {Duolikun Danier and Fan Zhang and David R. Bull},
  title = {{FloLPIPS: A Bespoke Video Quality Metric for Frame Interpoation}},
  booktitle = {PCS},
  year = {2022},
  doi = {10.1109/pcs56426.2022.10018062},
}
Figure