Enhanced Correlation Matching Based Video Frame Interpolation

Sungho Lee, Narae Choi, Woong Il ChoiWACV 2022

TL;DR

This paper introduces ECMNet, a novel deep learning framework for high-resolution (4K) video frame interpolation that excels at handling large motions and occlusions. It achieves state-of-the-art performance with significantly fewer parameters by employing an enhanced correlation matching strategy based on forward-warped features within a recurrent pyramid architecture.

What Problem Does This Paper Address?

Video Frame Interpolation (VFI) is crucial for applications like increasing frame rates or creating slow-motion videos. While deep learning has advanced VFI quality, existing methods struggle with high-resolution videos (like 4K) and scenes with significant motion or occlusions. This limitation stems from the fixed receptive field of CNNs, which restricts their ability to capture large-scale motion. Simply increasing network size or depth to address this leads to prohibitive computational costs and memory footprints. Existing pyramid-based approaches mitigate this by progressively refining optical flow, but they can still suffer from error propagation and require retraining for different resolutions.

Methods

The proposed ECMNet framework comprises three main sub-networks:

  • Bi-directional Optical Flow Estimation: This module is the core innovation. It utilizes a recurrent pyramid architecture, sharing parameters across pyramid layers to efficiently extend the receptive field without increasing model size. The key novelty here is an enhanced correlation matching algorithm that uses forward-warped features. Unlike traditional methods that use backward warping (which can create duplicated or ambiguous regions, especially around occlusions), forward warping aligns features more cleanly, leading to more accurate flow estimation. The algorithm then identifies the highest correlation to update the flow. Learnable importance weights are introduced to handle potential issues with multiple pixels mapping to the same target pixel.
  • Frame Synthesis: This module generates an intermediate frame at an arbitrary temporal position t (between 0 and 1). It first estimates intermediate flows using a flow reversal approach to avoid hole generation, compensating for potential errors with flow residuals. Then, it performs warping and blending of the input frames using a learned blending mask, which acts as a confidence ratio for the warped images.
  • Refinement Network: This final module enhances the synthesized frame by correcting artifacts that may arise from flow estimation failures, occlusions, or brightness changes. It employs dilated convolutions to increase its receptive field, effectively filling in missing regions and improving the overall quality of the interpolated frame.

Datasets

The paper evaluates its method on several datasets:

  • X4K1000FPS (4K): Used for training and evaluating the high-resolution model.
  • VVC (Versatile Video Coding reference dataset, 4K): Used for evaluating the high-resolution model.
  • Vimeo90K (240p): Used for training and evaluating the low-resolution model.
  • UCF-101: Used for evaluating the low-resolution model.

Key Insights and Ideas

  • Forward Warping for Correlation: The most significant insight is the advantage of using forward-warped features for correlation matching in optical flow estimation. This approach inherently handles occlusions better than backward warping, leading to more robust and accurate flow predictions, especially in challenging scenarios.
  • Recurrent Pyramid for Efficiency: Leveraging a recurrent pyramid structure with shared weights is a clever way to extend the receptive field and improve flow accuracy across multiple scales without the computational burden of a deep, unshared pyramid.
  • End-to-End Trainability: The integration of convolution layers and differentiable operators allows for seamless end-to-end training of the entire framework, simplifying the optimization process.
  • Lightweight Model for High Resolution: The paper demonstrates that achieving state-of-the-art performance on high-resolution data is possible with a significantly reduced number of model parameters, making the approach more practical.

Key Results

  • State-of-the-Art Performance: ECMNet achieves state-of-the-art results on the X4K1000FPS (4K) dataset, outperforming existing methods in terms of PSNR and SSIM.
  • Parameter Efficiency: The proposed method uses the fewest parameters among the compared methods for 4K video frame interpolation. For example, the "Ours4k" model with 6 pyramid layers has only 4.7 million parameters, significantly less than methods like XVFI.
  • Low-Resolution Performance: The model also performs competitively on low-resolution datasets like Vimeo90K and UCF-101, demonstrating its versatility.
  • Qualitative Improvements: Visual comparisons show that ECMNet better captures objects and details in challenging scenarios, particularly in areas with large motion and occlusions, compared to other methods.

Limitations and Future Work

  • Limitations: The paper acknowledges that while their method handles large motion and occlusions better than previous approaches, it's still a challenge in VFI. The performance is also dependent on the accuracy of the flow estimation.
  • Future Work: The authors suggest that further improvements could be made by exploring more advanced flow estimation techniques and potentially incorporating explicit depth information for even more robust handling of complex scenes. The recurrent pyramid structure offers flexibility, and future work could explore optimal pyramid configurations for various video types.

Summary updated

BibTeX
@inproceedings{lee2022enhanced,
  author = {Sungho Lee and Narae Choi and Woong Il Choi},
  title = {{Enhanced Correlation Matching Based Video Frame Interpolation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure