Frame Interpolation for Dynamic Scenes With Implicit Flow Encoding

Pedro Figueirêdo, Avinash Paliwal, Nima Khademi KalantariWACV 2023

TL;DR

This paper tackles the challenge of creating smooth video transitions between two images, especially when lighting conditions change significantly between them. They achieve this by using a novel approach that implicitly learns to interpolate optical flow fields using a hypernetwork and a coordinate-based neural network, leading to more realistic intermediate frames than existing methods.

What Problem Does This Paper Address?

Creating realistic intermediate frames between two images (frame interpolation) is crucial for generating smooth videos from sequences of photos. While deep learning has made significant strides, existing methods struggle when the input images have different lighting or illumination. This is because most frame interpolation models are trained on datasets with consistent lighting, causing their learned optical flow estimation to degrade under varying conditions. This leads to artifacts and unnatural motion in the interpolated frames. The authors aim to bridge this gap by developing a method that can robustly handle these illumination changes.

Methods

The core of this paper's contribution lies in its novel approach to interpolating optical flows. Instead of directly interpolating images, they focus on interpolating the flows between frames.

Here's a breakdown of their method:

  • Leveraging Pre-trained Robust Flow Networks: They start by using a pre-trained optical flow network (specifically RAFT [58]) that is known to be robust to illumination changes. This network provides bidirectional flows between the two input images ($F_{t_0 \to t_1}$ and $F_{t_1 \to t_0}$).
  • Implicit Flow Interpolation with Coordinate-Based Networks: The challenge is that these pre-trained networks only provide flows between the input frames, not for an intermediate frame. To address this, they propose using a coordinate-based neural network (specifically SIREN [55]) to learn a continuous representation of the flow field.
  • Hypernetwork for Time Encoding: A key innovation is the use of a hypernetwork. This hypernetwork takes the time coordinate ($t$) as input and outputs the weights for the SIREN network. This allows the SIREN to learn a mapping from spatial coordinates $(x, y)$ and time ($t$) to the flow vector. By encoding both input flows into a single hypernetwork, the resulting weights are more correlated and interpolatable, leading to smoother intermediate flows.
  • Blending with Existing Frameworks: Once the intermediate flows are estimated, they are fed into an existing blending network (FILM [47]) to synthesize the final interpolated image. This modular approach allows them to leverage the strengths of both robust flow estimation and effective image synthesis.

Datasets

The paper mentions using the following datasets for evaluation:

  • Xiph 2K and 4K [38]: Used for evaluating the quality of the interpolated flows.
  • Sintel dataset [23]: Also used for evaluating the quality of the interpolated flows.

The authors note that constructing datasets with input images exhibiting realistic lighting variations and their corresponding ground truth intermediate images is challenging, which is why they focus on numerically evaluating the intermediate flows.

Key Insights and Ideas

  • Flow is Key, Not Just Images: The paper highlights that the primary bottleneck for frame interpolation under varying illumination is the quality of the estimated optical flow, not necessarily the image blending itself.
  • Implicit Representation for Continuous Flows: Using coordinate-based neural networks (like SIREN) to represent optical flow allows for a continuous interpolation in time, enabling estimation at any arbitrary time step.
  • Hypernetwork for Smooth Interpolation: The hypernetwork is crucial for achieving smooth and accurate interpolation of flows. It effectively learns a "shape" of the flow that can be smoothly transitioned across time, preventing the "averaging" artifacts seen in simpler approaches.
  • Modularity for Future Improvements: By separating flow estimation and blending, their approach can benefit from future advancements in either component. For instance, a better pre-trained flow network could directly improve their results.

Key Results

  • Quantitative Flow Improvement: The paper reports significant improvements in the average end-point error (EPE) of interpolated flows compared to the FILM approach. For example, on the Xiph 4K dataset, their EPE is 17.46, while FILM's is 37.34. Similar improvements are seen on Xiph 2K and Sintel.
  • Qualitative Visual Quality: Visual comparisons on challenging scenes (e.g., BABY, HOUSE, HUG) demonstrate that their method produces more realistic intermediate images with fewer artifacts, better preservation of details, and more natural motion compared to ABME and FILM, especially in the presence of significant lighting changes.
  • Ablation Studies: The paper validates the effectiveness of their hypernetwork approach, showing it produces high-quality interpolated flows where other methods fail. They also identify optimal ranges for SIREN frequencies and time coordinate distances for best performance.

Limitations and Future Work

  • Dependence on Flow Network: The quality of their interpolated flows is inherently dependent on the accuracy of the pre-trained flow network (RAFT). If RAFT fails on a particularly challenging scenario, artifacts may appear in their results.
  • Direct Image Interpolation Challenges: The authors explored directly interpolating images but found it unsuccessful, suggesting that images are more complex than optical flows for this type of implicit representation.
  • Future Directions: They suggest that as better optical flow estimation methods become available, their framework can be easily updated to incorporate them for further improvements. They also propose further investigation into direct image interpolation methods.

Summary updated

BibTeX
@inproceedings{figueiredo2023frame,
  author = {Pedro Figueirêdo and Avinash Paliwal and Nima Khademi Kalantari},
  title = {{Frame Interpolation for Dynamic Scenes With Implicit Flow Encoding}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure