Video Frame Interpolation with Flow Transformer

Pan Gao, Haoyue Tian, Jie Qin

TL;DR

This paper introduces VFIFT, a novel Transformer-based approach for video frame interpolation that leverages optical flow to guide temporal self-attention. By precisely locating relevant features across frames based on motion, VFIFT achieves state-of-the-art visual quality and efficiency, outperforming existing methods, especially in scenarios with large motion.

What Problem Does This Paper Address?

Video frame interpolation (VFI) aims to generate intermediate frames between existing ones to create smoother video playback, enable slow-motion effects, and improve video quality. Traditional methods often rely on Convolutional Neural Networks (CNNs). However, CNNs, due to their inherent weight-sharing and local receptive fields, struggle with accurately interpolating frames that involve significant motion or large spatial displacements between frames. While Transformers have shown promise in capturing long-range dependencies, applying them effectively to VFI while incorporating motion information has remained a challenge. Existing Transformer-based VFI methods either don't fully utilize optical flow or are highly sensitive to its accuracy. This paper addresses the gap of effectively integrating motion dynamics, specifically optical flow, into a Transformer architecture for VFI to achieve superior performance and robustness.

Methods

The proposed method, Video Frame Interpolation Flow Transformer (VFIFT), is built upon a Transformer architecture. Its core innovation lies in the Flow Transformer Block (FTFB) and the Flow Transformer Attention (FTFA) module.

  • Information Extraction: The network first extracts features from the input frames and predicts optical flow using a U-Net-like structure. This flow indicates the motion between the target frame and the reference frames.
  • Flow Transformer Attention (FTFA): This is the key component. Instead of performing global self-attention, FTFA uses the predicted optical flow to dynamically locate a local reference area on the feature maps of the corresponding frame. This allows the attention mechanism to focus on relevant pixels that have moved, rather than just co-located pixels. This "flow-guided local attention" is crucial for handling large motions.
  • Multi-Scale Architecture: To account for varying motion scales, the model employs a multi-scale architecture, processing features at different resolutions.
  • Occlusion Prediction: To handle occlusions where parts of the scene might be visible in one frame but not the other, a dedicated occlusion prediction module estimates visibility weights for each frame, which are then used to blend the warped frames.
  • Residual Learning: The final interpolated frame is generated by adding a predicted residual to the blended frame, further refining the output.

Datasets

The paper evaluates its method on three standard VFI benchmarks:

  • Vimeo90K: A large-scale dataset designed for time frame interpolation, consisting of 3-frame sequences.
  • UCF101: A dataset of human actions, offering diverse motion patterns.
  • Davis480p: A dataset known for challenging scenarios with object occlusion, motion blur, and appearance changes.

Key Insights and Ideas

  • Optical Flow as a Precise Locator: The paper demonstrates that optical flow can be used not just for warping but as a precise guide to locate relevant features for temporal self-attention, significantly improving performance, especially for large motions.
  • Local Attention with Motion Guidance: By combining local attention with optical flow guidance, the method achieves a balance between capturing relevant temporal information and maintaining computational efficiency, avoiding the high cost of global attention.
  • Breaking CNN Limitations: The Transformer-based approach effectively overcomes the limitations of CNNs related to kernel weight sharing and locality, leading to better handling of complex motion.
  • Multi-Scale Motion Handling: The multi-scale architecture is essential for capturing and interpolating motions that occur at different scales within a video sequence.

Key Results

  • Quantitative Performance: VFIFT achieves state-of-the-art results on Vimeo90K, UCF101, and Davis480p datasets, outperforming both CNN-based and previous Transformer-based methods in terms of PSNR and SSIM. For instance, on Vimeo90K, it achieves a PSNR of 36.43 dB.
  • Efficiency: The proposed method demonstrates competitive or superior performance with a similar or even lower number of parameters compared to other advanced Transformer models. It also boasts the fastest inference speed among the compared methods.
  • Qualitative Results: Visual comparisons show that VFIFT generates smoother interpolated frames with fewer artifacts, particularly for scenes with fast-moving objects and complex motion, as evidenced in Figures 9 and 10.

Limitations and Future Work

  • Parameter Sensitivity: While the paper highlights efficiency, the performance can be sensitive to the size of the local attention window (e.g., 5x5 was found optimal).
  • Optical Flow Accuracy: Although the method is designed to be robust, extreme inaccuracies in optical flow prediction could still impact performance.
  • Future Work: The authors suggest further exploration of Transformer-based architectures for VFI, potentially incorporating more advanced motion modeling techniques or exploring different attention mechanisms. The current work focuses on single-frame interpolation; extending to multi-frame interpolation could be a future direction.

Summary updated

BibTeX
@inproceedings{gao2023video,
  author = {Pan Gao and Haoyue Tian and Jie Qin},
  title = {{Video Frame Interpolation with Flow Transformer}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3612440},
}

Loading related content...

Figure