Video Frame Interpolation with Flow Transformer

Pan Gao, Haoyue Tian, Jie Qin

TL;DR

This paper introduces VFIFT, a novel Transformer-based approach for video frame interpolation that leverages optical flow to guide temporal self-attention. By focusing attention on relevant local regions informed by motion, VFIFT achieves state-of-the-art visual quality and efficiency, outperforming existing methods, especially in scenarios with large motion.

What Problem Does This Paper Address?

Video Frame Interpolation (VFI) aims to generate intermediate frames between existing ones to create smoother video playback, reduce motion blur, and enable slow-motion effects. Traditional methods often rely on Convolutional Neural Networks (CNNs). However, CNNs, due to their inherent weight-sharing and local receptive fields, struggle with accurately interpolating frames involving large motion or complex object movements, often leading to artifacts and loss of detail. While Transformers have shown promise in capturing long-range dependencies, adapting them effectively for VFI, especially by incorporating motion dynamics, remains an open challenge. Existing Transformer-based VFI methods either don't fully leverage optical flow or are highly sensitive to its accuracy.

Methods

The core innovation of VFIFT lies in its Flow Transformer Block (FTFB), which integrates optical flow directly into the self-attention mechanism. Instead of applying global self-attention, VFIFT uses predicted optical flow to identify corresponding local regions in the input frames. Temporal self-attention is then computed within these flow-guided local areas. This approach allows the model to focus on relevant motion information, improving accuracy for large-motion scenes while maintaining computational efficiency.

Key components include:

  • Information Extraction: A U-Net-based network predicts optical flow (motion vectors) and extracts feature maps from input frames. It also includes an occlusion prediction module to handle visibility issues.
  • Flow Transformer Layer: This layer houses the FTFB. The Flow Transformer Attention (FTFA) module is the heart of the method. It takes input features, projects them into Query, Key, and Value, and then uses optical flow to warp the Key and Value to a local reference area around the query's predicted position. Self-attention is then performed within this localized, motion-aligned region.
  • Multi-scale Architecture: The model incorporates multi-scale processing to better handle diverse motion scales.
  • Residual Learning: The final interpolated frame is generated by adding a predicted residual to a blended frame, which helps in refining the output.

Datasets

The paper evaluates VFIFT on three standard VFI benchmarks:

  • Vimeo90K: A large-scale dataset designed for VFI, consisting of 3-frame sequences.
  • UCF101: A real-world action video dataset.
  • Davis480p: A dataset often used for video object segmentation, chosen here to test adaptability to challenging environments.

Key Insights and Ideas

  • Optical Flow as a Guide for Attention: The paper demonstrates that directly using optical flow to guide the spatial localization of self-attention is crucial for effective VFI, especially for handling large motions. This moves beyond simply using flow for warping.
  • Local Attention for Efficiency and Focus: By restricting self-attention to flow-guided local regions, VFIFT achieves better computational efficiency and a stronger focus on relevant object motion compared to global attention or standard CNN convolutions.
  • Transformer's Advantage in Motion: The Transformer architecture, when combined with flow-guided attention, proves superior to CNNs in breaking the limitations of kernel weight sharing and capturing complex motion dynamics.
  • Multi-scale Processing is Key: Accounting for multi-scale motion through a multi-scale architecture significantly boosts performance.

Key Results

  • State-of-the-Art Performance: VFIFT achieves superior quantitative results (PSNR and SSIM) on Vimeo90K, UCF101, and Davis480p compared to many existing state-of-the-art methods, including other Transformer-based approaches like VFIT and VFIformer.
  • Efficiency: VFIFT demonstrates faster inference times than most compared methods, including both CNN and Transformer-based ones, while achieving better or comparable performance.
  • Visual Quality: Qualitative results show that VFIFT generates smoother interpolated frames with fewer artifacts, particularly for sequences with fast-moving objects and complex motion, outperforming competitors visually.
  • Ablation Studies: Experiments confirm the importance of the flow attention mechanism, the multi-scale architecture, and the occlusion prediction module for overall performance. A 5x5 local attention window size was found to be optimal.

Limitations and Future Work

  • Sensitivity to Flow Accuracy: While VFIFT is designed to be robust, the paper acknowledges that its performance can still be sensitive to the accuracy of the predicted optical flow, especially in highly challenging scenarios.
  • Parameter Count: While VFIFT is efficient, it is noted that VFIformer-B achieves slightly better results on some datasets but with a higher parameter count. This suggests potential for further optimization or architectural improvements to match or exceed VFIformer-B's performance with similar or fewer parameters.
  • Future Work: The authors suggest exploring more advanced Transformer architectures and further optimizing the flow estimation and attention mechanisms for even greater accuracy and efficiency. The multi-scale aspect could also be further explored.

Summary updated

Abstract

Video frame interpolation has been actively studied with the development of convolutional neural networks. However, due to the intrinsic limitations of kernel weight sharing in convolution, the interpolated frame generated by it may lose details. In contrast, the attention mechanism in Transformer can better distinguish the contribution of each pixel, and it can also capture long-range pixel dependencies, which provides great potential for video interpolation. Nevertheless, the original Transformer is commonly used for 2D images; how to develop a Transformer-based framework with consideration of temporal self-attention for video frame interpolation remains an open issue. In this paper, we propose Video Frame Interpolation Flow Transformer to incorporate motion dynamics from optical flows into the self-attention mechanism. Specifically, we design a Flow Transformer Block that calculates the temporal self-attention in a matched local area with the guidance of flow, making our framework suitable for interpolating frames with large motion while maintaining reasonably low complexity. In addition, we construct a multi-scale architecture to account for multi-scale motion, further improving the overall performance. Extensive experiments on three benchmarks demonstrate that the proposed method can generate interpolated frames with better visual quality than state-of-the-art methods.
BibTeX
@inproceedings{gao2023video,
  author = {Pan Gao and Haoyue Tian and Jie Qin},
  title = {{Video Frame Interpolation with Flow Transformer}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3612440},
}
Figure