Frame Interpolation Transformer and Uncertainty Guidance

Markus Plack, Matthias B. Hullin, Karlis Martins Briedis, Markus Gross, Abdelaziz Djelouah, Christopher SchroersCVPR 2023 · DOI

Introduction

Video frame interpolation (VFI)—the process of generating intermediate frames between existing images—is a cornerstone of video editing, slow-motion generation, and computer graphics. However, traditional methods often struggle with "hard" cases involving complex lighting, occlusions, and extreme motion, which typically result in blurry artifacts. In professional production environments, such as those at DisneyResearch|Studios, these failures create two major challenges: the need for manual quality control and the high cost of rendering high-end CG.

Researchers from the University of Bonn, ETH Zurich, and Disney have developed a new transformer-based architecture that addresses these issues by predicting its own "uncertainty." This allows the model to guide a partial rendering process, achieving production-quality results by rendering only a small fraction of the most difficult pixels.

Proposed Architecture

The core of the system is a Frame Interpolation Transformer that adopts a unified approach to frame generation. Unlike models that treat start and end frames as fixed anchors, this architecture uses a masking system. It can process any combination of the start frame ($I_0$), the end frame ($I_2$), and—crucially—optional patches from the middle frame ($I_1$).

The hierarchical architecture of the Frame Interpolation Transformer, featuring weight-sharing across levels and specialized fusion blocks.

Technical Innovations

  • MACE Blocks: The model replaces standard linear layers with "Multihead-Attention Convolutional Encoders" (MACE). This allows the network to maintain spatial structure while benefiting from the global reasoning of attention mechanisms.
  • Unified Masking: By using binary masks to indicate valid input pixels, the network seamlessly transitions from standard two-frame interpolation to a "guided" mode where it fills in gaps around real rendered patches.
  • Dual Error Prediction: The network outputs five channels: three for the RGB image and two for "uncertainty maps" that predict both $L_2$ and perceptual error.

Uncertainty Guidance and Hybrid Rendering

The standout feature of this research is Uncertainty Guidance, which treats predicted error as an actionable control signal rather than just a metric. The model identifies 16x16 pixel tiles where it expects high error. In a CG workflow, a renderer can be instructed to render only those specific tiles. These high-quality patches are then fed back into the network for a second pass, resulting in a near-perfect final frame.

Visual comparison and demonstration of Uncertainty Guidance. The model identifies high-error regions (E1) and uses rendered patches (I1) to refine the final output, outperforming SOTA methods like ABME and VFIformer.

Key Results and Performance

The model was trained on the Vimeo-90K dataset and evaluated against standard benchmarks like SNU-FILM and DAVIS, as well as open-source Blender movies.

  • State-of-the-Art Robustness: The model achieved the best PSNR and SSIM scores on the "Hard" and "Extreme" categories of the SNU-FILM dataset, proving its effectiveness against large motion.
  • Perceptual Superiority: In a user study with 69 participants, users strongly preferred this method over existing state-of-the-art models like ABME and RIFE, particularly noting the sharpness of textures.
  • Efficiency through Sparsity: In animated film tests, using uncertainty guidance to replace just a fraction of the tiles improved the PSNR by 6.99 to 9.98 dB compared to standard interpolation. This demonstrates that rendering only 25% of the most "uncertain" tiles can bring an interpolated frame remarkably close to a full ground-truth render.

Conclusion

By integrating uncertainty estimation into a transformer-based VFI pipeline, this work provides a practical solution for high-end video production. While the model is computationally larger and slower than some alternatives, its ability to significantly reduce rendering costs through sparse, guided rendering offers a compelling trade-off for professional studios. Future work may explore the parallels between this masked training approach and masked language models to further improve video understanding.

Summary updated

BibTeX
@inproceedings{plack2023frame,
  author = {Markus Plack and Matthias B. Hullin and Karlis Martins Briedis and Markus Gross and Abdelaziz Djelouah and Christopher Schroers},
  title = {{Frame Interpolation Transformer and Uncertainty Guidance}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00946},
}
Figure