Abstract
Videos have recently become an omnipresent form of media, gathering much attention from industry as well as academia. In the video enhancement field, video frame interpolation is a long-studied topic that has dramatically improved due to the advancement of deep convolutional neural networks (CNN). However, conventional approaches utilizing two successive frames often exhibit ghosting or tearing artifacts for moving objects. We argue that this phenomenon comes from the lack of reliable information provided only by two frames. With this motivation, we propose a frame interpolation method by utilizing tridirectional information obtained from three input frames. Information extracted from triplet frames allows our model to learn rich and reliable inter-frame motion representations, including subtle nonlinear movement, which can be easily trained via any video frames in a self-supervised manner. We demonstrate that our method generalizes well to high-resolution content by evaluating on FHD resolution, and illustrates our approach's effectiveness via comparison to state-of-the-art methods on challenging video content.
Introduction
Video frame interpolation (VFI) is the process of generating intermediate frames between existing ones to create smoother motion, such as converting 30fps footage to 60fps. Most existing methods are bidirectional, relying on two consecutive frames ($I_1$ and $I_2$) and assuming linear motion between them. However, real-world motion is rarely linear; objects accelerate, rotate, and change direction. When models rely on only two frames, they often fail to account for these nonlinearities, resulting in "ghosting" (blurry double images) or "tearing" (jagged edges) around fast-moving objects.
Tridirectional Inference Framework
To address these limitations, we propose a Tridirectional Inference framework that utilizes a triplet of frames ($I_1, I_2, I_3$) to interpolate intermediate positions. By leveraging three points of reference, the model can estimate a "free-form curve" of motion rather than a simple straight-line midpoint. The architecture consists of three primary modules:
- Flow Estimation Module (FEM): Uses PWC-Net to calculate bidirectional optical flow for every combination of the three input frames.
- Nonlinear Motion Estimation Module (NLM): A U-Net-based architecture that refines the initial flow maps into four nonlinear flow maps. By analyzing the relationship between all three frames, the NLM "triangulates" the true path of an object.
- Frame Generation Module (FGM): Synthesizes the final interpolated frames using the refined flow to warp input frames, employing ResNet blocks and attention layers for robustness against residual errors.
To further enhance performance on challenging sequences, we introduced a "Flying Objects" data augmentation scheme, overlaying segmented objects from the PASCAL VOC dataset onto video frames with randomized, fast-moving trajectories.
Experimental Results
Our tridirectional model consistently outperforms state-of-the-art bidirectional methods like SepConv, SuperSlomo, and TOFlow across standard benchmarks, including Vimeo90k and SMBV. The model shows significant gains in categories featuring complex motion, such as fluttering cloth or spinning objects.
Qualitative analysis (as seen in the figure above) demonstrates that our method successfully reconstructs the trajectory of fast-moving objects, such as a ping-pong ball, where other models produce significant artifacts. Despite the increased input data, the model remains efficient with approximately 10.4 million parameters.
Conclusion
By shifting from bidirectional to tridirectional inference, we provide a more reliable foundation for estimating complex, nonlinear motion in video. This approach significantly reduces common interpolation artifacts and generalizes well to high-resolution content, offering a robust solution for professional video enhancement tasks.
Summary updated