Abstract
Video frame interpolation methodologies endeavor to create novel frames betwixt extant ones, with the intent of augmenting the video's frame frequency. However, current methods are prone to image blurring and spurious artifacts in challenging scenarios involving occlusions and discontinuous motion. Moreover, they typically rely on optical flow estimation, which adds complexity to modeling and computational costs. To address these issues, we introduce a Motion-Aware Video Frame Interpolation (MA-VFI) network, which directly estimates intermediate optical flow from consecutive frames by introducing a novel hierarchical pyramid module. It not only extracts global semantic relationships and spatial details from input frames with different receptive fields, enabling the model to capture intricate motion patterns, but also effectively reduces the required computational cost and complexity. Subsequently, a cross-scale motion structure is presented to estimate and refine intermediate flow maps by the extracted features. This approach facilitates the interplay between input frame features and flow maps during the frame interpolation process and markedly heightens the precision of the intervening flow delineations. Finally, a discerningly fashioned loss centered around an intermediate flow is meticulously contrived, serving as a deft rudder to skillfully guide the prognostication of said intermediate flow, thereby substantially refining the precision of the intervening flow mappings. Experiments illustrate that MA-VFI surpasses several representative VFI methods across various datasets, and can enhance efficiency while maintaining commendable efficacy.
Introduction
Video Frame Interpolation (VFI) is the task of generating "in-between" frames to enhance video smoothness, such as converting 30fps to 60fps. Despite advancements in deep learning, VFI faces two primary challenges: handling complex, non-linear motion and balancing computational efficiency with visual quality. Many existing methods assume linear motion, leading to "ghosting" artifacts or blurring when objects accelerate or rotate. Furthermore, high-quality methods often rely on heavy optical flow networks that are too slow for real-time use.
Motion-Aware Video Frame Interpolation (MA-VFI) addresses these hurdles by directly estimating intermediate optical flow through a lightweight, hierarchical architecture that balances real-time performance with state-of-the-art visual accuracy.
Methodology
The MA-VFI architecture is built upon three core components designed to capture intricate motion patterns while maintaining efficiency.
- Hierarchical Pyramid Feature Extraction: The model utilizes a four-layer pyramid with varying receptive fields (using $7\times7$ and $3\times3$ filters). This allows the network to capture both global semantic context for large-scale motion and fine spatial details for small-object motion simultaneously.
- Cross-Scale Motion Structure: This top-down refinement module estimates flow at a coarse scale and progressively refines it. By using warped features from previous levels to align current features, the network enables a symbiotic interaction between motion prediction and feature representation.
- Intermediate Flow-Directed Loss: To improve precision without ground-truth flow, the authors employ knowledge distillation. A pre-trained LiteFlownet acts as a "teacher" during training, penalizing the student network if its estimated intermediate flow deviates significantly from the teacher's output.
The model was trained on the Vimeo-90k dataset and evaluated on benchmarks including UCF101, Middlebury, and Adobe-240fps to ensure consistency across general motion and multi-frame interpolation tasks.
Experimental Results
MA-VFI demonstrates a superior speed-to-accuracy ratio compared to 31 other state-of-the-art (SOTA) methods.
- Accuracy: On the UCF101 dataset, MA-VFI achieved a PSNR of 35.31 dB, outperforming models like RIFE (35.28 dB) and DAIN (35.00 dB).
- Efficiency: The network processes a $448\times256$ resolution frame in approximately 0.021 seconds (~47 FPS). With only 17.87M parameters, it is significantly more compact than models like CAIN (42.78M) while delivering higher visual quality.
- Qualitative Performance: Visual analysis confirms that MA-VFI produces sharper edges on fast-moving objects, effectively reducing the blurring and artifacts seen in competing architectures.
Conclusion
MA-VFI provides a robust solution for real-time video frame interpolation by combining direct intermediate flow estimation with a hierarchical feature extraction strategy. While the model currently relies on GPU power for optimal performance, its efficiency and accuracy make it a strong candidate for practical applications. Future work will focus on specialized architectures for multi-scale flow and extending the framework to broader video restoration tasks.
Summary updated