Motion-Aware Video Frame Interpolation

Pengfei Han, Fuhua Zhang, Bin Zhao, Xuelong Li

TL;DR

Motion-Aware Video Frame Interpolation (MA-VFI) is a new lightweight network that achieves high-quality video frame up-sampling by directly estimating intermediate optical flow. By combining a hierarchical pyramid feature extractor with a cross-scale refinement strategy and a specialized flow-directed loss, it strikes an impressive balance between real-time performance and state-of-the-art visual accuracy.

What Problem Does This Paper Address?

Video Frame Interpolation (VFI) is the task of generating "in-between" frames to make videos smoother (e.g., converting 30fps to 60fps). While deep learning has advanced VFI, two major hurdles remain:

  1. Complex Motion: Most methods assume motion is linear between frames. In reality, objects accelerate, rotate, or get occluded, leading to "ghosting" artifacts or blurring.
  2. Efficiency vs. Quality: High-quality methods often rely on heavy optical flow networks or cost volumes that are too slow for real-time applications. Conversely, fast methods often sacrifice fine spatial details.

The authors propose MA-VFI to handle non-linear motion more effectively while remaining efficient enough for practical use.

Methods

The MA-VFI architecture consists of three core components:

  • Hierarchical Pyramid Feature Extraction: Instead of a simple encoder, the model uses a four-layer pyramid. It employs different receptive fields (using $7\times7$ and $3\times3$ filters) to capture both global semantic context (large-scale motion) and fine spatial details (small-object motion) simultaneously.
  • Cross-Scale Motion Structure: This is a top-down refinement module. It estimates flow at a coarse scale and then progressively refines it at finer scales. Crucially, it uses warped features from previous levels to align the current level's features, allowing the network to "interact" with the motion it is predicting.
  • Intermediate Flow-Directed Loss: To guide the network without requiring ground-truth optical flow (which is rarely available for real-world video), the authors use a pre-trained LiteFlownet as a "teacher." During training, the student network is penalized if its estimated flow deviates from the teacher's, significantly improving the precision of the intervening flow maps.

Datasets

The model was trained on the Vimeo-90k dataset (91,701 triplets). Evaluation was conducted across several standard benchmarks:

  • UCF101: For testing single-frame interpolation on human actions.
  • Vimeo-90k (Test set): For general motion and texture evaluation.
  • Middlebury: A classic benchmark for optical flow and interpolation accuracy.
  • Adobe-240fps: Used specifically for multi-frame interpolation (generating 3 or 5 frames between two inputs) to test the model's consistency over time.

Key Insights and Ideas

  • Direct Estimation is Better: Rather than calculating flow from $T_0 \rightarrow T_1$ and then trying to approximate the middle, MA-VFI directly estimates the flow to the intermediate time $t$. This bypasses the errors introduced by assuming linear motion.
  • Feature-Flow Symbiosis: The "aha moment" in the cross-scale structure is that flow shouldn't just move pixels; it should help the network align features. By warping features at each scale before the next refinement step, the model handles large displacements much more gracefully.
  • Knowledge Distillation for Flow: Using a heavy, pre-trained flow network (LiteFlownet) only during the loss calculation allows the final MA-VFI model to remain lightweight and fast during inference while still benefiting from the teacher's "expertise."

Key Results

MA-VFI demonstrates a superior "speed-to-accuracy" ratio compared to 31 other state-of-the-art (SOTA) methods:

  • Accuracy: On the UCF101 dataset, it achieved a PSNR of 35.31 dB, outperforming popular models like RIFE (35.28 dB) and DAIN (35.00 dB).
  • Speed: It processes a $448\times256$ resolution frame in just 0.021 seconds (approx. 47 FPS), making it suitable for real-time applications.
  • Efficiency: With 17.87M parameters, it is significantly more compact than heavyweights like CAIN (42.78M) while producing better visual results.
  • Qualitative: Visual results show much sharper edges on fast-moving objects (like a hula hoop or a moving car) where other methods produce significant blur.

Limitations and Future Work

Despite its strengths, the authors note:

  • GPU Dependency: Like most VFI models, it still relies heavily on GPU power. Deploying this on low-power mobile CPUs remains a challenge.
  • SOTA Gaps: While it beats most models, some specialized architectures like ProBoost-Net still achieve higher PSNR on specific datasets (like Vimeo-90k), albeit at much slower speeds.
  • Future Directions: The authors plan to explore conceptualizing a separate network specifically for handling different scales of optical flow and extending this architecture to more complex video restoration tasks.

Summary updated

Abstract

Highlights•MA-VFI directly estimates the intermediate optical flow diagram between adjacent frames, alleviating the complex modeling and computational costs of existing methods.•MA-VFI employs cross-scale motion intermediate flow estimation model to enhance the interaction between features and flow maps during the interpolation process, thus improving the representation of nonlinear motion in consecutive frames.•The intermediate flow directional loss has been designed to precisely guide the estimation of intermediate flow, thereby assisting the model in alleviating issues such as image blurring and spurious artifacts.
BibTeX
@inproceedings{han2024motion,
  author = {Pengfei Han and Fuhua Zhang and Bin Zhao and Xuelong Li},
  title = {{Motion-Aware Video Frame Interpolation}},
  booktitle = {Neural Networks},
  year = {2024},
  url = {https://openreview.net/forum?id=5mLBBnuA6l},
}

Loading related content...

Figure