IQ-VFI: Implicit Quadratic Motion Estimation for Video Frame Interpolation

Mengshun Hu, Kui Jiang, Zhihang Zhong, Zheng Wang, Yinqiang Zheng

Video Frame Interpolation (VFI) aims to synthesize intermediate frames between consecutive video frames, effectively increasing the frame rate. While standard approaches have achieved significant success, they often rely on a simplifying assumption: that motion between frames is linear. IQ-VFI challenges this paradigm by introducing an Implicit Quadratic motion estimation framework that captures the curvilinear trajectories of real-world objects.

The Problem: The Linear Motion Assumption

Most existing VFI algorithms assume pixels move in a straight line at a constant velocity. However, physical motion is rarely linear; objects accelerate, decelerate, and follow curved paths. When a linear model encounters such motion, it often produces "ghosting" artifacts or misplaces objects in the generated frame.

Linear vs. Quadratic Motion: (a) shows the standard linear assumption where motion follows a straight path. (b) illustrates the quadratic motion model used in IQ-VFI, which accounts for acceleration to reconstruct curved trajectories.

Modeling quadratic motion (which accounts for acceleration) typically requires more than two frames or ground-truth motion data, both of which are difficult to obtain in practical scenarios.

Methods: Implicit Quadratic Motion Estimation

The authors propose IQ-VFI, which models motion using a quadratic formula: $$f_{0t} = t \cdot f_{01} + \frac{a_\tau}{2} \cdot (t^2 - t)$$ In this equation, $a_\tau$ represents latent acceleration. Since only two input frames ($I_0, I_1$) are available at inference, the model must learn to predict this acceleration implicitly.

Knowledge Distillation via Teacher-Student Setup

To train the model without explicit ground-truth motion maps, IQ-VFI employs a novel distillation strategy.

Comparison of VFI paradigms and the proposed IQ-VFI method. Unlike linear (a), task-oriented (b), or two-stage (c) methods, IQ-VFI (d) uses a Teacher-Student distillation strategy where the teacher leverages the ground-truth intermediate frame to guide the student's implicit acceleration estimation.
  • The Teacher: During training, the teacher network has access to the ground-truth intermediate frame ($I_t$). With three points ($I_0, I_t, I_1$), the teacher can accurately determine the curved trajectory and acceleration.
  • The Student: The student network only sees the endpoint frames ($I_0, I_1$). It is trained to mimic the teacher's predicted acceleration and motion cues, effectively learning a "prior" for how objects typically move and curve.

Implicit Motion Modulation (IMM)

The predicted acceleration is treated as a latent feature map rather than a single scalar. The Implicit Motion Modulation (IMM) module uses these features to "bend" initial linear flow estimates into quadratic flows in a coarse-to-fine manner, refining the final output.

Key Results and Benchmarks

IQ-VFI was evaluated against state-of-the-art (SOTA) methods on several standard benchmarks, including Vimeo90K, UCF101, SNU-FILM, and Xiph.

  • Vimeo90K: Achieved a PSNR of 36.60 dB, outperforming recent high-performers like AMT (36.53) and RIFE (35.65).
  • SNU-FILM (Extreme): On the most challenging category involving large displacements, IQ-VFI reached 25.45 dB, demonstrating superior robustness to complex motion compared to linear models.
  • High Resolution: On Xiph 4K, it reached 34.72 dB, proving its scalability to high-resolution video.

Qualitative results confirm that IQ-VFI correctly places fast-moving, curving objects—such as the wheels of a turning vehicle or a swinging pendulum—where linear methods often fail.

Conclusion

IQ-VFI demonstrates that explicit physical modeling isn't always necessary for complex motion; instead, acceleration can be treated as a latent property learned through task-oriented distillation. By bridging the gap between simple linear models and complex real-world dynamics, IQ-VFI sets a new standard for high-fidelity video frame interpolation.

Summary updated

Abstract

Advanced video frame interpolation (VFI) algorithms approximate intermediate motions between two input frames to synthesize intermediate frame. However, they struggle to handle complex scenarios with curvilinear motions since they overlook the latent acceleration information between the input frames. Moreover, the supervision of predicted motions is tricky because ground-truth motions are not available. To this end, we propose a novel frame-work for implicit quadratic video frame interpolation (IQ-VFI), which explores latent acceleration information and accurate intermediate motions via knowledge distillation. Specifically, the proposed IQ-VFI consists of an implicit acceleration estimation network (IANet) and a VFI back-bone, the former fully leverages spatio-temporal information to explore latent acceleration priors between two input frames, which is then used to progressively modulate linear motions from the latter into quadratic motions in coarse-to-fine manner. Furthermore, to encourage both components to distill more acceleration and motion cues oriented towards VFI, we propose a knowledge distillation strategy in which implicit acceleration distillation loss and implicit motion distillation loss are employed to adaptively guide latent acceleration priors and intermediate motions learning, respectively. Extensive experiments show that our proposed IQ-VFI can achieve state-of-the-art performances on various benchmark datasets.
BibTeX
@inproceedings{hu2024iq,
  author = {Mengshun Hu and Kui Jiang and Zhihang Zhong and Zheng Wang and Yinqiang Zheng},
  title = {{IQ-VFI: Implicit Quadratic Motion Estimation for Video Frame Interpolation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00613},
}
Figure