Video Frame Interpolation (VFI) aims to synthesize intermediate frames between consecutive video frames, effectively increasing the frame rate. While standard approaches have achieved significant success, they often rely on a simplifying assumption: that motion between frames is linear. IQ-VFI challenges this paradigm by introducing an Implicit Quadratic motion estimation framework that captures the curvilinear trajectories of real-world objects.
The Problem: The Linear Motion Assumption
Most existing VFI algorithms assume pixels move in a straight line at a constant velocity. However, physical motion is rarely linear; objects accelerate, decelerate, and follow curved paths. When a linear model encounters such motion, it often produces "ghosting" artifacts or misplaces objects in the generated frame.
Modeling quadratic motion (which accounts for acceleration) typically requires more than two frames or ground-truth motion data, both of which are difficult to obtain in practical scenarios.
Methods: Implicit Quadratic Motion Estimation
The authors propose IQ-VFI, which models motion using a quadratic formula: $$f_{0t} = t \cdot f_{01} + \frac{a_\tau}{2} \cdot (t^2 - t)$$ In this equation, $a_\tau$ represents latent acceleration. Since only two input frames ($I_0, I_1$) are available at inference, the model must learn to predict this acceleration implicitly.
Knowledge Distillation via Teacher-Student Setup
To train the model without explicit ground-truth motion maps, IQ-VFI employs a novel distillation strategy.
- The Teacher: During training, the teacher network has access to the ground-truth intermediate frame ($I_t$). With three points ($I_0, I_t, I_1$), the teacher can accurately determine the curved trajectory and acceleration.
- The Student: The student network only sees the endpoint frames ($I_0, I_1$). It is trained to mimic the teacher's predicted acceleration and motion cues, effectively learning a "prior" for how objects typically move and curve.
Implicit Motion Modulation (IMM)
The predicted acceleration is treated as a latent feature map rather than a single scalar. The Implicit Motion Modulation (IMM) module uses these features to "bend" initial linear flow estimates into quadratic flows in a coarse-to-fine manner, refining the final output.
Key Results and Benchmarks
IQ-VFI was evaluated against state-of-the-art (SOTA) methods on several standard benchmarks, including Vimeo90K, UCF101, SNU-FILM, and Xiph.
- Vimeo90K: Achieved a PSNR of 36.60 dB, outperforming recent high-performers like AMT (36.53) and RIFE (35.65).
- SNU-FILM (Extreme): On the most challenging category involving large displacements, IQ-VFI reached 25.45 dB, demonstrating superior robustness to complex motion compared to linear models.
- High Resolution: On Xiph 4K, it reached 34.72 dB, proving its scalability to high-resolution video.
Qualitative results confirm that IQ-VFI correctly places fast-moving, curving objects—such as the wheels of a turning vehicle or a swinging pendulum—where linear methods often fail.
Conclusion
IQ-VFI demonstrates that explicit physical modeling isn't always necessary for complex motion; instead, acceleration can be treated as a latent property learned through task-oriented distillation. By bridging the gap between simple linear models and complex real-world dynamics, IQ-VFI sets a new standard for high-fidelity video frame interpolation.
Summary updated