Abstract
Continuous-time video frame interpolation (VFI) is a fundamental technique in computer vision for its flexibility in synthesizing motion trajectories and novel video frames at arbitrary intermediate time steps. Yet, how to infer accurate intermediate motion and synthesize high-quality video frames are two critical challenges. In this paper, we present a novel VFI framework with improved treatment for these challenges. To address the former, we propose focalized trajectory fitting, which performs confidence-aware motion trajectory estimation by learning to pay focus to reliable optical flow candidates while suppressing the outliers. The second is range-nullspace synthesis, a novel frame renderer cast as solving an ill-posed problem addressed by learning decoupled components in orthogonal subspaces. The proposed framework sets new records on 7 of 10 public VFI benchmarks.
Introduction
Video Frame Interpolation (VFI) is the process of generating intermediate frames between existing ones to create smooth slow-motion or increase a video's frame rate. "Continuous-time" VFI is particularly challenging because it requires the model to synthesize a frame at any arbitrary point in time ($t$), not just the midpoint. Current state-of-the-art methods often struggle with two primary issues: motion errors caused by unreliable optical flow and synthesis artifacts in complex regions like occlusions.
Focalized Motion Estimation (FME)
Most VFI methods fit a mathematical curve, such as a parabola, to motion vectors (optical flow). However, if a single flow estimate is incorrect—common during sudden occlusions or with fast-moving objects—it can "pull" the entire trajectory away from reality, leading to blurry results.
To address this, we introduce Focalized Trajectory Fitting (FTF). Instead of treating all optical flow vectors as equally important, the model learns to estimate confidence weights by analyzing the "warping residual." The system uses these weights to perform a weighted least-squares fit of a 2nd-order Taylor polynomial. By "focalizing" on high-confidence flows and suppressing outliers, the motion estimation remains accurate even near tricky occlusion boundaries.
Range-Nullspace Synthesis (RNS)
Once motion is estimated, the frame must be rendered. We treat frame synthesis as a physics-guided inverse problem, decomposing the target frame into two orthogonal components:
- The Range Space: This represents the parts of the image that can be explained by warping the input frames, adhering to the physical motion of the scene.
- The Nullspace: This represents the parts that cannot be explained by warping, such as content that was hidden in the original frames.
By learning these components separately in orthogonal subspaces, the network avoids "entangled" representations. One part of the network focuses on correcting warping errors, while another focuses on inpainting missing content. This physics-guided decoupling moves away from traditional "black-box" rendering, ensuring the synthesis process respects the underlying motion logic.
Experimental Results
The proposed framework was evaluated on a wide variety of benchmarks, including GoPro, X4K1000FPS, and Vimeo-90k. It set new state-of-the-art (SOTA) records on 7 of 10 public VFI benchmarks. Notably, on the GoPro benchmark, it achieved a PSNR of 32.31 dB, outperforming the previous leader by 0.58 dB. On the X4K1000FPS dataset, it achieved 31.97 dB, a significant 0.87 dB improvement over previous methods.
Qualitative results demonstrate much sharper edges around moving objects, such as a hula hoop or a spinning wheel, where other models typically produce ghosting or smearing artifacts. Despite its mathematical complexity, the model is remarkably efficient, using only ~10 million parameters—significantly fewer than many competitors like FLAVR (42M) or DBVI (15M).
Conclusion
By rethinking motion estimation through focalized fitting and frame synthesis via range-nullspace decomposition, our framework provides a robust and efficient solution for continuous-time VFI. The combination of confidence-aware trajectory fitting and physics-guided synthesis allows for high-quality frame generation that sets new standards for the field.
Summary updated