Range-nullspace Video Frame Interpolation with Focalized Motion Estimation

Zhiyang Yu, Yu Zhang, Dongqing Zou, Xijun Chen, Jimmy S. Ren, Shunqing RenCVPR 2023 · OpenReview

Abstract

Continuous-time video frame interpolation (VFI) is a fundamental technique in computer vision for its flexibility in synthesizing motion trajectories and novel video frames at arbitrary intermediate time steps. Yet, how to infer accurate intermediate motion and synthesize high-quality video frames are two critical challenges. In this paper, we present a novel VFI framework with improved treatment for these challenges. To address the former, we propose focalized trajectory fitting, which performs confidence-aware motion trajectory estimation by learning to pay focus to reliable optical flow candidates while suppressing the outliers. The second is range-nullspace synthesis, a novel frame renderer cast as solving an ill-posed problem addressed by learning decoupled components in orthogonal subspaces. The proposed framework sets new records on 7 of 10 public VFI benchmarks.

Introduction

Video Frame Interpolation (VFI) is the process of generating intermediate frames between existing ones to create smooth slow-motion or increase a video's frame rate. "Continuous-time" VFI is particularly challenging because it requires the model to synthesize a frame at any arbitrary point in time ($t$), not just the midpoint. Current state-of-the-art methods often struggle with two primary issues: motion errors caused by unreliable optical flow and synthesis artifacts in complex regions like occlusions.

Comparison of the conventional VFI pipeline and the proposed pipeline featuring Focalized Motion Estimation and a Deep Range-Nullspace Solver.

Focalized Motion Estimation (FME)

Most VFI methods fit a mathematical curve, such as a parabola, to motion vectors (optical flow). However, if a single flow estimate is incorrect—common during sudden occlusions or with fast-moving objects—it can "pull" the entire trajectory away from reality, leading to blurry results.

To address this, we introduce Focalized Trajectory Fitting (FTF). Instead of treating all optical flow vectors as equally important, the model learns to estimate confidence weights by analyzing the "warping residual." The system uses these weights to perform a weighted least-squares fit of a 2nd-order Taylor polynomial. By "focalizing" on high-confidence flows and suppressing outliers, the motion estimation remains accurate even near tricky occlusion boundaries.

Range-Nullspace Synthesis (RNS)

Once motion is estimated, the frame must be rendered. We treat frame synthesis as a physics-guided inverse problem, decomposing the target frame into two orthogonal components:

  • The Range Space: This represents the parts of the image that can be explained by warping the input frames, adhering to the physical motion of the scene.
  • The Nullspace: This represents the parts that cannot be explained by warping, such as content that was hidden in the original frames.

By learning these components separately in orthogonal subspaces, the network avoids "entangled" representations. One part of the network focuses on correcting warping errors, while another focuses on inpainting missing content. This physics-guided decoupling moves away from traditional "black-box" rendering, ensuring the synthesis process respects the underlying motion logic.

Experimental Results

The proposed framework was evaluated on a wide variety of benchmarks, including GoPro, X4K1000FPS, and Vimeo-90k. It set new state-of-the-art (SOTA) records on 7 of 10 public VFI benchmarks. Notably, on the GoPro benchmark, it achieved a PSNR of 32.31 dB, outperforming the previous leader by 0.58 dB. On the X4K1000FPS dataset, it achieved 31.97 dB, a significant 0.87 dB improvement over previous methods.

Qualitative comparison of the proposed method against existing VFI techniques (EQVI, M2M, RIFE, DBVI) across various challenging scenes.

Qualitative results demonstrate much sharper edges around moving objects, such as a hula hoop or a spinning wheel, where other models typically produce ghosting or smearing artifacts. Despite its mathematical complexity, the model is remarkably efficient, using only ~10 million parameters—significantly fewer than many competitors like FLAVR (42M) or DBVI (15M).

Conclusion

By rethinking motion estimation through focalized fitting and frame synthesis via range-nullspace decomposition, our framework provides a robust and efficient solution for continuous-time VFI. The combination of confidence-aware trajectory fitting and physics-guided synthesis allows for high-quality frame generation that sets new standards for the field.

Summary updated

Abstract

Continuous-time video frame interpolation is a fundamental technique in computer vision for its flexibility in synthesizing motion trajectories and novel video frames at arbitrary intermediate time steps. Yet, how to infer accurate intermediate motion and synthesize high-quality video frames are two critical challenges. In this paper, we present a novel VFI framework with improved treatment for these challenges. To address the former, we propose focalized trajectory fitting, which performs confidence-aware motion trajectory estimation by learning to pay focus to reliable optical flow candidates while suppressing the outliers. The second is range-nullspace synthesis, a novel frame renderer cast as solving an ill-posed problem addressed by learning decoupled components in orthogonal subspaces. The proposed framework sets new records on 7 of 10 public VFI benchmarks.
BibTeX
@inproceedings{yu2023range,
  author = {Zhiyang Yu and Yu Zhang and Dongqing Zou and Xijun Chen and Jimmy S. Ren and Shunqing Ren},
  title = {{Range-nullspace Video Frame Interpolation with Focalized Motion Estimation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=eu6CcJNpDt},
}
Figure