Generalizable Implicit Motion Modeling for Video Frame Interpolation

Zujin Guo, Wei Li, Chen Change Loy

Abstract

Motion modeling is critical in flow-based Video Frame Interpolation (VFI). Existing paradigms either consider linear combinations of bidirectional flows or directly predict bilateral flows for given timestamps without exploring favorable motion priors, thus lacking the capability of effectively modeling spatiotemporal dynamics in real-world videos. To address this limitation, in this study, we introduce Generalizable Implicit Motion Modeling (GIMM), a novel and effective approach to motion modeling for VFI. Specifically, to enable GIMM as an effective motion modeling paradigm, we design a motion encoding pipeline to model spatiotemporal motion latent from bidirectional flows extracted from pre-trained flow estimators, effectively representing input-specific motion priors. Then, we implicitly predict arbitrary-timestep optical flows within two adjacent input frames via an adaptive coordinate-based neural network, with spatiotemporal coordinates and motion latent as inputs. Our GIMM can be easily integrated with existing flow-based VFI works by supplying accurately modeled motion. We show that GIMM performs better than the current state of the art on standard VFI benchmarks.

The Challenge of Motion Modeling in VFI

Video Frame Interpolation (VFI) aims to synthesize intermediate frames between existing ones to increase video framerate. Most modern VFI methods rely on optical flow to estimate pixel movement. However, traditional motion modeling paradigms often suffer from two major limitations:

  1. The Linear Assumption: Many models assume pixels move at constant speeds along straight lines. This assumption fails for complex, non-linear motions like rotations or rapid accelerations.
  2. Discrete vs. Continuous Modeling: Most models are optimized for specific intervals (e.g., $t=0.5$). When asked to generate frames at arbitrary timestamps (e.g., $t=0.37$), they often struggle to maintain temporal consistency.

While Implicit Neural Representations (INRs) offer a solution for continuous signal modeling, they typically require per-instance optimization. GIMM overcomes this by introducing a generalizable INR that works across different video contexts without retraining.

Generalizable Implicit Motion Modeling (GIMM)

GIMM introduces a three-stage pipeline to model motion implicitly by encoding motion priors into a latent space. This allows for accurate optical flow prediction at any arbitrary timestamp between two frames.

The GIMM architecture: from bidirectional flows to a continuous bilateral flow prediction via a coordinate-based MLP.

As shown in the architecture above, the process involves:

  1. Motion Encoding: Extracting spatial features from bidirectional flows (e.g., from RAFT) and normalizing them.
  2. Spatiotemporal Latent Refinement: Using forward warping to shift features to their expected positions at time $t$, followed by a Latent Refiner to create a "motion latent" ($L_t$).
  3. Implicit Prediction: A coordinate-based SIREN MLP takes the 3D coordinates $(x, y, t)$ and the motion latent $L_t$ to predict precise bilateral optical flows ($F_{t\to0}, F_{t\to1}$).

Experimental Results and Evaluation

The authors evaluated GIMM across several benchmarks, including Vimeo90K and a newly adapted arbitrary-time version of SNU-FILM (SNU-FILM-arb). GIMM sets a new state-of-the-art for arbitrary-timestep VFI, achieving a PSNR of 37.56 dB on the Vimeo-Triplet-Flow benchmark, significantly outperforming standard linear modeling (35.03 dB).

Qualitative results on SNU-FILM-arb-Hard, showing GIMM's superior ability to reconstruct complex movements compared to baselines.

Qualitative analysis demonstrates that GIMM preserves sharp edges and correct textures in challenging scenarios, such as the fast-moving legs of a runner or a dog catching a ball. In these cases, traditional methods like BMBC or linear interpolation often produce significant blurring or ghosting artifacts.

Key Insights and Future Directions

  • Efficiency: The coordinate-based MLP in GIMM is remarkably lightweight, using only 0.25M parameters compared to 4.27M for a U-Net alternative, while delivering superior accuracy.
  • Continuous Transitions: By using $t$ as a direct input to the MLP, the model naturally learns smooth motion transitions, which is vital for high-quality slow-motion synthesis.

Despite its success, GIMM's performance is tied to the quality of the initial bidirectional flows from the pre-trained estimator. Furthermore, the current two-frame input limit may restrict its ability to model highly complex, multi-frame non-linear trajectories. Future work may explore extending this implicit modeling to longer temporal sequences to further enhance motion accuracy.

Summary updated

Abstract

Motion modeling is critical in flow-based Video Frame Interpolation (VFI). Existing paradigms either consider linear combinations of bidirectional flows or directly predict bilateral flows for given timestamps without exploring favorable motion priors, thus lacking the capability of effectively modeling spatiotemporal dynamics in real-world videos. To address this limitation, in this study, we introduce Generalizable Implicit Motion Modeling (GIMM), a novel and effective approach to motion modeling for VFI. Specifically, to enable GIMM as an effective motion modeling paradigm, we design a motion encoding pipeline to model spatiotemporal motion latent from bidirectional flows extracted from pre-trained flow estimators, effectively representing input-specific motion priors. Then, we implicitly predict arbitrary-timestep optical flows within two adjacent input frames via an adaptive coordinate-based neural network, with spatiotemporal coordinates and motion latent as inputs. Our GIMM can be easily integrated with existing flow-based VFI works by supplying accurately modeled motion. We show that GIMM performs better than the current state of the art on standard VFI benchmarks.
BibTeX
@inproceedings{guo2024generalizable,
  author = {Zujin Guo and Wei Li and Chen Change Loy},
  title = {{Generalizable Implicit Motion Modeling for Video Frame Interpolation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=ZlpJLQsr2v},
}
Figure