Video frame interpolation (VFI) is the art of synthesizing missing frames between captured images to create smooth, high-frame-rate slow-motion video. For years, the foundational building block of VFI algorithms was the linear motion assumption (or constant velocity constraint). This assumption presumes that between any two consecutive frames, pixels travel along straight lines at a perfectly uniform speed.
While mathematically convenient, the real world rarely conforms to linear physics. Objects accelerate, decelerate, spin, bounce, and curve. When an algorithm constrained by a linear motion prior encounters these complex dynamics, it suffers from a fundamental mismatch between its internal model and physical reality. The result is a suite of distinct visual artifacts: ghosting, haloing, and unnatural blurring.
To overcome these limitations, recent research has shifted toward modeling the true, non-linear trajectories of moving scenes. This transition relies on three core strategies: higher-order polynomial modeling, continuous event-driven tracking, and implicit geometric representations.
Accounting for Acceleration: Quadratic Motion Models
The simplest extension of a linear model is a quadratic one. Mathematically, a linear model ($x(t) = vt + x_0$) only requires a single velocity vector to map a pixel's position over time. However, to capture acceleration ($x(t) = at^2 + bt + c$), an algorithm must solve for an additional unknown variable: the acceleration vector $a$.
To extract this acceleration from standard video, models require more temporal context. Xiangyu Xu et al. address this in Quadratic Video Interpolation by expanding the historical window to four consecutive frames. By analyzing the velocity shift between frames 1-2 and frames 3-4, the network can compute a reliable acceleration map, drastically reducing artifacts in scenes featuring complex physics. The efficacy of this approach was widely recognized during the AIM 2020 Challenge on Video Temporal Super-Resolution, where Li Siyao et al. noted that quadratic models had become the state-of-the-art standard for resolving non-linear dynamics like camera shake.
Linear (2 Frames): [Frame 1] -------------> [Frame 2] (Constant Speed)
Quadratic (4 Frames): [Frame 1] -> [Frame 2] ---> [Frame 3] -----> [Frame 4] (Captures Acceleration)
When extra frames are unavailable, networks must find alternative ways to infer acceleration. Kui Jiang et al. introduce IQ-VFI, which uses a teacher-student distillation strategy to estimate implicit quadratic motion from just two input frames. For highly irregular trajectories, Chenming Xu proposes JNMR, modeling motion as a temporal concatenation of multi-stage quadratic models to handle sudden, erratic changes.
Because not all parts of a video contain non-linear motion, computing complex trajectories everywhere can be computationally wasteful. To balance efficiency and accuracy, Saikat Dutta et al. design a 3D space-time convolution network in Non-linear Motion Estimation for Video Frame Interpolation using Space-time Convolutions that adaptively switches between linear and quadratic models depending on the local pixel dynamics.
Filling the Blind Time with Event Cameras
Even high-order polynomials are ultimately approximations over a discrete "blind time" interval. If an object undergoes an abrupt change in direction—such as a tennis ball hitting a racket—between two exposures, discrete frames cannot capture the exact moment of impact. As Jiaben Chen et al. observe in Revisiting Event-based Video Frame Interpolation, standard camera sensors fail to record the continuous trajectory of high-speed objects.
Event-based sensors resolve this by operating asynchronously, recording continuous log-intensity brightness changes (events) at microsecond resolution between standard RGB exposures.
Standard RGB: [Frame 0] ------------------ BLIND TIME ------------------ [Frame 1]
Event Camera: [Frame 0] --e-e--e---e-e-e--e--e---e-e--e-e---e--e--e-e--> [Frame 1]
Daniel Gehrig et al. leverage this paradigm in Time Lens: Event-Based Video Frame Interpolation, demonstrating that the continuous event stream effectively bridges the blind time, allowing the network to reconstruct complex, non-linear movements like rapid rotations or sudden deformations. To map these trajectories precisely, Haoyue Liu et al. introduce TimeTracker, which uses event streams to perform continuous point tracking across non-linear paths.
Interestingly, event data can also simplify the interpolation architecture itself. Instead of fitting complex, high-order curves to represent large displacements, Yongrui Ma et al. demonstrate in TimeLens-XL: Real-Time Event-Based Video Frame Interpolation with Large Motion that large, non-linear motions can be recursively decomposed into a sequence of tiny, locally linear intervals. By leveraging the ultra-high temporal density of events, the model achieves state-of-the-art accuracy while maintaining real-time processing speeds.
Implicit Fields and Geometric Surfaces
A third paradigm avoids the limitations of the linear motion assumption by rethinking how motion is represented. Traditional VFI relies on explicit 2D optical flow vectors, which struggle to maintain coherence when pixel displacements are large or unpredictable.
Instead of treating frames as flat 2D grids, Zaoming Yan et al. propose a geometric approach in Surface-Aware Feed-Forward Quadratic Gaussian for Frame Interpolation with Large Motion. They treat video frames as 3D differential surfaces mapped to Quadratic Gaussians. By aligning these surfaces using structural properties like curvature and normals, the framework naturally accommodates curvilinear paths that cause traditional pixel-matching algorithms to fail.
Similarly, coordinate-based neural networks can model motion implicitly. Zujin Guo et al. introduce Generalizable Implicit Motion Modeling for Video Frame Interpolation, which maps space-time coordinates to a continuous "motion latent." This allows the network to learn complex, continuous dynamics without being bound to rigid kinematic equations.
Resolving Time-to-Location Ambiguity
When motion is non-uniform, algorithms face a severe "time-to-location" ambiguity. In a uniform motion model, knowing the target frame's timestamp ($t=0.5$) tells you exactly where a pixel should be (exactly halfway along the flow vector). In a non-uniform model, a pixel might accelerate early in the interval and decelerate later, meaning its position at $t=0.5$ is not simply halfway.
To resolve this ambiguity, Jihyong Oh et al. develop BiM-VFI: Bidirectional Motion Field-Guided Frame Interpolation for Video with Non-uniform Motions. They introduce a bidirectional motion descriptor that encodes both magnitude and angular information, providing the network with the precise directional changes needed to map non-linear trajectories accurately.
Finally, when motion is highly stylized or involves extreme dis-occlusions—such as in hand-drawn animation—traditional optical flow and pixel-warping strategies break down entirely. In ToonCrafter: Generative Cartoon Interpolation, Jinbo Xing et al. abandon explicit motion tracking altogether. They frame interpolation as a generative process, adapting live-action video diffusion models to synthesize the exaggerated, non-linear physics characteristic of cartoons, ensuring visual coherence even across massive temporal gaps.
Key Foundational Work
Real-time detection and tracking of multiple objects with partial decoding in H.264/AVC bitstream domain (2009) — contributed 1 concept elements to this direction.
Capturing the Uncertainty of Moving-Object Representations (1999) — contributed 1 concept elements to this direction.