Digital video has traditionally been treated as a discrete stack of static images. We define a video by its frame rate—24, 30, or 60 frames per second—and represent it computationally as a rigid 3D tensor of pixels or a sequence of discrete latent states.
However, this discrete formulation imposes a severe fixed-interval constraint. Because standard architectures treat time as a sequence of integer indices, they struggle to generate content at arbitrary timestamps, adapt to varying frame rates post-training, or model fluid real-world motion. As noted by Chen Song et al. in E-CIR: Event-Enhanced Continuous Intensity Recovery, treating video as a sequence of discrete frames ignores the inherently continuous nature of light and motion, which often results in temporal jitter or ghosting artifacts.
To overcome these limitations, a major research direction has emerged: Continuous Temporal Modeling. Instead of predicting a fixed sequence of frames, these approaches treat video as a continuous signal over time, allowing users to query or generate the state of a scene at any arbitrary fractional timestamp.
The Midpoint Bottleneck in Video Interpolation
The limitations of discrete modeling are most visible in traditional Video Frame Interpolation (VFI). Many classic architectures are hard-coded to synthesize a single frame at the exact midpoint ($t = 0.5$) between two input frames. As Xianhang Cheng et al. point out in Video Frame Interpolation via Deformable Separable Convolution, this constraint prevents the direct synthesis of arbitrary timestamps for variable slow-motion effects.
To achieve higher frame rates (such as $4\times$ or $8\times$ upsampling), fixed-interval models must be applied recursively—first computing the frame at $t=0.5$, then using that result to compute frames at $t=0.25$ and $t=0.75$. Boyuan Jiang et al. demonstrate in Dynamic Frame Interpolation in Wavelet Domain and Yu-Lun Liu 0001 et al. show in Deep Video Frame Interpolation Using Cyclic Frame Generation that this recursive doubling leads to significant error accumulation over multiple steps.
Furthermore, Long Mai et al. argue in Revisiting Adaptive Convolutions for Video Frame Interpolation and Xiangyu Xu notes in Video Frame Interpolation Transformer that supporting variable frame rates requires a fundamental architectural shift: treating time not as a discrete step, but as a continuous input variable.
The challenge of temporal rigidity is historically rooted. Even in early video delivery, Wei Zhang observed in Seamless Bit-Stream Switching in Multirate-Based Video Streaming Systems that relying on preselected synchronization points prevented seamless transitions between bitrates at arbitrary frames. Modern continuous temporal modeling solves this across three main paradigms: Coordinate Networks, Neural Differential Equations, and Temporal Modulation.
Mechanism 1: Time as a Coordinate (Implicit Neural Representations)
The most direct way to achieve temporal continuity is to replace pixel grids with Implicit Neural Representations (INRs). Instead of storing a video as an array of pixels indexed by integers, an INR parameterizes the video as a neural network $f_\theta(x, y, t) = \text{RGB}$ or $f_\theta(t) = \text{Latent State}$.
Discrete Grid Paradigm:
[Frame 0] ---> [Frame 1] ---> [Frame 2] (Cannot query t = 0.54)
Continuous Coordinate Paradigm (INR):
Continuous t ───> [ Neural Network f_θ ] ───> Synthesized Frame at t
Because the time parameter $t$ is a continuous floating-point value, you can query the network at any arbitrary timestamp.
- Direct Coordinate Mapping: Sihyun Yu et al. leverage this in Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks, demonstrating that modeling videos as continuous signals bypasses the rigid 3D grid structure and eliminates temporal jitter.
- Unified Processing: Xi Ye introduced Neural Process Video Prediction in A unified model for continuous conditional video prediction, combining Neural Processes with INRs to handle future prediction, interpolation, and missing frame completion within a single continuous framework.
- Weight Synthesis: Taking this a step further, Yixuan Ren et al. introduce NeRV-Diffusion: Diffuse Implicit Neural Representation for Video Synthesis, where a diffusion model generates the weights of a small neural network representing the video, unlocking "for-free" temporal interpolation.
Mechanism 2: Motion as a Vector Field (Neural Differential Equations)
While INRs map coordinates directly to outputs, they can struggle to enforce physical motion constraints. A second paradigm models the dynamics of the video trajectory using Neural Ordinary or Stochastic Differential Equations (ODEs/SDEs) and Flow Matching.
Instead of predicting the next frame directly, the network models the derivative of the latent state over time:
$$\frac{dh(t)}{dt} = f_\theta(h(t), t)$$
To find the state of the video at any arbitrary time $t$, an ODE solver integrates this learned vector field from a starting state $h(t_0)$:
$$h(t) = h(t_0) + \int_{t_0}^t f_\theta(h(\tau), \tau) d\tau$$
Discrete Steps (e.g., RNN/GRU):
State_0 ───[Step 1]───> State_1 ───[Step 2]───> State_2
Continuous Flow (Neural ODE):
State_0 ───[ Integrate Vector Field along Smooth Curve ]───> State_t
This approach fundamentally changes how models handle temporal dynamics:
- Continuous Latent Flows: Cade Gordon et al. pioneered this in Latent Neural Differential Equations for Video Generation, replacing discrete recurrent networks with continuous flows to handle varying frame rates naturally.
- Stochastic Predictions: Xi Ye et al. expanded this to stochastic settings in STDiff: Spatio-Temporal Diffusion for Continuous Stochastic Video Prediction, utilizing Neural SDEs to predict diverse future frames at any arbitrary frame rate.
- Consistency and Efficiency: Abhinav Shrivastava et al. developed Efficient Continuous Video Flow Model for Video Prediction, which models the continuous flow between frames in latent space to eliminate temporal flickering without heavy attention mechanisms. Similarly, Yang Cao et al. combine Flow Matching with optimal polynomial projections in Video Latent Flow Matching: Optimal Polynomial Projections for Video Interpolation and Extrapolation to achieve smooth interpolation and long-range extrapolation.
- Dynamic Computation: This continuous formulation also benefits low-level motion estimation. The authors of Learning Optical Flow Field via Neural Ordinary Differential Equation replaced traditional fixed-step recurrent units with Neural ODEs, allowing the network to dynamically adjust its computation based on the complexity of the underlying pixel movement.
Mechanism 3: Temporal Modulation Blocks
If rebuilding an entire architecture around an INR or an ODE solver is computationally prohibitive, a third mechanism introduces feature-level Temporal Modulation.
Instead of changing the core representation of the video, these models inject a continuous time scalar $\Delta t$ directly into standard convolutional or transformer layers to dynamically shift the network's internal activations.
Frame A ───┐
├───> [ Temporal Modulation Block ] ───> Interpolated Frame at t
Frame B ───┤ ▲
│
Continuous Timestamp t ─────────┘
- Controllable Scaling: Gang Xu et al. demonstrate this in Temporal Modulation Network for Controllable Space-Time Video Super-Resolution. By using a specialized Temporal Modulation Block, their network can generate high-resolution frames at any arbitrary timestamp from standard convolutional backbones.
- The Scaling Challenge: Despite these advances, Zhicheng Geng et al. note in RSTT: Real-time Spatial Temporal Transformer for Space-Time Video Super-Resolution and Saikat Dutta et al. point out in Efficient Space-Time Video Super Resolution Using Low-Resolution Flow and Mask Upsampling that many unified space-time super-resolution models still struggle with temporal flexibility, making the design of arbitrary temporal scaling mechanisms a critical ongoing focus.
Why It Works: Aligning Representation with Physical Reality
Continuous temporal modeling works because it resolves a fundamental mismatch: physical motion is continuous, but discrete frame indices are not. This alignment provides distinct advantages across several domains:
1. Synergy with Neuromorphic Sensors
Event cameras do not capture frames; they record a continuous stream of microsecond-level intensity changes. Continuous temporal models are uniquely positioned to leverage this data. Yunfan LU et al. demonstrate in UniINR: Event-Guided Unified Rolling Shutter Correction, Deblurring, and Interpolation that a unified INR can fix rolling shutter distortions and interpolate frames simultaneously.
Similarly, Juanjuan Zhu et al. in Video Frame Prediction from a Single Image and Events and Ziyun Wang et al. in Event-based Continuous Color Video Decompression from Single Frames show that pairing continuous motion modeling with event streams allows for the reconstruction of high-frequency visual data at any arbitrary timestamp, bypassing the frame-rate limitations of traditional cameras.
2. Flexible Downstream Generation
When generation tasks involve complex physical constraints, discrete sequences become highly restrictive. For instance, Jingbo Wang et al. show in Task-Oriented Human-Object Interactions Generation with Implicit Neural Representations that fixed-framerate motion sequences make it difficult to adjust velocity or sample densely. By switching to an INR, motion density can be altered post-training.
In creative workflows, Guy Tevet et al. demonstrate in Flexible Motion In-betweening with Diffusion Models that continuous modeling allows animators to place keyframe constraints at arbitrary, non-uniform points in time rather than forcing them onto a rigid grid.
3. Resolving the Continuous-Discrete Tension
Even in high-level video understanding and action detection, the tension between continuous spaces and discrete indices persists. Zhenkun Gao et al. argue in TPRU: Advancing Temporal and Procedural Understanding in Large Multimodal Models that moving from static frame analysis to continuous video integration is essential for capturing fluid real-world actions.
Finally, Sauradip Nag et al. highlight in DiffTAD: Temporal Action Detection with Proposal Denoising Diffusion that bridging the gap between the continuous coordinate spaces used by generative diffusion models and the discrete nature of video frame indices is a core requirement for precisely locating action boundaries.
By treating time as a fluid, first-class dimension, continuous temporal modeling shifts video synthesis away from the constraints of the digital "flipbook" and closer to the continuous dynamics of the physical world.
Key Foundational Work
Video Frame Interpolation via Adaptive Convolution (2017) — contributed 1 concept elements to this direction.
Seamless Bit-Stream Switching in Multirate-Based Video Streaming Systems (2006) — contributed 1 concept elements to this direction.