Continuous Temporal Modeling for Video Synthesis

37 concepts35 papers2 proposals

Digital video has traditionally been treated as a discrete stack of static images. We define a video by its frame rate—24, 30, or 60 frames per second—and represent it computationally as a rigid 3D tensor of pixels or a sequence of discrete latent states.

However, this discrete formulation imposes a severe fixed-interval constraint. Because standard architectures treat time as a sequence of integer indices, they struggle to generate content at arbitrary timestamps, adapt to varying frame rates post-training, or model fluid real-world motion. As noted by Chen Song et al. in E-CIR: Event-Enhanced Continuous Intensity Recovery, treating video as a sequence of discrete frames ignores the inherently continuous nature of light and motion, which often results in temporal jitter or ghosting artifacts.

To overcome these limitations, a major research direction has emerged: Continuous Temporal Modeling. Instead of predicting a fixed sequence of frames, these approaches treat video as a continuous signal over time, allowing users to query or generate the state of a scene at any arbitrary fractional timestamp.

The Midpoint Bottleneck in Video Interpolation

The limitations of discrete modeling are most visible in traditional Video Frame Interpolation (VFI). Many classic architectures are hard-coded to synthesize a single frame at the exact midpoint ($t = 0.5$) between two input frames. As Xianhang Cheng et al. point out in Video Frame Interpolation via Deformable Separable Convolution, this constraint prevents the direct synthesis of arbitrary timestamps for variable slow-motion effects.

To achieve higher frame rates (such as $4\times$ or $8\times$ upsampling), fixed-interval models must be applied recursively—first computing the frame at $t=0.5$, then using that result to compute frames at $t=0.25$ and $t=0.75$. Boyuan Jiang et al. demonstrate in Dynamic Frame Interpolation in Wavelet Domain and Yu-Lun Liu 0001 et al. show in Deep Video Frame Interpolation Using Cyclic Frame Generation that this recursive doubling leads to significant error accumulation over multiple steps.

Furthermore, Long Mai et al. argue in Revisiting Adaptive Convolutions for Video Frame Interpolation and Xiangyu Xu notes in Video Frame Interpolation Transformer that supporting variable frame rates requires a fundamental architectural shift: treating time not as a discrete step, but as a continuous input variable.

The challenge of temporal rigidity is historically rooted. Even in early video delivery, Wei Zhang observed in Seamless Bit-Stream Switching in Multirate-Based Video Streaming Systems that relying on preselected synchronization points prevented seamless transitions between bitrates at arbitrary frames. Modern continuous temporal modeling solves this across three main paradigms: Coordinate Networks, Neural Differential Equations, and Temporal Modulation.


Mechanism 1: Time as a Coordinate (Implicit Neural Representations)

The most direct way to achieve temporal continuity is to replace pixel grids with Implicit Neural Representations (INRs). Instead of storing a video as an array of pixels indexed by integers, an INR parameterizes the video as a neural network $f_\theta(x, y, t) = \text{RGB}$ or $f_\theta(t) = \text{Latent State}$.

Discrete Grid Paradigm:
[Frame 0] ---> [Frame 1] ---> [Frame 2]   (Cannot query t = 0.54)

Continuous Coordinate Paradigm (INR):
Continuous t ───> [ Neural Network f_θ ] ───> Synthesized Frame at t

Because the time parameter $t$ is a continuous floating-point value, you can query the network at any arbitrary timestamp.


Mechanism 2: Motion as a Vector Field (Neural Differential Equations)

While INRs map coordinates directly to outputs, they can struggle to enforce physical motion constraints. A second paradigm models the dynamics of the video trajectory using Neural Ordinary or Stochastic Differential Equations (ODEs/SDEs) and Flow Matching.

Instead of predicting the next frame directly, the network models the derivative of the latent state over time:

$$\frac{dh(t)}{dt} = f_\theta(h(t), t)$$

To find the state of the video at any arbitrary time $t$, an ODE solver integrates this learned vector field from a starting state $h(t_0)$:

$$h(t) = h(t_0) + \int_{t_0}^t f_\theta(h(\tau), \tau) d\tau$$

Discrete Steps (e.g., RNN/GRU):
State_0 ───[Step 1]───> State_1 ───[Step 2]───> State_2

Continuous Flow (Neural ODE):
State_0 ───[ Integrate Vector Field along Smooth Curve ]───> State_t

This approach fundamentally changes how models handle temporal dynamics:


Mechanism 3: Temporal Modulation Blocks

If rebuilding an entire architecture around an INR or an ODE solver is computationally prohibitive, a third mechanism introduces feature-level Temporal Modulation.

Instead of changing the core representation of the video, these models inject a continuous time scalar $\Delta t$ directly into standard convolutional or transformer layers to dynamically shift the network's internal activations.

Frame A ───┐
           ├───> [ Temporal Modulation Block ] ───> Interpolated Frame at t
Frame B ───┤                    ▲
                                │
Continuous Timestamp t ─────────┘

Why It Works: Aligning Representation with Physical Reality

Continuous temporal modeling works because it resolves a fundamental mismatch: physical motion is continuous, but discrete frame indices are not. This alignment provides distinct advantages across several domains:

1. Synergy with Neuromorphic Sensors

Event cameras do not capture frames; they record a continuous stream of microsecond-level intensity changes. Continuous temporal models are uniquely positioned to leverage this data. Yunfan LU et al. demonstrate in UniINR: Event-Guided Unified Rolling Shutter Correction, Deblurring, and Interpolation that a unified INR can fix rolling shutter distortions and interpolate frames simultaneously.

Similarly, Juanjuan Zhu et al. in Video Frame Prediction from a Single Image and Events and Ziyun Wang et al. in Event-based Continuous Color Video Decompression from Single Frames show that pairing continuous motion modeling with event streams allows for the reconstruction of high-frequency visual data at any arbitrary timestamp, bypassing the frame-rate limitations of traditional cameras.

2. Flexible Downstream Generation

When generation tasks involve complex physical constraints, discrete sequences become highly restrictive. For instance, Jingbo Wang et al. show in Task-Oriented Human-Object Interactions Generation with Implicit Neural Representations that fixed-framerate motion sequences make it difficult to adjust velocity or sample densely. By switching to an INR, motion density can be altered post-training.

In creative workflows, Guy Tevet et al. demonstrate in Flexible Motion In-betweening with Diffusion Models that continuous modeling allows animators to place keyframe constraints at arbitrary, non-uniform points in time rather than forcing them onto a rigid grid.

3. Resolving the Continuous-Discrete Tension

Even in high-level video understanding and action detection, the tension between continuous spaces and discrete indices persists. Zhenkun Gao et al. argue in TPRU: Advancing Temporal and Procedural Understanding in Large Multimodal Models that moving from static frame analysis to continuous video integration is essential for capturing fluid real-world actions.

Finally, Sauradip Nag et al. highlight in DiffTAD: Temporal Action Detection with Proposal Denoising Diffusion that bridging the gap between the continuous coordinate spaces used by generative diffusion models and the discrete nature of video frame indices is a core requirement for precisely locating action boundaries.

By treating time as a fluid, first-class dimension, continuous temporal modeling shifts video synthesis away from the constraints of the digital "flipbook" and closer to the continuous dynamics of the physical world.

Key Foundational Work

Video Frame Interpolation via Adaptive Convolution (2017) — contributed 1 concept elements to this direction.
Seamless Bit-Stream Switching in Multirate-Based Video Streaming Systems (2006) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_fixed_interval_interpolation_8d09a8f3,
  title = {fixed-interval interpolation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fixed-interval-interpolation/rdr_8d09a8f3f6274fb98c50bfbf2bc1c593},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_fixed_interval_interpolation_8d09a8f3_v1,
  title = {fixed-interval interpolation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fixed-interval-interpolation/rdr_8d09a8f3f6274fb98c50bfbf2bc1c593/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake