Continuous Neural Representations for Video Dynamics

This direction is split into other directions: fixed-interval interpolation. Its last version stays readable and citable here.

Version 1 of 1171 papers183 conceptsfrozen 2026-10-04

Traditional computer vision treats video as a "flipbook"—a discrete sequence of static frames captured at fixed intervals. While this approach is computationally convenient, it fundamentally clashes with the physical world, where motion is fluid and time is continuous. When we force a continuous phenomenon into a discrete grid of pixels and timestamps, we lose the ability to model higher-order dynamics like acceleration, handle irregular sensor sampling, or synthesize smooth motion between frames.

A growing body of research is shifting away from this discrete "stack of images" paradigm toward continuous neural representations. By modeling video as a continuous function or a differential equation, researchers are building models that understand the underlying physics of motion rather than just the patterns of pixel changes.

The Fallacy of the Discrete Frame

The core problem with discrete modeling is that it treats time as a series of independent snapshots. As noted in The Fallacy in the Paradox of Achilles and the Tortoise by Xuanze Zhao, there is a persistent difficulty in reconciling discrete logical descriptions with continuous physical reality. In video, this manifests as "temporal jitter" or "ghosting" when models try to interpolate between frames.

Standard architectures like RNNs or Transformers operate on fixed steps. If a sensor's frame rate changes or a data point is missing, these models struggle because their internal logic is tied to the index of the frame ($t=1, 2, 3...$) rather than the actual time ($t \in \mathbb{R}$). Abhinav Shrivastava and Gaurav Shrivastava argue in Continuous Video Process that this discretization ignores the inherent physical continuity of motion, requiring heavy-handed attention mechanisms to artificially enforce consistency.

Modeling Motion as a Differential Equation

One powerful solution is to replace discrete state updates with Neural Ordinary Differential Equations (NeuralODEs). Instead of learning a mapping from one frame to the next, the model learns the derivative of the system's state.

In Latent Neural Differential Equations for Video Generation, Cade Gordon and Natalie Parde demonstrate that modeling the latent trajectory of a video as a continuous flow allows for the generation of frames at any arbitrary time point. This approach is particularly effective for physical reasoning; Xin Wang and colleagues introduced PhyVLLM, which uses NeuralODEs to help Video LLMs understand concepts like acceleration and deceleration—physics-based nuances that standard discrete models often miss.

This continuous framing also improves real-time performance. Seunggeun Chi and Hyung‐gun Chi developed InfoGCN++, which uses NeuralODEs to "predict the future" from partial skeleton observations, enabling low-latency action recognition before a movement is even finished. Similarly, Leyla Mirvakhabova and Hong Cai showed that replacing discrete recurrent units with Neural ODEs for optical flow allows the network to dynamically adjust its computation based on the complexity of the motion.

Implicit Neural Representations for Space and Time

Another facet of this direction is the use of Implicit Neural Representations (INRs). Rather than storing video as a 3D grid of pixels ($H \times W \times T$), the video is represented as a continuous function $f(x, y, t) \to RGB$.

This shift enables "infinite" resolution in both space and time. Zeyuan Chen and Yinbo Chen introduced VideoINR, which allows for continuous space-time super-resolution. Because the representation is continuous, you can query the model for a frame at $t=1.5$ just as easily as $t=1.0$, enabling perfectly smooth slow-motion. Sihyun Yu and Jihoon Tack applied this to generative modeling with DIGAN, generating high-quality videos that avoid the temporal artifacts common in grid-based GANs.

Handling Stochasticity and Irregularity

Real-world motion isn't just continuous; it's often unpredictable. Kuk-Jin Yoon and Daehee Park addressed this by using Neural Stochastic Differential Equations (NSDEs) in TrajSDE. By modeling movement as a continuous stochastic process, their framework can handle varying sensor noise and irregular sampling rates across different datasets.

This perspective extends to how models learn representations. Daqing Liu and Bing Su argue in Modeling Video as Stochastic Processes that treating video as a continuous, evolving process is essential for fine-grained understanding. Their work on Temporal Concurrent Processes (TCP) uses differentiable sequence alignment to ensure that both visual and textual narratives are treated as continuous, coupled flows rather than discrete clips and sentences.

Why Continuity Works

The intuition behind the success of these methods is twofold:

  1. Inductive Bias: By using ODEs or INRs, we bake the laws of physics (smoothness, inertia, continuity) directly into the model's architecture.
  2. Flexibility: Continuous models are "resolution-agnostic." They can be trained on 30fps video and queried at 1000fps, or trained on sparse trajectories and queried for dense motion fields, as seen in the DOMA model by Qianli Ma.

By moving from the "flipbook" to the "flow," researchers are creating a more mathematically rigorous and physically grounded foundation for how machines perceive and generate the moving world.

Papers in this version

Showing 60 of 171 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_continuous_neural_representations_for_vi_81e95dde,
  title = {Continuous Neural Representations for Video Dynamics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continuous-neural-representations-for-video-dynamics/rdr_81e95dde15494ae3b2583b5bcc8bb66a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_continuous_neural_representations_for_vi_81e95dde_v1,
  title = {Continuous Neural Representations for Video Dynamics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continuous-neural-representations-for-video-dynamics/rdr_81e95dde15494ae3b2583b5bcc8bb66a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}