Implicit Neural Video Representations

49 concepts37 papers1 proposal

Traditionally, digital video is treated as a discrete stack of pixel grids sampled at fixed intervals—for instance, 30 frames per second at a $1920 \times 1080$ resolution. While intuitive, this discrete representation introduces fundamental limitations. If you want to know what happens between two frames (e.g., at $t = 1.5$), or if you need to scale the video up to a higher resolution, you must rely on external interpolation heuristics like optical flow or pixel blending, which frequently introduce visual artifacts. Furthermore, storing videos as dense 3D grids causes memory requirements to scale poorly with length and resolution.

Implicit Neural Video Representations redefine video by shifting from discrete pixel grids to continuous coordinate spaces. Instead of storing an array of colors, a video is parameterized as a continuous function—typically a coordinate-based neural network like a Multi-Layer Perceptron (MLP)—that maps spatiotemporal coordinates directly to pixel values:

$$f(x, y, t) \to (R, G, B)$$

Because the domain of this network is continuous, you can query it at any arbitrary spatial coordinate or fractional timestamp. This simple shift in perspective unlocks resolution-independent synthesis, zero-cost temporal interpolation, and a natural framework for processing asynchronous sensor data.


From Discrete Grids to Continuous Functions

To understand why this works, consider how a neural network represents high-frequency visual details. By utilizing positional encodings or periodic activation functions, an MLP can map low-dimensional coordinates $(x, y, t)$ to highly complex color spaces.

Yunfan Zhang et al. leverage this property in Implicit Neural Video Compression by overfitting small MLPs to specific video sequences. Instead of saving pixel arrays, they save the weights of the network as the compressed bitstream itself, bypassing the need for traditional pixel-based decoders.

Bypassing the grid structure also eliminates the memory bottlenecks that plague standard video generation models. As demonstrated by Sihyun Yu et al. in Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks, treating video as a continuous function allows generative adversarial networks (GANs) to produce long, high-resolution video sequences without being constrained by the memory limits of 3D convolutional grids.


Continuous-Time Querying and Zero-Cost Interpolation

In a traditional video pipeline, changing the frame rate requires explicitly predicting intermediate motion vectors or synthesizing new frames with a separate model. In an implicit representation, temporal interpolation comes "for free" because time $t$ is simply a continuous input variable.

Discrete Video Grid:
[Frame 1 (t=1)] ---- (Interpolation Model) ----> [Frame 2 (t=2)]

Implicit Neural Representation:
Continuous t ───> [ MLP (Network Weights) ] ───> Color (R,G,B)
                   (Query any t, e.g., t=1.5)

Yixuan Ren et al. exploit this in NeRV-Diffusion: Diffuse Implicit Neural Representation for Video Synthesis, showing that a diffusion model trained to output the weights of a continuous-time network can perform zero-shot temporal interpolation simply by querying fractional time steps.

To better capture complex physical trajectories over time, Sunghyun Park et al. introduce Vid-ODE: Continuous-Time Video Generation with Neural Ordinary Differential Equation. By modeling the underlying scene dynamics as a continuous differential equation, their framework naturally synthesizes smooth transitions even when the training data is irregularly sampled.

Similarly, Xi Ye et al. show in A unified model for continuous conditional video prediction that framing video prediction as a coordinate-based conditional mapping allows a single model to handle future frame prediction, interpolation, and missing frame completion interchangeably.


Processing Asynchronous and Blurry Sensors

The real world does not operate in discrete frames, and neither do many modern sensors. Event cameras, for instance, do not capture full images; instead, they record an asynchronous stream of microsecond-level brightness changes (events). Matching these sparse, continuous events to a rigid, discrete video grid is mathematically awkward.

Implicit representations resolve this mismatch because they accommodate asynchronous data natively. In UniINR: Event-Guided Unified Rolling Shutter Correction, Deblurring, and Interpolation, Yunfan LU et al. use a unified implicit network to ingest event streams and reconstruct high-frame-rate, sharp video from a single motion-blurred exposure. Because the representation is continuous, it corrects rolling shutter distortions and motion blur simultaneously without step-by-step error accumulation.

Furthermore, Zipeng Wang et al. demonstrate in Revisit Event Generation Model: Self-supervised Learning of Event-to-Video Reconstruction with Implicit Neural Representations that mapping sparse event data to a continuous MLP acts as a natural noise filter, allowing users to cleanly query scene intensity at any arbitrary spatiotemporal coordinate.


Generalizing Beyond a Single Video

Early iterations of implicit neural representations required optimizing a brand-new network from scratch for every single video sequence. To make this paradigm practical for downstream applications, models must learn generalizable priors across diverse video datasets.

One approach is to condition the coordinate network on learned spatiotemporal features or latents extracted from context frames. Wentao Shangguan et al. implement this in Learning Cross-Video Neural Representations for High-Quality Frame Interpolation, enabling fast, feed-forward frame interpolation without per-video optimization. To handle non-linear and complex object trajectories across different videos, Zujin Guo et al. introduce Generalizable Implicit Motion Modeling for Video Frame Interpolation, which conditions the coordinate network on a learned motion latent space.

Alternatively, meta-learning can be used to find a set of initial network weights that can quickly adapt to any new video. Ke Liu demonstrates this in MetaNeRV: Meta Neural Representations for Videos with Spatial-Temporal Guidance, showing that incorporating multi-resolution spatial-temporal guidance allows a meta-network to adapt to unseen videos in just a few gradient steps, combining the speed of feed-forward models with the high fidelity of overfitted representations.


Inherent Regularization and 4D Scene Control

Why do coordinate-based networks perform so well at reconstructing clean video from noisy or incomplete data? The answer lies in the spectral bias of neural networks: MLPs naturally favor learning smooth, low-frequency functions before fitting high-frequency noise.

Jianlang Chen et al. exploit this architectural bias in LRR: Language-Driven Resamplable Continuous Representation against Adversarial Tracking Attacks. They show that modeling video frames as continuous implicit functions acts as a natural low-pass filter, stripping away the high-frequency adversarial perturbations meant to trick visual trackers while preserving the underlying smooth spatiotemporal structure of the video.

Finally, this continuous formulation scales seamlessly from 2D pixel spaces to 4D spatiotemporal representations ($x, y, z, t$), allowing models to disentangle camera movement from scene dynamics. As shown by Sherwin Bahmani et al. in 3D-Aware Video Generation, querying a continuous 4D neural implicit field allows for independent control over camera viewpoints and temporal progression, generating multi-view consistent videos from unstructured monocular data.

Key Foundational Work

Time-Dependent Deep Image Prior for Dynamic MRI (2019) — contributed 2 concept elements to this direction.
Vid-ODE: Continuous-Time Video Generation with Neural Ordinary Differential Equation (2021) — contributed 2 concept elements to this direction.
Neural Radiance Flow for 4D View Synthesis and Video Processing (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_implicit_neural_video_representation_161e3d39,
  title = {implicit neural video representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/implicit-neural-video-representation/rdr_161e3d39f7cf4b6cbe6f5c1be927d7a1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_implicit_neural_video_representation_161e3d39_v1,
  title = {implicit neural video representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/implicit-neural-video-representation/rdr_161e3d39f7cf4b6cbe6f5c1be927d7a1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake