Traditional video compression treats a movie as a sequence of pixel grids. To save space, codecs like H.264 look for patterns between these grids, storing only the differences. But a new research direction, pioneered by Hao Chen and colleagues in NeRV: Neural Representations for Videos, proposes a radical shift: Video is not a sequence of images; it is a continuous function.
In this paradigm, a video is represented as a neural network. If you want to see frame #50, you don't look up a file on a disk; you feed the number "50" into a network, and the network outputs the RGB pixels for that frame.
Overfitting as Encoding
The core intuition of this field is that overfitting is a feature, not a bug. Usually, we want neural networks to generalize to new data. Here, we want the opposite. We take a specific video and train a network to memorize it perfectly.
The "compressed" version of the video is simply the weights of that trained network. Because neural networks are excellent at finding and exploiting patterns in data, the weights of the network can be much smaller than the raw pixel data. To "decode" the video, a user just needs the model weights and a standard deep learning framework to run a single forward pass per frame.
The Per-Video Optimization Bottleneck
While this approach allows for incredibly fast decoding and high-quality interpolation, it faces a massive hurdle: the encoding phase. In traditional codecs, encoding is a deterministic algorithm that happens in milliseconds. In the world of Implicit Neural Representations (INR), encoding is a training process.
As noted by Joo Chan Lee in FFNeRV, training a unique network from scratch for every new video sequence is computationally expensive. This "per-video optimization" can take minutes or even hours for a short clip, making it impractical for live streaming or real-time applications. Much of the recent research in this area focuses on breaking this bottleneck.
From Content-Agnostic to Content-Adaptive
The first generation of these models, like the original NeRV, was "content-agnostic." The input was just a timestamp or a frame index. The network had to do all the heavy lifting of figuring out what the video looked like from just a single number.
The field evolved with HNeRV: A Hybrid Neural Representation for Videos, which introduced content-adaptive embeddings. Instead of a simple frame index, the model uses a learnable encoder to produce a "hint" (a latent code) for each frame. This hybrid approach allows the decoder to be more efficient, leading to a 16x faster convergence speed and significantly better image quality.
Learning the Prior
If every video is a brand-new training task, the model "forgets" everything it knows about how the world looks (like edges, textures, or common motion) every time it starts a new video. Researchers are now using meta-learning to solve this.
- Meta-Learning: Jialong Guo introduced MetaNeRV, which uses a meta-learning framework to find a "weight initialization" that works well for most videos. Instead of starting from scratch, the model starts from a point where it already understands basic visual structures, allowing it to adapt to a new video in just a few gradient steps.
- Hypernetworks: Maria Pilligua explored HyperNVD, which uses a second "hypernetwork" to predict the weights of the video network directly from a video embedding. This provides a high-quality starting point that can cut training time by over 30 minutes.
- Cross-Video Priors: Wentao Shangguan developed CURE, which learns a general prior across multiple videos to enable feed-forward inference—essentially "encoding" a video in a single pass without any per-video optimization at all.
Beyond Simple Playback
Because the video is now a differentiable function, we can do things that are impossible with standard pixels. DS-NeRV decomposes the video into separate "static" and "dynamic" codes, effectively separating the background from moving objects. This makes tasks like video inpainting (removing an object) or video interpolation (creating slow-motion) a natural byproduct of how the video is stored.
By treating video as a neural network, the field is moving toward a future where "compression" is no longer about discarding pixels, but about learning the most efficient mathematical representation of a scene.