Implicit Neural Representations for Video Compression

47 concepts32 papers1 proposal

Traditional video compression standards like HEVC and VVC rely on hand-crafted, block-based motion estimation and residual coding to reduce spatial and temporal redundancies. While early end-to-end learned video codecs attempted to replace these pipelines with deep autoencoders, they frequently struggled to consistently outperform mature traditional standards across all evaluation metrics (Zhengxue Cheng et al.; Yinhao Zhu et al.).

A fundamentally different paradigm has emerged: Implicit Neural Representations (INRs). Instead of treating a video as a discrete sequence of pixel grids or latent tokens, INR-based codecs treat the video as a continuous coordinate function. The network takes spatial-temporal coordinates $(x, y, t)$ as inputs and outputs the corresponding RGB values. In this setup, the video is entirely contained within the parameters of the network itself. To compress the video, one simply compresses the network's weights.

This approach offers two massive intuitive advantages:

  1. Asymmetric Complexity: As Ho Man Kwan et al. demonstrate, decoding an INR reduces to a simple forward pass through the network. This eliminates the heavy autoregressive loops or complex prior computations of autoencoder codecs, making INRs exceptionally well-suited for low-complexity client-side playback.
  2. High-Dimensional Scaling: For complex, multi-dimensional data like volumetric or 3D video, traditional block-matching breaks down. Kaiyuan Hu et al. and Ge Gao et al. show that neural representations naturally scale to capture intricate spatial-temporal distributions, offering significant coding gains over legacy standards like MPEG TMIV.

The Parameter Coding Bottleneck

The core challenge of INR video compression is the weight storage overhead. To represent a video with high fidelity, a coordinate network must be "overfitted" to that specific sequence via gradient descent. However, if the resulting network parameters require more storage bits than the original raw or traditionally compressed video, the approach fails.

As Yannick Strümpler et al. observed, without specialized optimization, the storage cost of network weights quickly outweighs the quality of the reconstructed signal. Early coordinate-based networks exhibited a severe performance gap compared to mature video standards, particularly at high bitrates (Yunfan Zhang et al.). Standard network compression techniques—such as uniform quantization or heuristic pruning—are insufficient because they are not optimized for the strict rate-distortion trade-off of the reconstructed video signal.

Embedding Motion and Flow into Coordinates

To make the network's weights easier to compress, the architecture must be given structural inductive priors that reflect how videos actually change over time. If a network has to independently memorize every frame, its weights will remain highly complex and difficult to compress.

To resolve this, researchers began explicitly separating motion from static content within the coordinate space:

  • Implicit Pixel Flow: Yunfan Zhang et al. introduced a coordinate-based warping technique. Instead of mapping $(x, y, t)$ directly to color, the network learns to predict motion trajectories, warping pixels from reference frames to perform neural motion compensation.
  • Difference Streams: Qi Zhao et al. developed DNeRV: Modeling Inherent Dynamics via Difference Neural Representation for Videos, which explicitly models video dynamics by running a "difference stream" alongside a standard content stream, fusing them via a gated mechanism to handle high-motion scenes.
  • Decoupled Flow and Color Networks: Daniel Silver proposed CoordFlow: Coordinate Flow for Pixel-wise Neural Video Representation, which splits the architecture into a dedicated "Flow Network" for motion and a "Color Network" for texture. This separation allows the continuous coordinate network to achieve high reconstruction quality without sacrificing compression efficiency.
  • Frequency-Aware Representations: To prevent the blurry reconstructions typical of naive coordinate networks, Li Yu 0004 et al. incorporated wavelet-based encoders and adaptive harmonic activations. This hybrid approach allows the network to efficiently capture fine textures and high-frequency edges, outperforming traditional H.265 codecs.

End-to-End Rate-Distortion Optimization

The definitive breakthrough that allowed INR video codecs to surpass the state-of-the-art traditional standards came from treating the network parameters as the actual payload in an end-to-end rate-distortion optimization (RDO) framework.

Traditional Video:  [Pixels] ----> [Block Motion / Residuals] ----> [Entropy Coding] ----> Bitstream
Implicit Neural:     [Video]  ----> [End-to-End Weight RDO]   ----> [Quantized Weights] -> Bitstream

Ho Man Kwan et al. identified that the primary bottleneck in neural video representation was not the network's architectural capacity, but the simplistic compression of its parameters. In their framework, NVRC: Neural Video Representation Compression, they introduced an end-to-end optimized hierarchical parameter compression scheme. By directly minimizing a joint loss function of weight bitrate and video distortion during the overfitting process, they achieved a 23% coding gain over the VVC VTM standard.

Taking a different angle on parameter efficiency, Hee Min Choi et al. challenged the idea that network weights need to be trained at all. By leveraging the Strong Lottery Ticket hypothesis, the authors demonstrated that one can compress high-definition video simply by finding an optimal subnetwork within a completely random, untrained initialization, bypassing traditional weight optimization entirely while maintaining competitive compression performance.

Overcoming the Encoding Speed Barrier

Despite their excellent decoding speeds and compression performance, early INR codecs suffered from a critical flaw: encoding was computationally prohibitive. Overfitting a network to a single video required thousands of iterations of gradient descent, taking hours or even days.

To make INRs viable for real-time applications, Hao Chen et al. introduced a hyper-network approach called Fast NeRV. Instead of optimizing network weights from scratch for every video, they trained a large, generalized transformer-based hyper-network. When a new video arrives, the hyper-network predicts the optimal coordinate network weights in a single forward pass. This architectural shift accelerates the video encoding process by a staggering 10,000× and decoding by 11×, transforming implicit neural representations from a theoretical curiosity into a practical tool for next-generation video delivery.

Key Foundational Work

End-to-End Rate-Distortion Optimization for Bi-Directional Learned Video Compression (2020) — contributed 1 concept elements to this direction.
Joint Learned and Traditional Video Compression for P Frame (2020) — contributed 1 concept elements to this direction.
Adversarial Distortion for Learned Video Compression (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_state_of_the_art_inr_video_compression_0216d241,
  title = {state-of-the-art INR video compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/state-of-the-art-inr-video-compression/rdr_0216d241a2384811842eecd2349d0ee8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_state_of_the_art_inr_video_compression_0216d241_v1,
  title = {state-of-the-art INR video compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/state-of-the-art-inr-video-compression/rdr_0216d241a2384811842eecd2349d0ee8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake