NVRC: Neural Video Representation Compression

Ho Man Kwan, Ge Gao, Fan Zhang, Andrew Gower, David Bull

TL;DR

NVRC (Neural Video Representation Compression) is the first implicit neural representation (INR) video codec to outperform the state-of-the-art conventional standard, VVC VTM. By shifting the focus from network architecture design to a fully end-to-end optimized hierarchical compression framework, NVRC achieves a massive 23% average coding gain over VVC.

What Problem Does This Paper Address?

Learning-based video compression has shown immense promise, but standard autoencoder-based approaches suffer from high computational complexity at the decoder, making practical deployment difficult. To solve this, researchers turned to Implicit Neural Representations (INRs), where a neural network is trained to "overfit" a specific video sequence. In an INR codec, the video is stored simply as the compressed weights and feature grids of the neural network; decoding a frame is as fast as running a single forward pass.

However, INR-based codecs have historically lagged behind traditional video standards like H.266/VVC. The bottleneck hasn't been the network architectures themselves, but rather how the network's parameters are compressed. Previous methods relied on crude, separate steps for pruning, quantization, and entropy coding, failing to optimize the entire system for the ultimate rate-distortion (RD) trade-off. NVRC bridges this gap by introducing a fully end-to-end rate-distortion optimized framework for INR compression.

Methods

NVRC introduces a novel hierarchical parameter coding strategy to compress the neural representation without letting the metadata overhead blow up the bitrate. The framework optimizes three distinct layers of parameters simultaneously:

  1. Network Parameters ($\theta$): Comprising the explicit feature grids ($\theta_{\text{grid}}$) and the network layer weights ($\theta_{\text{layer}}$).
  2. Compression Parameters ($\phi$): The learned quantization scales and entropy model parameters used to encode $\theta$.
  3. Hyper-parameters ($\psi$): A ultra-lightweight layer used to compress $\phi$ itself.
Figure 2: In NVRC, the parameters are encoded in a hierarchical structure, where (Middle-left) per-block quantization scales and (bottom-left) context-based model are utilized for encoding feature grids, and (Middle-right and bottom-right) per-axis quantization scales and dual-axis Gaussian model are applied for encoding network layer parameters.

Feature Grid Coding

To exploit spatial-temporal redundancies within the video's underlying feature grids, NVRC partitions them into small 3D blocks. It applies per-block learned quantization scales and encodes them using a context-based Gaussian entropy model powered by 3D masked convolutions. This allows high parallelism during decoding.

Network Layer Parameter Coding

Unlike feature grids, neural network weights lack spatial correlation. NVRC reshapes these weights into 2D tensors (representing output vs. input channels) and applies a dual-axis conditional Gaussian model. Quantization and entropy parameters are learned and mixed across both axes simultaneously, capitalizing on the fact that if an input or output channel is pruned to zero, surrounding parameters follow suit.

Alternating Optimization

To avoid the massive computational and memory overhead of calculating complex entropy models at every single training step, the authors propose an alternating optimization pipeline. For every $K+1$ steps, the distortion loss ($D$) is minimized for the first $K$ steps, and the rate loss ($R$) is updated only on the final step.

Datasets

The authors evaluated NVRC against a comprehensive suite of benchmarks using three standard video datasets:

  • UVG: 7 high-quality video sequences (300/600 frames).
  • MCL-JCV: 30 video clips (120–150 frames) tailored for human-perceived quality evaluation.
  • JVET-CTC Class B: Standardized test sequences for video codec development.

To ensure a fair trial, evaluations were conducted across both RGB 4:4:4 (favored by learned codecs) and YUV 4:2:0 (the standard for conventional codecs) color spaces.

Key Insights and Ideas

  • Compression Over Architecture: While prior literature focused heavily on designing better INR network backbones (like NeRV or HiNeRV), this paper proves that the real performance unlock comes from treating the network parameters as a highly structured data source ripe for advanced entropy coding.
  • Mitigating the Overhead Tax: Advanced entropy models require sending "side information" (quantization scales, distributions) to the decoder. NVRC's "aha moment" is realizing that this side information can also be modeled and compressed via a third hierarchical layer ($\psi$), reducing the metadata tax to a negligible fraction of the total bitrate.
  • Dual-Axis Weight Modeling: Treating neural network layers as 2D matrices and compressing them along both dimensions simultaneously yields significantly better rate-distortion performance than traditional row-wise or column-wise quantization schemes.

Key Results

NVRC achieves a historic milestone in neural video compression, soundly defeating both prior INR methods and top-tier traditional codecs:

  • Vs. VVC VTM (Random Access): NVRC achieves a 23.4% average BD-rate saving (coding gain) on the UVG dataset measured in PSNR. This is the first time an INR codec has significantly outperformed VVC.
  • Vs. Prior INR State-of-the-Art (HiNeRV): NVRC delivers up to 50.1% BD-rate savings, effectively cutting the required bitrate in half for the same visual quality.
Figure 1: Comparison between the output from HiNeRV [30] and the proposed NVRC. The image is from UVG dataset (Jockey/ReadySetGo sequence) [45].

As seen in Figure 1, NVRC preserves sharp textures and structural details (like the text on the background screen and the horse's gear) far better than HiNeRV at even lower bitrates (bpp).

Limitations and Future Work

Despite its stellar decoding efficiency and rate-distortion performance, NVRC shares a common limitation with other INR approaches: Encoder Complexity. Because the network must be iteratively overfitted to every new video sequence, encoding requires substantial time and GPU compute resources. This makes the current framework unsuitable for real-time upstream applications like live video conferencing.

Additionally, because the encoder needs to look at all frames of a video sequence simultaneously to optimize the global representation, it introduces significant system latency. Future work will need to focus on accelerating the training pipeline and exploring frame-by-frame incremental updates to make INR codecs viable for real-time streaming ecosystems.

Summary updated

Abstract

Recent advances in implicit neural representation (INR)-based video coding have demonstrated its potential to compete with both conventional and other learning-based approaches. With INR methods, a neural network is trained to overfit a video sequence, with its parameters compressed to obtain a compact representation of the video content. However, although promising results have been achieved, the best INR-based methods are still out-performed by the latest standard codecs, such as VVC VTM, partially due to the simple model compression techniques employed. In this paper, rather than focusing on representation architectures as in many existing works, we propose a novel INR-based video compression framework, Neural Video Representation Compression (NVRC), targeting compression of the representation. Based on the novel entropy coding and quantization models proposed, NVRC, for the first time, is able to optimize an INR-based video codec in a fully end-to-end manner. To further minimize the additional bitrate overhead introduced by the entropy models, we have also proposed a new model compression framework for coding all the network, quantization and entropy model parameters hierarchically. Our experiments show that NVRC outperforms many conventional and learning-based benchmark codecs, with a 24% average coding gain over VVC VTM (Random Access) on the UVG dataset, measured in PSNR. As far as we are aware, this is the first time an INR-based video codec achieving such performance. The implementation of NVRC will be released.
BibTeX
@inproceedings{kwan2024nvrc,
  author = {Ho Man Kwan and Ge Gao and Fan Zhang and Andrew Gower and David Bull},
  title = {{NVRC: Neural Video Representation Compression}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=8v51GxwQ0H},
}
Figure