NVRC: Neural Video Representation Compression

Ho Man Kwan, Ge Gao, Fan Zhang, Andrew Peter Gower, David Bull

Abstract

Implicit Neural Representations (INR) have emerged as a promising alternative to traditional video compression by representing video data as the weights of a neural network. However, previous INR-based methods have consistently underperformed compared to established standards like VVC due to suboptimal compression backends. This paper introduces NVRC (Neural Video Representation Compression), the first INR-based video codec to officially outperform the state-of-the-art VVC standard. By treating video compression as an end-to-end hierarchical model compression problem, NVRC achieves a 23.4% bitrate saving over VVC VTM on standard benchmarks.

Introduction

Traditional video compression standards, such as H.264 and HEVC, rely on hand-crafted algorithms for motion estimation and compensation. Recently, Implicit Neural Representations (INR) proposed a radical shift: instead of storing pixel values, a small neural network is trained to "overfit" a specific video sequence. The network's weights then serve as the compressed representation. While methods like NeRV and HiNeRV demonstrated fast decoding, they lagged behind traditional standards because their compression pipelines—often relying on simple pruning or fixed quantization—were not optimized for the critical rate-distortion (R-D) trade-off. NVRC addresses this by making the entire compression pipeline differentiable and end-to-end optimized.

Hierarchical Parameter Coding

The core innovation of NVRC is its hierarchical parameter coding framework. The authors posit that to achieve peak efficiency, the system must compress not only the primary network parameters but also the metadata used for compression.

Overview of the NVRC hierarchical parameter coding framework, illustrating the multi-level compression of parameters θ, φ, and ψ.

As shown in the architecture above, NVRC utilizes a three-level hierarchy:

  1. Primary Parameters (θ): The weights of the main network.
  2. Entropy Model Parameters (φ): Parameters governing the quantization and entropy models.
  3. Hyper-parameters (ψ): High-level parameters for the entropy models themselves.

To further enhance efficiency, NVRC employs Feature Grid Coding using 3D block partitioning and a context-based Gaussian entropy model to exploit spatial-temporal redundancies. Additionally, a Dual-Axis Layer Coding scheme is used for standard layers, learning quantization scales for both input and output channels simultaneously for finer bit allocation control.

Optimization Strategy

Calculating complex entropy models at every training step is computationally prohibitive. NVRC introduces an alternating optimization pipeline that balances distortion (visual quality) and rate (file size). The training process involves a two-stage approach: a "soft-rounding" stage using Kumaraswamy noise to simulate quantization differentiably, followed by a "Quant-Noise" fine-tuning stage to stabilize the model at high quantization levels.

Experimental Results

NVRC was evaluated against industry-standard benchmarks including UVG, MCL-JCV, and JVET-CTC Class B.

Performance Metrics

  • VVC Comparison: NVRC achieved an average 23.4% BD-rate saving (measured in PSNR) over the VVC VTM (Random Access) configuration on the UVG dataset.
  • INR Comparison: It outperformed the previous state-of-the-art INR codec, HiNeRV, by approximately 50% in bitrate savings.

Visual Quality

Qualitative analysis demonstrates that NVRC preserves significantly more texture and detail than previous INR methods at similar bitrates, effectively avoiding the "muddy" artifacts common in heavily compressed neural video.

Visual and quantitative comparison between HiNeRV and NVRC. NVRC achieves higher PSNR at lower bits-per-pixel (bpp) across different video sequences.

Limitations and Future Work

Despite its performance gains, NVRC faces challenges regarding encoder complexity. Because it requires training a unique network for every video, the encoding process is currently too slow for real-time applications. Furthermore, the system introduces significant latency as the encoder must process large chunks of the video sequence to optimize the representation. Future work will focus on improving encoding speed and porting operations to integer math to ensure bit-exact reproducibility across diverse hardware platforms.

Summary updated

Abstract

Recent advances in implicit neural representation (INR)-based video coding have demonstrated its potential to compete with both conventional and other learning- based approaches. With INR methods, a neural network is trained to overfit a video sequence, with its parameters compressed to obtain a compact representation of the video content. However, although promising results have been achieved, the best INR-based methods are still out-performed by the latest standard codecs, such as VVC VTM, partially due to the simple model compression techniques employed. In this paper, rather than focusing on representation architectures, which is a common focus in many existing works, we propose a novel INR-based video compression framework, Neural Video Representation Compression (NVRC), targeting compression of the representation. Based on its novel quantization and entropy coding approaches, NVRC is the first framework capable of optimizing an INR-based video representation in a fully end-to-end manner for the rate-distortion trade-off. To further minimize the additional bitrate overhead introduced by the entropy models, NVRC also compresses all the network, quantization and entropy model parameters hierarchically. Our experiments show that NVRC outperforms many conventional and learning-based benchmark codecs, with a 23% average coding gain over VVC VTM (Random Access) on the UVG dataset, measured in PSNR. As far as we are aware, this is the first time an INR-based video codec achieving such performance.
BibTeX
@inproceedings{kwan2024nvrc,
  author = {Ho Man Kwan and Ge Gao and Fan Zhang and Andrew Peter Gower and David Bull},
  title = {{NVRC: Neural Video Representation Compression}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=8v51GxwQ0H},
}
Figure