Abstract
Implicit Neural Representations (INR) have emerged as a promising alternative to traditional video compression by representing video data as the weights of a neural network. However, previous INR-based methods have consistently underperformed compared to established standards like VVC due to suboptimal compression backends. This paper introduces NVRC (Neural Video Representation Compression), the first INR-based video codec to officially outperform the state-of-the-art VVC standard. By treating video compression as an end-to-end hierarchical model compression problem, NVRC achieves a 23.4% bitrate saving over VVC VTM on standard benchmarks.
Introduction
Traditional video compression standards, such as H.264 and HEVC, rely on hand-crafted algorithms for motion estimation and compensation. Recently, Implicit Neural Representations (INR) proposed a radical shift: instead of storing pixel values, a small neural network is trained to "overfit" a specific video sequence. The network's weights then serve as the compressed representation. While methods like NeRV and HiNeRV demonstrated fast decoding, they lagged behind traditional standards because their compression pipelines—often relying on simple pruning or fixed quantization—were not optimized for the critical rate-distortion (R-D) trade-off. NVRC addresses this by making the entire compression pipeline differentiable and end-to-end optimized.
Hierarchical Parameter Coding
The core innovation of NVRC is its hierarchical parameter coding framework. The authors posit that to achieve peak efficiency, the system must compress not only the primary network parameters but also the metadata used for compression.
As shown in the architecture above, NVRC utilizes a three-level hierarchy:
- Primary Parameters (θ): The weights of the main network.
- Entropy Model Parameters (φ): Parameters governing the quantization and entropy models.
- Hyper-parameters (ψ): High-level parameters for the entropy models themselves.
To further enhance efficiency, NVRC employs Feature Grid Coding using 3D block partitioning and a context-based Gaussian entropy model to exploit spatial-temporal redundancies. Additionally, a Dual-Axis Layer Coding scheme is used for standard layers, learning quantization scales for both input and output channels simultaneously for finer bit allocation control.
Optimization Strategy
Calculating complex entropy models at every training step is computationally prohibitive. NVRC introduces an alternating optimization pipeline that balances distortion (visual quality) and rate (file size). The training process involves a two-stage approach: a "soft-rounding" stage using Kumaraswamy noise to simulate quantization differentiably, followed by a "Quant-Noise" fine-tuning stage to stabilize the model at high quantization levels.
Experimental Results
NVRC was evaluated against industry-standard benchmarks including UVG, MCL-JCV, and JVET-CTC Class B.
Performance Metrics
- VVC Comparison: NVRC achieved an average 23.4% BD-rate saving (measured in PSNR) over the VVC VTM (Random Access) configuration on the UVG dataset.
- INR Comparison: It outperformed the previous state-of-the-art INR codec, HiNeRV, by approximately 50% in bitrate savings.
Visual Quality
Qualitative analysis demonstrates that NVRC preserves significantly more texture and detail than previous INR methods at similar bitrates, effectively avoiding the "muddy" artifacts common in heavily compressed neural video.
Limitations and Future Work
Despite its performance gains, NVRC faces challenges regarding encoder complexity. Because it requires training a unique network for every video, the encoding process is currently too slow for real-time applications. Furthermore, the system introduces significant latency as the encoder must process large chunks of the video sequence to optimize the representation. Future work will focus on improving encoding speed and porting operations to integer math to ensure bit-exact reproducibility across diverse hardware platforms.
Summary updated