TL;DR
NVRC (Neural Video Representation Compression) is the first implicit neural representation (INR) video codec to outperform the state-of-the-art conventional standard, VVC VTM. By shifting the focus from network architecture design to a fully end-to-end optimized hierarchical compression framework, NVRC achieves a massive 23% average coding gain over VVC.
What Problem Does This Paper Address?
Learning-based video compression has shown immense promise, but standard autoencoder-based approaches suffer from high computational complexity at the decoder, making practical deployment difficult. To solve this, researchers turned to Implicit Neural Representations (INRs), where a neural network is trained to "overfit" a specific video sequence. In an INR codec, the video is stored simply as the compressed weights and feature grids of the neural network; decoding a frame is as fast as running a single forward pass.
However, INR-based codecs have historically lagged behind traditional video standards like H.266/VVC. The bottleneck hasn't been the network architectures themselves, but rather how the network's parameters are compressed. Previous methods relied on crude, separate steps for pruning, quantization, and entropy coding, failing to optimize the entire system for the ultimate rate-distortion (RD) trade-off. NVRC bridges this gap by introducing a fully end-to-end rate-distortion optimized framework for INR compression.
Methods
NVRC introduces a novel hierarchical parameter coding strategy to compress the neural representation without letting the metadata overhead blow up the bitrate. The framework optimizes three distinct layers of parameters simultaneously:
- Network Parameters ($\theta$): Comprising the explicit feature grids ($\theta_{\text{grid}}$) and the network layer weights ($\theta_{\text{layer}}$).
- Compression Parameters ($\phi$): The learned quantization scales and entropy model parameters used to encode $\theta$.
- Hyper-parameters ($\psi$): A ultra-lightweight layer used to compress $\phi$ itself.
Feature Grid Coding
To exploit spatial-temporal redundancies within the video's underlying feature grids, NVRC partitions them into small 3D blocks. It applies per-block learned quantization scales and encodes them using a context-based Gaussian entropy model powered by 3D masked convolutions. This allows high parallelism during decoding.
Network Layer Parameter Coding
Unlike feature grids, neural network weights lack spatial correlation. NVRC reshapes these weights into 2D tensors (representing output vs. input channels) and applies a dual-axis conditional Gaussian model. Quantization and entropy parameters are learned and mixed across both axes simultaneously, capitalizing on the fact that if an input or output channel is pruned to zero, surrounding parameters follow suit.
Alternating Optimization
To avoid the massive computational and memory overhead of calculating complex entropy models at every single training step, the authors propose an alternating optimization pipeline. For every $K+1$ steps, the distortion loss ($D$) is minimized for the first $K$ steps, and the rate loss ($R$) is updated only on the final step.
Datasets
The authors evaluated NVRC against a comprehensive suite of benchmarks using three standard video datasets:
- UVG: 7 high-quality video sequences (300/600 frames).
- MCL-JCV: 30 video clips (120–150 frames) tailored for human-perceived quality evaluation.
- JVET-CTC Class B: Standardized test sequences for video codec development.
To ensure a fair trial, evaluations were conducted across both RGB 4:4:4 (favored by learned codecs) and YUV 4:2:0 (the standard for conventional codecs) color spaces.
Key Insights and Ideas
- Compression Over Architecture: While prior literature focused heavily on designing better INR network backbones (like NeRV or HiNeRV), this paper proves that the real performance unlock comes from treating the network parameters as a highly structured data source ripe for advanced entropy coding.
- Mitigating the Overhead Tax: Advanced entropy models require sending "side information" (quantization scales, distributions) to the decoder. NVRC's "aha moment" is realizing that this side information can also be modeled and compressed via a third hierarchical layer ($\psi$), reducing the metadata tax to a negligible fraction of the total bitrate.
- Dual-Axis Weight Modeling: Treating neural network layers as 2D matrices and compressing them along both dimensions simultaneously yields significantly better rate-distortion performance than traditional row-wise or column-wise quantization schemes.
Key Results
NVRC achieves a historic milestone in neural video compression, soundly defeating both prior INR methods and top-tier traditional codecs:
- Vs. VVC VTM (Random Access): NVRC achieves a 23.4% average BD-rate saving (coding gain) on the UVG dataset measured in PSNR. This is the first time an INR codec has significantly outperformed VVC.
- Vs. Prior INR State-of-the-Art (HiNeRV): NVRC delivers up to 50.1% BD-rate savings, effectively cutting the required bitrate in half for the same visual quality.
As seen in Figure 1, NVRC preserves sharp textures and structural details (like the text on the background screen and the horse's gear) far better than HiNeRV at even lower bitrates (bpp).
Limitations and Future Work
Despite its stellar decoding efficiency and rate-distortion performance, NVRC shares a common limitation with other INR approaches: Encoder Complexity. Because the network must be iteratively overfitted to every new video sequence, encoding requires substantial time and GPU compute resources. This makes the current framework unsuitable for real-time upstream applications like live video conferencing.
Additionally, because the encoder needs to look at all frames of a video sequence simultaneously to optimize the global representation, it introduces significant system latency. Future work will need to focus on accelerating the training pipeline and exploring frame-by-frame incremental updates to make INR codecs viable for real-time streaming ecosystems.
Summary updated