JointRF: End-to-End Joint Optimization for Dynamic Neural Radiance Field Representation and Compression

Zihan Zheng, Houqiang Zhong, Qiang Hu, Xiaoyun Zhang, Li Song, Ya Zhang, Yanfeng Wang

Abstract

Neural Radiance Field (NeRF) excels in photo-realistically static scenes, inspiring numerous efforts to facilitate volumetric videos. However, rendering dynamic and long-sequence radiance fields remains challenging due to the significant data required to represent volumetric videos. In this paper, we propose a novel end-to-end joint optimization scheme of dynamic NeRF representation and compression, called JointRF, thus achieving significantly improved quality and compression efficiency against the previous methods. Specifically, JointRF employs a compact residual feature grid and a coefficient feature grid to represent the dynamic NeRF. This representation handles large motions without compromising quality while concurrently diminishing temporal redundancy. We also introduce a sequential feature compression subnetwork to further reduce spatial-temporal redundancy. Finally, the representation and compression subnetworks are end-to-end trained combined within the JointRF. Extensive experiments demonstrate that JointRF can achieve superior compression performance across various datasets.

Introduction

While static NeRFs have revolutionized 3D scene reconstruction, dynamic NeRFs—or volumetric videos—face a massive data bottleneck. Capturing moving subjects over time requires representing changes in every frame, leading to enormous file sizes that are difficult to stream or store. Existing solutions often rely on canonical warping, which fails during complex motions or topological changes, or separate compression stages that lead to quality loss because the model is unaware of the downstream quantization.

JointRF addresses these challenges by integrating the compression process directly into the learning objective, optimizing both representation and bitrate in a single end-to-end pipeline.

Methodology

The JointRF framework introduces a novel approach to dynamic scene representation by borrowing concepts from traditional video coding standards.

Residual Feature Grids and GOF Structure

Instead of storing a full 3D grid for every frame, JointRF organizes frames into a Group of Frames (GOF). It establishes a base feature grid for a keyframe, and subsequent frames are represented using residual feature grids and coefficient feature grids. This "delta-based" approach effectively reduces temporal redundancy and handles large motions more robustly than geometric warping methods.

The JointRF architecture utilizes a Group of Frames (GOF) structure with residual and coefficient feature grids for efficient dynamic scene representation.

End-to-End Joint Optimization

To make the compression pipeline differentiable, JointRF employs:

  1. Simulated Quantization: Uniform noise is added during training to simulate the effects of rounding data to integers, making the model resilient to compression artifacts.
  2. Entropy Modeling: A probability model estimates the bitrate during training, allowing the loss function to penalize high-entropy (hard-to-compress) features.
  3. Sequential Compression Subnetwork: A dedicated module further reduces spatial-temporal redundancy within the feature grids.

Experimental Results

The authors evaluated JointRF on the ReRF Dataset (74 camera views of human subjects) and the DNA-Rendering Dataset (high-resolution human-centric rendering).

JointRF significantly outperforms state-of-the-art methods like ReRF and TiNeuVox. On the ReRF dataset, it achieved a BDBR (Bjøntegaard Delta Bit Rate) reduction of approximately 51% to 70%. For instance, JointRF achieved a PSNR of 33.23 dB with a model size of only 0.47 MB, whereas a standard per-frame method like INGP required 49.00 MB for a lower quality of 28.57 dB.

Qualitative and quantitative comparison between JointRF and ReRF. JointRF achieves higher PSNR at smaller file sizes.

Qualitative results demonstrate that JointRF preserves fine textures and details that are often blurred out by other compression-heavy methods.

Conclusion

JointRF demonstrates that joint optimization of representation and compression is key to practical volumetric video. By utilizing residual feature grids and a differentiable compression loop, it achieves superior visual fidelity at a fraction of the storage cost. Future work may focus on improving training speeds for long sequences and optimizing real-time decoding on edge devices to enable applications like immersive VR and volumetric telepresence.

Summary updated

Abstract

Neural Radiance Field (NeRF) excels in photo-realistically static scenes, inspiring numerous efforts to facilitate volumetric videos. However, rendering dynamic and long-sequence radiance fields remains challenging due to the significant data required to represent volumetric videos. In this paper, we propose a novel end-to-end joint optimization scheme of dynamic NeRF representation and compression, called JointRF, thus achieving significantly improved quality and compression efficiency against the previous methods. Specifically, JointRF employs a compact residual feature grid and a coefficient feature grid to represent the dynamic NeRF. This representation handles large motions without compromising quality while concurrently diminishing temporal redundancy. We also introduce a sequential feature compression subnetwork to further reduce spatial-temporal redundancy. Finally, the representation and compression subnetworks are end-to-end trained combined within the JointRF. Extensive experiments demonstrate that JointRF can achieve superior compression performance across various datasets.
BibTeX
@inproceedings{zheng2024jointrf,
  author = {Zihan Zheng and Houqiang Zhong and Qiang Hu and Xiaoyun Zhang and Li Song and Ya Zhang and Yanfeng Wang},
  title = {{JointRF: End-to-End Joint Optimization for Dynamic Neural Radiance Field Representation and Compression}},
  booktitle = {ICIP},
  year = {2024},
  doi = {10.1109/icip51287.2024.10647336},
}
Figure