JOINTRF: End-To-End Joint Optimization for Dynamic Neural Radiance Field Representation and Compression

Zihan Zheng, Houqiang Zhong, Qiang Hu, Xiaoyun Zhang, Li Song, Ya Zhang, Yanfeng Wang

This paper presents JointRF, a framework designed to make volumetric video—the 3D equivalent of a movie—practical for storage and streaming. By jointly optimizing how a dynamic scene is represented and how it is compressed in a single end-to-end pipeline, the authors achieve high-fidelity reconstructions at a fraction of the storage cost of previous methods.

TL;DR

JointRF is an end-to-end framework that jointly optimizes the representation and compression of dynamic Neural Radiance Fields (NeRFs). By utilizing a novel residual feature grid and a differentiable compression pipeline, it achieves state-of-the-art volumetric video quality with up to 70% better compression efficiency than existing methods.

What Problem Does This Paper Address?

While static NeRFs have revolutionized 3D scene reconstruction, dynamic NeRFs (volumetric videos) face a massive data bottleneck. Capturing a moving person or scene over time requires representing changes in every frame, leading to enormous file sizes that are nearly impossible to stream or store efficiently.

Existing solutions usually fall into two camps:

  1. Canonical Warping: Attempting to "warp" every frame back to a single static template. This fails when there is significant motion or when the topology changes (e.g., a person putting on a jacket).
  2. Separate Compression: Training a model first and then applying traditional image compression to the resulting data. Because the model wasn't "aware" it was going to be compressed, this often leads to significant quality loss or suboptimal file sizes.

JointRF addresses these gaps by making the compression process part of the learning objective itself.

Methods

The core of JointRF lies in two main innovations: a compact representation and a differentiable optimization loop.

  • Residual Feature Grids: Instead of storing a full 3D grid for every frame, JointRF uses a Group of Frames (GOF) structure similar to traditional video coding (like H.264). It starts with a "keyframe" (a base feature grid). Subsequent frames only store a residual feature grid—essentially just the "delta" or the changes needed to update the base grid. This naturally handles large motions and reduces temporal redundancy.
  • End-to-End Joint Optimization: This is the "secret sauce." To make compression differentiable, the authors use:
  • Simulated Quantization: They add uniform noise during training to mimic the effects of rounding data to integers (quantization), making the model robust to the "errors" introduced by compression.
  • Entropy Modeling: They use a probability model to estimate the bitrate during training. This allows the loss function to penalize the model not just for being blurry, but also for being "hard to compress."
  • Sequential Compression Subnetwork: A dedicated module that further squeezes spatial-temporal redundancy out of the feature grids.

Datasets

The authors evaluated JointRF on two primary benchmarks:

  1. ReRF Dataset: Contains 74 camera views of human subjects at 1920x1080 resolution.
  2. DNA-Rendering Dataset: A high-resolution (2048x2448) dataset focused on complex human-centric rendering with 48 views.

Key Insights and Ideas

  • The Power of "Jointness": The paper demonstrates that when a model "knows" it will be compressed, it learns to organize its internal features into a low-entropy state. This makes the final bitstream much smaller without sacrificing visual detail.
  • Residuals > Warping: By focusing on residuals in the feature space rather than geometric warping in 3D space, the model is much more resilient to complex motions that typically break "canonical space" methods.
  • Borrowing from Video Standards: Applying the concept of GOFs (Group of Frames) to neural feature grids proves to be a highly effective bridge between the worlds of deep learning and traditional signal processing.

Key Results

JointRF significantly outperforms previous state-of-the-art methods like ReRF and TiNeuVox:

  • Compression Efficiency: On the ReRF dataset, JointRF achieved a BDBR (Bjøntegaard Delta Bit Rate) reduction of approximately 51% to 70%. This means it can provide the same quality as previous methods while using less than half the data.
  • Storage vs. Quality: For the ReRF dataset, JointRF achieved a PSNR of 33.23 dB (higher is better) with a model size of only 0.47 MB. In comparison, a standard per-frame method (INGP) required 49.00 MB to achieve a lower quality of 28.57 dB.
  • Visual Fidelity: Qualitative results show that JointRF preserves fine details (like textures on clothing) that other compression-heavy methods blur out.

Limitations and Future Work

While the results are impressive, the paper notes or implies a few areas for further exploration:

  • Sequential Training: The model is trained sequentially per frame within a GOF, which might limit training speed for extremely long sequences.
  • Real-time Constraints: While the representation is "streamable," the paper does not focus extensively on the real-time decoding speeds on mobile or edge devices.
  • Future Directions: The authors suggest this approach lays the foundation for practical volumetric telepresence and immersive VR applications where bandwidth is at a premium.

Summary updated

Abstract

Neural Radiance Field (NeRF) excels in photo-realistically static scenes, inspiring numerous efforts to facilitate volumetric videos. However, rendering dynamic and long-sequence radiance fields remains challenging due to the significant data required to represent volumetric videos. In this paper, we propose a novel end-to-end joint optimization scheme of dynamic NeRF representation and compression, called JointRF, thus achieving significantly improved quality and compression efficiency against the previous methods. Specifically, JointRF employs a compact residual feature grid and a coefficient feature grid to represent the dynamic NeRF. This representation handles large motions without compromising quality while concurrently diminishing temporal redundancy. We also introduce a sequential feature compression subnetwork to further reduce spatial-temporal redundancy. Finally, the representation and compression subnetworks are end-to-end trained combined within the JointRF. Extensive experiments demonstrate that JointRF can achieve superior compression performance across various datasets.
BibTeX
@inproceedings{zheng2024jointrf,
  author = {Zihan Zheng and Houqiang Zhong and Qiang Hu and Xiaoyun Zhang and Li Song and Ya Zhang and Yanfeng Wang},
  title = {{JOINTRF: End-To-End Joint Optimization for Dynamic Neural Radiance Field Representation and Compression}},
  booktitle = {ICIP},
  year = {2024},
  doi = {10.1109/icip51287.2024.10647336},
}

Loading related content...

Figure