Volumetric Video Compression Through Neural-based Representation

Yuang Shi, Ruoyu Zhao, Simone Gasparini, Géraldine Morin, Wei Tsang OoiMmve@mmsys 2024 · DOI

TL;DR

This paper introduces a novel neural-based approach for compressing volumetric video by representing each frame as a Neural Radiance Field (NeRF). By exploiting the temporal coherence between successive NeRFs, they achieve significant bitrate reduction with minimal loss in visual quality, outperforming traditional point cloud compression methods.

What Problem Does This Paper Address?

Volumetric video offers immersive 3D experiences, but its high data requirements pose a significant challenge for storage and streaming. Traditional methods like point cloud compression (PCC) struggle to maintain high visual fidelity, often introducing artifacts. While Neural Radiance Fields (NeRFs) have shown promise in generating photorealistic novel views of static scenes, their application to dynamic volumetric video compression, especially at low bitrates, remains an open problem. The core challenge is to compress these complex neural representations efficiently without sacrificing the quality of the rendered output.

Methods

The authors propose an end-to-end pipeline for volumetric video compression that leverages NeRFs. The key steps are:

  • NeRF-based Representation: Each frame of the volumetric video is represented by a NeRF. Instead of training each NeRF from scratch, they employ a transfer-learning approach where each subsequent NeRF is initialized with the weights of the previous frame's NeRF. This not only speeds up training but also encourages temporal coherence.
  • Temporal Compression: The core innovation lies in exploiting the temporal redundancy observed between consecutive NeRFs. Instead of encoding each NeRF independently, they encode only the differences (deltas) between successive NeRFs. The first NeRF is treated as a "key-frame" and encoded fully, while subsequent NeRFs are reconstructed by applying their decoded deltas to the previous frame's NeRF.
  • Model Compression: To further reduce the size of the NeRF models, they explore simplifying the neural architecture (network depth and width) and apply an exponent-based non-uniform quantization scheme to the delta values. Huffman coding is then used for final compression.

Datasets

The authors evaluate their method on four dynamic point cloud sequences:

  • RedAndBlack
  • Loot
  • Soldier
  • Thaidancer

The first three are from the 8iVFBv2 Dataset, and Thaidancer is from the 8iVSLF Dataset, which features a higher number of points and bitrate. They use the first 30 frames of each sequence for their experiments.

Key Insights and Ideas

  • Temporal Coherence in Latent Space: The paper demonstrates that temporal redundancy, evident in explicit representations like point clouds, also exists in the latent neural space of NeRFs. This is a crucial insight that enables efficient temporal compression.
  • NeRFs as a Compact Representation: By learning an implicit representation, NeRFs can capture complex geometry and appearance without being directly proportional to the scene's complexity, unlike explicit representations. This decoupling is key to achieving high compression ratios.
  • Transfer Learning for NeRF Sequences: Initializing subsequent NeRFs with previous ones significantly reduces training time and encourages temporal consistency, which is beneficial for compression.
  • Neural Architecture Search for Compression: The authors show that by carefully selecting the network architecture (depth and width), they can achieve a good trade-off between model size and rendering quality, further enhancing compression efficiency.

Key Results

  • Outperforms Geometry-Based PCC: The proposed method significantly outperforms geometry-based PCC codecs like G-PCC, achieving substantially better PSNR and SSIM at the same bitrate across all tested sequences. For example, they report up to 15.92 dB PSNR improvement on RedAndBlack.
  • Competitive with V-PCC: For high-bitrate volumetric videos, their method achieves comparable rate-distortion performance to V-PCC, a state-of-the-art PCC codec.
  • Effective Temporal Compression: The temporal compression technique effectively reduces NeRF size with only a minor increase in rendering distortion, highlighting its efficiency.
  • High Compression Ratios: The method achieves a wide range of compression ratios (35x to 442x for the Soldier sequence) with minimal degradation in SSIM (0.1% to 3.6% drop).
  • Qualitative Results: Sample images demonstrate that the compressed NeRF models can restore cloth texture details and maintain visual quality even at low bitrates.

Limitations and Future Work

  • User Studies: The authors acknowledge that while PSNR and SSIM provide objective metrics, user studies would offer a more comprehensive understanding of visual quality.
  • Frame-by-Frame Modeling: The current approach models volumetric videos frame by frame. Future work could explore dynamic NeRFs that represent entire dynamic scenes with a single model, potentially by splitting videos into groups of frames (GOFs) and applying temporal compression to these dynamic NeRFs.
  • Adaptability to Other NeRF Variants: While the original NeRF was used as a baseline, the methodology is adaptable to other NeRF variations, allowing for further exploration of compression trade-offs.

Summary updated

BibTeX
@inproceedings{shi2024volumetric,
  author = {Yuang Shi and Ruoyu Zhao and Simone Gasparini and Géraldine Morin and Wei Tsang Ooi},
  title = {{Volumetric Video Compression Through Neural-based Representation}},
  booktitle = {Mmve@mmsys},
  year = {2024},
  doi = {10.1145/3652212.3652220},
}
Figure