Light Field Compression Via Compact Neural Scene Representation

Jinglei Shi, Christine GuillemotICASSP 2023 · DOI

TL;DR

This paper introduces a novel light field compression method that leverages a compact neural scene representation. By optimizing a Neural Radiance Field (NeRF) with a low-rank constraint and then distilling it into a smaller network, the authors achieve significant compression while preserving high-quality, coherent view synthesis.

What Problem Does This Paper Address?

Light fields, which capture the direction and position of light rays, offer rich information for applications like digital refocusing and view synthesis. However, this richness comes at the cost of massive data volumes, posing significant challenges for storage, transmission, and display. Existing compression methods, such as those based on pseudo video sequences or transform-based approaches, often struggle with large baselines between views or require explicit depth estimation, limiting their effectiveness. While neural scene representations like NeRF offer a promising alternative, their large model sizes have hindered practical compression applications. This paper aims to bridge this gap by developing a highly compressed yet effective neural representation for light fields.

Methods

The core of this work lies in a multi-stage approach to compress a Neural Radiance Field (NeRF):

  • Low-Rank Constrained NeRF Initialization: The authors first train a NeRF model on the input light field views. Crucially, they optimize the NeRF's weights under a low-rank constraint using the Alternating Direction Method of Multipliers (ADMM). This step forces the network's weight matrices into a low-rank structure, preparing them for further compression.
  • Network Distillation with Tensor Train (TT) Decomposition: The low-rank weights are then decomposed into Tensor Train (TT) components. These TT components are used to initialize a distilled NeRF (DLR-NeRF) with significantly fewer parameters. This distillation process effectively transfers the scene information into a more compact representation.
  • Quantization-Aware Training: To further reduce the model size, the weights of the DLR-NeRF are quantized. The authors employ a quantization-aware training strategy, where they apply different quantization schemes for weights inside and outside the [-1, 1] interval. This involves using a uniform 16-bit quantizer for values outside this range and a non-uniform optimized quantizer (learned via k-means clustering) for values within it. This process results in the final compact representation, QDLR-NeRF.
  • Huffman Coding: Finally, the quantized weights are encoded using Huffman coding for further bitrate reduction.

Datasets

The paper evaluates their method on both real-world and synthetic light fields:

  • Real-world: Four Illum-captured scenes from the EPFL dataset: "Bikes," "Danger," "Stone Pillars Outside (SPO)," and "Fountain Vincent 2 (FV2)." These light fields have a resolution of 432 × 624 × 9 × 9.
  • Synthetic: Four scenes from the HCI dataset: "boxes," "sideboard," "cotton," and "dino." These light fields have a resolution of 512 × 512 × 9 × 9.

The authors note that they used estimated camera poses for the synthetic data, even though ground truth was available.

Key Insights and Ideas

  • Low-rank constraint as a precursor to compression: Forcing NeRF weights into a low-rank structure is a critical first step that enables subsequent efficient decomposition and distillation without significant quality loss.
  • Network distillation for parameter reduction: Decomposing the low-rank weights into TT components and using them to initialize a smaller network is an effective way to distill scene information into a compact form.
  • Quantization-aware training is crucial for stability: Directly quantizing NeRF weights can lead to rendering artifacts. The proposed quantization-aware training strategy, where quantization is integrated into the training loop, helps maintain stability and quality.
  • Differential quantization strategies: Recognizing that not all weight values are equally important, using different quantization schemes for values inside and outside [-1, 1] allows for finer control over compression and quality.
  • Coherent view synthesis: The method's ability to maintain high inter-view coherence is a significant advantage over methods that might rely on individual view compression, leading to a more consistent visual experience.

Key Results

The proposed QDLR-NeRF method demonstrates strong performance:

  • High Compression Ratio: The model size is reduced by a factor of 33 compared to the original NeRF, achieving a final model size of only 3.3% of the original.
  • Competitive Rate-Distortion Performance: The method outperforms HEVC, JPEG Pleno, and three learning-based methods (RLVC, HLVC, OpenDVC) in terms of BD-PSNR gains, achieving an average gain of 1.63 dB over HEVC.
  • Preservation of View Synthesis Capability: The method maintains the ability to generate coherent views across different viewpoints, a key advantage of neural scene representations.
  • Reduced Inter-View Errors: Qualitative results (error maps) show lower and more consistent errors across all views compared to other methods.

Limitations and Future Work

  • Artifacts with very low rank: The authors note that reconstructed light fields can be prone to artifacts when using a rank inferior to 40. This suggests a trade-off between extreme compression and visual fidelity.
  • NeRF limitations on real-world data: For Lytro-captured light fields, artifacts like vignetting and blurriness due to hardware limitations prevent the NeRF-based method from retrieving a perfectly precise scene.
  • Future Work: While not explicitly detailed, the paper's success in achieving high compression with acceptable quality suggests future work could explore even more aggressive quantization strategies, investigate different tensor decomposition techniques, or apply the method to other types of volumetric data. The authors also mention that their method is not limited by baseline or depth priors, hinting at its potential for datasets with challenging camera configurations.

Summary updated

BibTeX
@inproceedings{shi2023light,
  author = {Jinglei Shi and Christine Guillemot},
  title = {{Light Field Compression Via Compact Neural Scene Representation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095668},
}
Figure