Learning-based Spatial and Angular Information Separation for Light Field Compression

Jinglei Shi, Yihong Xu, Christine Guillemot

TL;DR

This paper introduces a novel neural network designed to explicitly separate spatial and angular information within light fields, a rich form of image data. By achieving this separation through specialized kernels, the network significantly improves light field compression performance and demonstrates potential for view synthesis by transferring learned angular features to new scenes.

What Problem Does This Paper Address?

Light fields capture intricate 4D information about a scene, encompassing both its static spatial layout and how it appears from different viewpoints (angular information). While this rich data is beneficial for tasks like depth estimation and object segmentation, it also leads to significant redundancy, making efficient compression a major challenge. Existing light field compression methods often struggle with quality variations across views, reliance on scene-specific priors, or limitations in handling wide-baseline captures. The core problem this paper tackles is how to effectively disentangle the static spatial content from the view-dependent angular variations to achieve more efficient and robust light field compression.

Methods

The authors propose a novel neural network architecture designed for explicit spatial and angular information separation. The key innovation lies in the use of two distinct types of kernels:

  • Spatial Kernels: These are shared across all Sub-Aperture Images (SAIs) of a light field, capturing the static, viewpoint-invariant spatial features of the scene.
  • Angular Kernels: These are view-specific and capture the variations in appearance due to changes in perspective.

To further enhance efficiency and representation capability without ballooning parameter counts, they introduce two mechanisms:

  • Angular Kernel Allocation: Instead of having unique kernels for every single view, angular information is represented by kernels allocated to horizontal and vertical directions. This significantly reduces the number of parameters, especially for high angular resolution light fields.
  • Kernel Tensor Decomposition: Both spatial and angular kernels are decomposed into shared bases and coefficient volumes. This technique, inspired by prior work in network compression, allows for a more compact representation of the kernel parameters.

The network is trained by taking uniform noise as input and is supervised by SAIs of the target light field, learning to reconstruct these SAIs by implicitly representing the light field through its learned parameters. Quantization-aware training (QAT) is also employed to further reduce model size for compression.

Datasets

The paper evaluates its method on a combination of synthetic and real-world light field datasets:

  • Synthetic Datasets: 'boxes', 'sideboard', 'cotton', 'dino' (rendered using 3D graphics software). These datasets feature wide baselines and low noise.
  • Real-World Datasets: 'bikes', 'danger', 'FountainVincent2', 'StonePillarsOutside' (captured with Lytro Illum cameras). These datasets are characterized by narrow baselines, noise, and vignetting artifacts.

Key Insights and Ideas

  • Explicit Separation is Key: The paper strongly advocates for the explicit separation of spatial and angular information as a fundamental principle for effective light field compression. This contrasts with methods that implicitly handle these aspects.
  • Complementary Kernels Work: The design of shared spatial kernels and view-specific angular kernels naturally enforces the separation during training, leading to a more robust representation.
  • Angular Kernels are Transferable: A significant finding is that the learned angular kernels can be "transferred" to new scenes. By freezing these kernels and fine-tuning the spatial kernels on a new light field, the network can effectively synthesize new views, demonstrating the successful disentanglement of information and opening avenues for view synthesis.
  • Compactness Through Design: The proposed kernel allocation and tensor decomposition mechanisms are crucial for achieving a compact network, making it suitable for compression applications where efficiency is paramount.

Key Results

  • State-of-the-Art Compression: The proposed method achieves superior rate-distortion performance compared to several state-of-the-art methods, including HEVC, JPEG-Pleno, HLVC, RLVC, DDLF, and QDLR-NeRF, across various light fields. For instance, it shows significant BD-PSNR gains over the HEVC baseline, with an average gain of 2.71 dB.
  • Robustness: The method demonstrates remarkable robustness across different types of light fields (synthetic vs. real-world, wide vs. narrow baseline, noisy vs. clean), outperforming methods that struggle with specific challenges like wide baselines or view inconsistencies.
  • Efficient Memory and Decoding: The network exhibits significantly lower memory usage and faster decoding times per SAI compared to other learning-based methods like DDLF, HLVC, and RLVC, due to its view-by-view processing and fully convolutional design.
  • View Synthesis Potential: Experiments show that transferred angular kernels can be used to render high-quality views for new scenes, validating the effectiveness of the spatial-angular separation.

Limitations and Future Work

  • Code Release: The authors plan to release the code upon acceptance, which is a positive step for future research but not a limitation of the current work.
  • Future Work: The paper suggests that the successful transfer of angular information for view synthesis opens up exciting possibilities for further research in synthesis methods via information transfer. This implies that exploring more advanced view synthesis techniques using the separated components is a promising direction.

Summary updated

Abstract

Light fields are a type of image data that capture both spatial and angular scene information by recording light rays emitted by a scene from different orientations. In this context, spatial information is defined as features that remain static regardless of perspectives, while angular information refers to features that vary between viewpoints. We propose a novel neural network that, by design, can separate angular and spatial information of a light field. The network represents spatial information using spatial kernels shared among all Sub-Aperture Images (SAIs), and angular information using sets of angular kernels for each SAI. To further improve the representation capability of the network without increasing parameter number, we also introduce angular kernel allocation and kernel tensor decomposition mechanisms. Extensive experiments demonstrate the benefits of information separation: when applied to the compression task, our network outperforms other state-of-the-art methods by a large margin. And angular information can be easily transferred to other scenes for rendering dense views, showing the successful separation and the potential use case for the view synthesis task. We plan to release the code upon acceptance of the paper to encourage further research on this topic.
BibTeX
@misc{shi2023learning,
  author = {Jinglei Shi and Yihong Xu and Christine Guillemot},
  title = {{Learning-based Spatial and Angular Information Separation for Light Field Compression}},
  year = {2023},
  eprint = {2304.06322},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.06322},
}
Figure