A Generative Model for Generic Light Field Reconstruction

Paramanand Chandramouli, Kanchana Vaishnavi Gandikota, Andreas Görlitz, Andreas Kolb, Michael MöllerIEEE TPAMI 2022 · DOI · OpenReview

TL;DR

This paper introduces a novel generative model for light field (LF) reconstruction, treating LF patches as data points on a lower-dimensional manifold. By conditioning a variational autoencoder on the central view, they create a flexible prior that can be integrated into energy minimization frameworks for various LF reconstruction tasks, outperforming traditional methods and matching end-to-end networks in robustness.

What Problem Does This Paper Address?

High-quality light field (LF) images are crucial for applications like 3D scene rendering and geometry estimation. However, acquiring them is challenging due to hardware limitations, often resulting in sparse or distorted data. Traditional reconstruction methods rely on specific observation models, making them inflexible. Purely data-driven approaches, while powerful, require retraining for each new task or observation model. This paper addresses the need for a flexible and robust LF reconstruction method that can leverage the power of generative models without sacrificing adaptability.

Methods

The core of this work is a Conditional Variational Autoencoder (CVAE) trained on LF patches. Instead of modeling the entire 4D light field, they focus on patches, which are more amenable to training and generalize better. The CVAE is conditioned on the central view (CV) of the LF patch. This means the encoder learns to map an LF patch and its corresponding CV to a latent representation, and the decoder (generator) reconstructs the LF patch using this latent code and the CV features. This approach allows the model to capture parallax and occlusion effects.

The key innovation is using this trained CVAE as a generative prior within an energy minimization framework. For reconstruction tasks, they optimize a latent code (and potentially the central view itself if unavailable) to minimize the difference between the observed measurements and the model's prediction, guided by the generative prior. This hybrid approach combines the flexibility of model-based methods with the representational power of deep learning.

Datasets

The authors trained their CVAE models on LF patches extracted from:

  • The training set used by Kalantari et al. [4].
  • The training set used in CNN-based depth estimation for light fields by Heber et al. [57].
  • The training set used in encoder-decoder-based light field intrinsics [39].

These datasets were chosen for their diverse samples, including occlusions and specular reflections. For evaluation, they used:

  • The test set of Kalantari et al. [4] for 7x7 view synthesis.
  • The New HCI dataset [58] (scenes: "Dino", "Kitchen", "Medieval 2", "Tower") for 5x5 view synthesis and spatial-angular super-resolution.
  • Real light field data from [6] for coded aperture reconstruction.

Key Insights and Ideas

  • LF Patches as Manifolds: The paper posits that LF patches lie on a lower-dimensional manifold, making patch-based generative modeling a viable and efficient approach.
  • Central View as a Powerful Prior: Conditioning the generative model on the central view is highly effective, allowing the model to learn scene geometry and parallax implicitly.
  • Generative Prior for Flexibility: Integrating a learned generative model as a prior into an energy minimization framework provides a powerful and adaptable solution for various LF reconstruction problems, unlike fixed end-to-end networks.
  • Robustness to Distortions: The generative prior, combined with the optimization framework, makes the reconstruction process inherently more robust to noise and missing data compared to purely data-driven methods.

Key Results

  • View Synthesis: Achieved performance close to state-of-the-art CNN-based methods like [28], with only a small PSNR drop (1.4 dB with overlapping patches) when upsampling from 3x3 to 7x7 views. Outperformed traditional depth-based methods significantly.
  • Spatial-Angular Super-Resolution: Outperformed the dictionary-based approach [2] by over 2 dB without overlapping patches and over 4 dB with overlapping patches.
  • Coded Aperture Reconstruction: Achieved comparable results to end-to-end trained models [6] and significantly outperformed model-based approaches [2], especially when using overlapping patches.
  • Robustness: Demonstrated remarkable robustness to various corruptions (Gaussian noise, salt-and-pepper noise, pixel drops) where end-to-end networks like [28] failed or showed significant performance degradation. For instance, with 50% pixel drops, [28] failed, while their method remained robust.

Limitations and Future Work

  • Disparity Limit: The current model struggles with very large disparities (greater than two pixels) due to the limited spatial extent (25x25 pixels) of the LF patches used for training.
  • Future Work: The authors suggest that increasing the spatial extent of LF patches used for training could address the disparity limitation. They believe their work is a foundational step for further research into generative light field models.

Summary updated

Abstract

Recently deep generative models have achieved impressive progress in modeling the distribution of training data. In this work, we present for the first time a generative model for 4D light field patches using variational autoencoders to capture the data distribution of light field patches. We develop a generative model conditioned on the central view of the light field and incorporate this as a prior in an energy minimization framework to address diverse light field reconstruction tasks. While pure learning-based approaches do achieve excellent results on each instance of such a problem, their applicability is limited to the specific observation model they have been trained on. On the contrary, our trained light field generative model can be incorporated as a prior into any model-based optimization approach and therefore extend to diverse reconstruction tasks including light field view synthesis, spatial-angular super resolution and reconstruction from coded projections. Our proposed method demonstrates good reconstruction, with performance approaching end-to-end trained networks, while outperforming traditional model-based approaches on both synthetic and real scenes. Furthermore, we show that our approach enables reliable light field recovery despite distortions in the input.
BibTeX
@article{chandramouli2022generative,
  author = {Paramanand Chandramouli and Kanchana Vaishnavi Gandikota and Andreas Görlitz and Andreas Kolb and Michael Möller},
  title = {{A Generative Model for Generic Light Field Reconstruction}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  doi = {10.1109/tpami.2020.3039841},
}
Figure