Generative Models for Generic Light Field Reconstruction

Paramanand Chandramouli, Kanchana Vaishnavi Gandikota, Andreas Görlitz, Andreas Kolb, Michael MoellerarXiv 2020 · OpenReview

Abstract

Recently deep generative models have achieved impressive progress in modeling the distribution of training data. In this work, we present for the first time a generative model for 4D light field patches using variational autoencoders to capture the data distribution of light field patches. We develop a generative model conditioned on the central view of the light field and incorporate this as a prior in an energy minimization framework to address diverse light field reconstruction tasks. While pure learning-based approaches do achieve excellent results on each instance of such a problem, their applicability is limited to the specific observation model they have been trained on. On the contrary, our trained light field generative model can be incorporated as a prior into any model-based optimization approach and therefore extend to diverse reconstruction tasks including light field view synthesis, spatial-angular super resolution and reconstruction from coded projections. Our proposed method demonstrates good reconstruction, with performance approaching end-to-end trained networks, while outperforming traditional model-based approaches on both synthetic and real scenes. Furthermore, we show that our approach enables reliable light field recovery despite distortions in the input.

Introduction

Capturing high-quality light fields (LF)—which record both the intensity and direction of light rays—presents a significant data challenge. Hardware constraints typically force a trade-off between spatial resolution (sharpness) and angular resolution (smoothness of viewpoints).

A 4D light field represented as a grid of 2D viewpoints.

To address this, researchers have traditionally relied on two methods: model-based optimization, which is flexible but relies on hand-crafted priors, and end-to-end deep learning, which is powerful but task-specific. A network trained for 2x super-resolution, for instance, cannot perform 4x super-resolution without retraining. This paper introduces a generic generative model that acts as a flexible prior, capable of solving various reconstruction tasks within a single framework.

The CVAE Generative Prior

The core of this approach is a Conditional Variational Autoencoder (CVAE) trained on 4D light field patches. Modeling an entire high-resolution light field is computationally prohibitive, so the authors focus on small patches (e.g., 25x25 pixels spatially and 5x5 angularly).

The CVAE architecture for light field patch generation. (a) Input light field, (b) central view patch selection, (c) feature extraction from the central view (CVF), and (d) the CVAE pipeline where the generator $G_1$ is conditioned on CVF to produce the reconstructed patch.

To simplify the learning process, the model is conditioned on the central view of the light field. This allows the encoder to learn the "residual" information—the parallax and occlusions—relative to that central image. The model utilizes a Wasserstein framework with a Maximum Mean Discrepancy (MMD) penalty to more accurately capture the true distribution of light field data compared to standard VAE losses.

Reconstruction via Optimization

Instead of a simple forward pass, reconstruction is treated as an energy minimization problem. The goal is to find a latent code $z$ that, when passed through the generator, produces a light field matching the limited observations. This is expressed as:

$$\min_z \| \Phi(G(z|c)) - y \|^2 + \lambda \|z\|^2$$

where $\Phi$ is the observation operator, $G$ is the generator, $c$ is the condition (central view), and $y$ is the observed data. By simply changing the operator $\Phi$, the same pre-trained model can solve diverse inverse problems, such as view synthesis, spatial-angular super-resolution, and reconstruction from coded projections.

Key Results and Robustness

The model was evaluated on prominent datasets including the Kalantari real-world scenes and the New HCI synthetic scenes.

  • Competitive Performance: In view synthesis (upsampling from 3x3 to 7x7 views), the model achieved an average PSNR of 39.77 dB, coming within 1.4 dB of state-of-the-art end-to-end networks while remaining far more flexible.
  • Outperforming Traditional Methods: It consistently beat traditional dictionary-based methods by 2 to 4 dB across various tasks.
  • Noise Resilience: In a "stress test" with 50% pixel loss, end-to-end baselines failed (PSNR dropped to 13.6 dB), while the generative approach maintained a usable reconstruction at 33.39 dB.

Limitations and Future Work

A primary limitation is the disparity range. Because the model operates on small 25x25 patches, it struggles when the shift between adjacent views exceeds two pixels, as the patch lacks sufficient context to capture the motion. Future work may involve scaling the architecture to handle larger patches or implementing a multi-scale approach to capture both fine details and large-scale scene motion.

Summary updated

Abstract

Recently deep generative models have achieved impressive progress in modeling the distribution of training data. In this work, we present for the first time a generative model for 4D light field patches using variational autoencoders to capture the data distribution of light field patches. We develop a generative model conditioned on the central view of the light field and incorporate this as a prior in an energy minimization framework to address diverse light field reconstruction tasks. While pure learning-based approaches do achieve excellent results on each instance of such a problem, their applicability is limited to the specific observation model they have been trained on. On the contrary, our trained light field generative model can be incorporated as a prior into any model-based optimization approach and therefore extend to diverse reconstruction tasks including light field view synthesis, spatial-angular super resolution and reconstruction from coded projections. Our proposed method demonstrates good reconstruction, with performance approaching end-to-end trained networks, while outperforming traditional model-based approaches on both synthetic and real scenes. Furthermore, we show that our approach enables reliable light field recovery despite distortions in the input.
BibTeX
@misc{chandramouli2020generative,
  author = {Paramanand Chandramouli and Kanchana Vaishnavi Gandikota and Andreas Görlitz and Andreas Kolb and Michael Moeller},
  title = {{Generative Models for Generic Light Field Reconstruction}},
  year = {2020},
  eprint = {2005.06508},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.06508},
}
Figure