Towards Multimodal Depth Estimation from Light Fields

Titus Leistner, Radek Mackowiak, Lynton Ardizzone, Ullrich Köthe, Carsten Rother

Abstract

Light field applications, particularly rendering and depth estimation, have seen significant advancements. However, while state-of-the-art rendering handles semi-transparent and reflective surfaces effectively, depth estimation methods often fail or deliver weak performance in these scenarios. This paper argues that the limitation stems from the assumption of a single "true" depth per pixel. We propose a shift toward predicting posterior depth distributions and introduce three deep-learning architectures to address this. Furthermore, we contribute the first "multimodal light field depth dataset," which provides ground truth for all objects contributing to a pixel's color. Our analysis demonstrates that modeling depth as a distribution allows for superior handling of occlusions and transparency, measured via KL divergence and uncertainty quantification.

Introduction

In traditional computer vision, depth estimation is treated as a regression problem: for every pixel $(x, y)$, the goal is to find a single distance $z$. This approach assumes that each pixel corresponds to exactly one point in 3D space. However, real-world scenes are often more complex. Consider a screen door or a glass window: a single pixel may capture light from both the foreground (the mesh or glass) and the background (the scene behind it).

Light field technology, which captures both the intensity and direction of light, is uniquely suited to resolve these ambiguities. Yet, current depth estimation algorithms struggle with these "corner cases," often producing blurry depth maps or incorrect averages. This paper addresses these limitations by moving from single-value estimates to Bayesian posteriors.

Methodology

We adapt a state-of-the-art backbone, EPI-Net, into three distinct architectures designed to predict a full probability distribution of depths per pixel:

  1. Unimodal Posterior Regression (UPR): This model outputs the parameters of a Laplacian distribution (mean and width). While it remains unimodal, the predicted width serves as a direct measure of uncertainty, indicating how much the model trusts its estimate.
  2. EPI-Shift-Ensemble (ESE): This method leverages the geometric properties of light fields. By "shearing" the input data—a process known as EPI-shifting—the model observes the scene from different virtual focal planes. An ensemble of these shifts allows the model to detect multiple depth "modes" or peaks.
  3. Discrete Posterior Prediction (DPP): This approach discretizes the depth range into 108 bins and treats depth estimation as a classification task. By assigning a probability to each bin, the model can naturally represent any number of depth peaks without assuming a specific mathematical distribution. This is the most flexible of the three proposed methods.

The Multimodal Light Field Depth Dataset

A significant hurdle in multimodal depth research is the lack of ground-truth data. To solve this, we developed a new synthetic dataset consisting of 110 indoor scenes. Unlike existing benchmarks, this dataset provides the depth and opacity ($\eta$) for every object contributing to a pixel’s color.

A key conceptual bridge in our work is treating an object's opacity as the probability ($p$) of that depth in the posterior distribution. For example, if a pixel is composed of 30% foreground and 70% background, the ground-truth distribution should feature two peaks with corresponding weights. This allows for the use of KL Divergence as a primary metric for validation.

Key Insights and Results

Our evaluation reveals several critical findings regarding the nature of depth estimation:

  • The Failure of Mean Estimates: Standard regression models minimizing Mean Squared Error (MSE) are mathematically biased toward predicting the average depth in ambiguous areas. This often results in a predicted distance where no physical object exists.
  • Multimodal Accuracy: In scenes with overlapping objects, the DPP method significantly outperformed baselines, achieving a KL Divergence of 2.987 compared to the baseline's 8.514. Its ability to remain "undecided" between two distant objects prevents the blurring seen in traditional models.
  • Uncertainty Quantification: The UPR method proved most effective at identifying its own errors. Through "sparsification curves," we demonstrated that removing pixels with high predicted uncertainty leads to a much faster reduction in error compared to other methods.
  • Efficiency vs. Performance: While the ESE ensemble is highly accurate, it requires 70 forward passes, making it computationally expensive. The DPP model provides comparable multimodal benefits in a single pass, offering a more practical solution for real-time applications.

Conclusion and Future Work

By shifting the paradigm from single-value regression to posterior distribution prediction, we can overcome long-standing limitations in light field depth estimation. While our current dataset is synthetic and focuses on alpha-transparency, it provides a foundation for future research into more complex optical phenomena like curved glass reflections and refraction. Future work will focus on calibrating the overconfidence observed in discrete classifiers and extending these multimodal approaches to real-world captured light fields.

Summary updated

Abstract

Light field applications, especially light field rendering and depth estimation, developed rapidly in recent years. While state-of-the-art light field rendering methods handle semi-transparent and reflective objects well, depth estimation methods either ignore these cases altogether or only deliver a weak performance. We argue that this is due current methods only considering a single "true" depth, even when multiple objects at different depths contributed to the color of a single pixel. Based on the simple idea of outputting a posterior depth distribution instead of only a single estimate, we develop and explore several different deep-learning-based approaches to the problem. Additionally, we contribute the first "multimodal light field depth dataset" that contains the depths of all objects which contribute to the color of a pixel. This allows us to supervise the multimodal depth prediction and also validate all methods by measuring the KL divergence of the predicted posteriors. With our thorough analysis and novel dataset, we aim to start a new line of depth estimation research that overcomes some of the long-standing limitations of this field.
BibTeX
@inproceedings{leistner2022towards,
  author = {Titus Leistner and Radek Mackowiak and Lynton Ardizzone and Ullrich Köthe and Carsten Rother},
  title = {{Towards Multimodal Depth Estimation from Light Fields}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  url = {https://openreview.net/forum?id=ATOq2YgAiL},
}
Figure