Abstract: Light field applications, especially light field rendering and depth estimation, developed rapidly in recent years. While state-of-the-art light field rendering methods handle semi-transparent and reflective objects well, depth estimation methods either ignore these cases altogether or only deliver a weak performance. We argue that this is due current methods only considering a single “true” depth, even when multiple objects at different depths contributed to the color of a single pixel. Based on the simple idea of outputting a posterior depth distribution instead of only a single estimate, we develop and explore several different deep-learning-based approaches to the problem. Additionally, we contribute the first “multimodal light field depth dataset” that contains the depths of all objects which contribute to the color of a pixel. This allows us to supervise the multimodal depth prediction and also validate all methods by measuring the KL divergence of the predicted posteriors. With our thorough analysis and novel dataset, we aim to start a new line of depth estimation research that overcomes some of the longstanding limitations of this field.
The Multimodality Problem
In standard computer vision, depth estimation is typically treated as a regression problem where each pixel is assigned a single "true" distance value. However, light fields—which capture light from multiple directions—reveal that this assumption is often flawed. In complex scenes, a single pixel may capture light from multiple sources: the edge of a foreground object and the background behind it, or a semi-transparent surface and the reflection it carries.
When models are forced to predict a single value for these multimodal pixels, they often "average" the depths, resulting in blurry edges or values that correspond to empty space. This paper proposes moving toward posterior depth distributions, allowing models to estimate the probability of multiple depths contributing to a single pixel.
Proposed Architectures
The authors explore three distinct deep-learning architectures to output these distributions, using a modified EPI-Net as their backbone:
- Unimodal Posterior Regression (UPR): Instead of just predicting depth ($\mu$), the network also predicts the "width" ($b$) of a Laplacian distribution. This allows the model to express uncertainty: a high $b$ value means the model knows its prediction might be unreliable.
- EPI-Shift-Ensemble (ESE): This method exploits the geometry of light fields. By "shearing" the input data (EPI-Shift), the model focuses on different depth layers. By running multiple shifted passes and ensembling them with a specialized masked loss, the model can identify multiple distinct depth peaks.
- Discrete Posterior Prediction (DPP): The authors turn depth estimation into a classification problem. They divide the possible depth range into 108 discrete "bins." The network outputs a probability for each bin, allowing it to naturally represent complex, multi-peaked distributions (e.g., 30% chance of foreground, 70% chance of background).
The Multimodal Light Field Depth Dataset
A significant contribution of this work is the Multimodal Light Field Depth Dataset. Existing benchmarks typically provide only one ground-truth depth per pixel. This new synthetic dataset of 110 indoor scenes provides the exact depth, color, and opacity ($\eta$) for every object contributing to a pixel’s color. This allows for the first-ever supervised training and validation of multimodal depth predictions using metrics like Kullback-Leibler (KL) divergence.
Key Results
The evaluation highlights a trade-off between simplicity and multimodal accuracy. The Discrete Posterior Prediction (DPP) method proved most effective at capturing multiple depths, achieving the lowest KL divergence in multimodal areas. Meanwhile, the UPR method provided the most reliable uncertainty scores for unimodal regions.
As shown in the figure above, while UPR (orange) is limited to a single peak, the ESE method (green) can successfully recover multiple modes present in the ground truth (blue), preventing the "averaging" error common in standard regression.
Conclusion and Future Directions
This research demonstrates that predicting depth distributions rather than single values significantly improves performance in challenging visual scenarios. While methods like ESE are computationally intensive, requiring multiple forward passes, they offer scalability to wide-baseline light fields. Future work will focus on extending these models to handle more complex physical phenomena, such as non-Lambertian reflections, and reducing the computational overhead of ensemble-based methods.
Summary updated