Abstract
Depth map estimation is a crucial task in computer vision, and new approaches have recently emerged taking advantage of light fields, as this new imaging modality captures much more information about the angular direction of light rays compared to common approaches based on stereoscopic images or multi-view. In this paper, we propose a novel depth estimation method from light fields based on existing optical flow estimation methods. The optical flow estimator is applied on a sequence of images taken along an angular dimension of the light field, which produces several disparity map estimates. Considering both accuracy and efficiency, we choose the feature flow method as our optical flow estimator. Thanks to its spatio-temporal edge-aware filtering properties, the different disparity map estimates that we obtain are very consistent, which allows a fast and simple aggregation step to create a single disparity map, which can then converted into a depth map. Since the disparity map estimates are consistent, we can also create a depth map from each disparity estimate, and then aggregate the different depth maps in the 3D space to create a single dense depth map.
Introduction
Estimating depth from light fields is a classic computer vision challenge. Unlike standard 2D images, a 4D light field captures both the intensity and the direction of light rays. This extra "angular" information allows for highly accurate depth maps, but processing it efficiently is difficult. Existing methods generally fall into two camps: local methods, which are fast but often produce noisy or inconsistent results, and global methods, which produce smooth depth maps but are computationally expensive.
The core insight of this work is that the angular dimension of a light field can be treated like the time dimension in a video. By enforcing consistency during a specialized filtering stage, the "hard work" of depth estimation is done early, simplifying the final aggregation and drastically reducing processing time.
Proposed Method
The proposed pipeline re-imagines the light field structure as a sequence of views to leverage optical flow tools designed for video consistency.
The method consists of three main stages:
- 3D Volume Extraction: A sequence of sub-aperture images (views) along one angular dimension is extracted and treated as a "video" sequence.
- Optical Flow Estimation:
- Initialization: Coarse-to-fine Patch Matching (CPM) is used to find sparse correspondences between views. This is faster and more robust than traditional SIFT-based methods.
- Feature Flow: A spat-angular edge-aware filter (domain transform filter) is applied to diffuse sparse matches into a dense flow field. Operating across the angular dimension ensures that disparity estimates are consistent from one view to the next.
- Aggregation: Due to the high consistency of the disparity maps, heavy global optimization is bypassed. A simple median filter followed by a single-step variational energy minimization produces the final depth map.
Results and Discussion
The approach was evaluated using the HCI 4D Light Field Dataset, a standard benchmark providing synthetic light fields with ground truth depth.
The proposed method achieved a performance comparable to state-of-the-art global methods but with significantly lower running times. It effectively sits in the "Goldilocks" zone of the error-vs-time trade-off. Qualitatively, the method shows excellent edge preservation, particularly in complex scenes where fine details are often blurred by other algorithms.
Furthermore, because the method generates consistent depth maps for every view in the sequence, these can be projected and fused in 3D space. This technique produced "super-dense" point clouds with over 21 million points, offering a level of detail far exceeding standard methods that typically produce around 260,000 points.
Conclusion
This paper demonstrates that treating light field angular views as a video sequence allows for fast and accurate depth estimation using optical flow. While local filtering can occasionally be less smooth than global optimization in extremely noisy scenes, the speed and density of the resulting point clouds make it a compelling approach for practical applications. Future work will explore different spatio-angular filtering techniques and testing on real-world data from lenslet cameras.
Summary updated