FPattNet: A Multi-Scale Feature Fusion Network with Occlusion Awareness for Depth Estimation of Light Field Images

Min Xiao, Chen Lv, Xiaomin LiuSensors 2023 · DOI

TL;DR

This paper introduces FPattNet, a novel deep learning network for estimating depth from light field images. FPattNet tackles the challenge of occluded and textureless regions by dynamically selecting the most informative views using an attention mechanism and employing a multi-scale feature fusion strategy for enhanced contextual understanding.

What Problem Does This Paper Address?

Accurate depth estimation is crucial for many computer vision applications, from autonomous driving to augmented reality. Light field cameras, by capturing light from multiple viewpoints, offer a rich source of information for depth estimation. However, existing methods often struggle with challenging scenarios like occlusions (where one object hides another) and textureless regions (where there's little visual detail to guide depth inference). These methods typically rely on a fixed set of views, which may not be optimal for every scene. The paper aims to improve depth estimation accuracy by intelligently selecting and fusing information from all available views, specifically addressing the limitations posed by occlusions and lack of texture.

Methods

FPattNet is built upon a convolutional neural network architecture. The core innovations include:

  • Feature Extraction and Feature Pyramid Network (FPN): The network uses residual blocks for initial feature extraction. To overcome the limited receptive field of standard CNNs, especially in challenging regions, an FPN is incorporated. This FPN captures global contextual information by fusing features from different levels of the network, allowing for a more comprehensive understanding of the scene at multiple scales.
  • Cost Volume Construction: To effectively explore the disparity space across multiple viewpoints, a cost volume is constructed. This involves shifting feature maps from different views based on candidate disparities and concatenating them.
  • Attentional View Selection Module: Recognizing that not all views contribute equally to depth estimation, an attention mechanism (specifically, the Convolutional Block Attention Module - CBAM) is employed. This module dynamically assigns weights to each view, allowing the network to focus on the most informative ones for a given scene. This adaptive weighting helps improve accuracy and reduce computational costs.
  • Disparity Regression and Loss: The aggregated cost volume is then used for disparity regression. A smoothed L₁ loss function is used for training, which is robust to outliers and helps prevent overfitting.

Datasets

The primary dataset used for training and evaluation is the New HCI light field dataset. This dataset comprises 24 scenes, including occlusion and fine-textured scenarios, with each scene featuring a 9x9 multi-view sequence at a spatial resolution of 512x512. The dataset provides ground truth depth maps. The paper also mentions evaluating on the UrbanLF dataset for additional validation.

Key Insights and Ideas

  • Dynamic View Selection is Key: The paper strongly advocates for moving beyond fixed view selection. The attention mechanism's ability to adaptively weight views based on scene content is a significant improvement, especially for handling varying scene complexities.
  • Multi-Scale Context for Occlusions: The FPN's ability to fuse features across different scales is crucial for understanding occluded regions. By combining low-level details with high-level semantic information, the network can infer depth even when direct visual cues are missing.
  • Balancing Accuracy and Efficiency: The proposed architecture aims to strike a balance between achieving high accuracy and maintaining computational efficiency, which is a common challenge in deep learning for computer vision.

Key Results

FPattNet demonstrates strong performance on the HCI 4D LF benchmark.

  • Quantitative Results: The paper reports achieving the best results in 11 out of 32 columns across various metrics (BadPix 0.01, 0.03, 0.07, and MSE) in Table 2. Specifically, it achieved the best average BadPix 0.01, 0.03, and 0.07 across eight scenes. For instance, on the "sideboard" scene, FPattNet achieved a BadPix 0.03 of 1.975 and an MSE of 0.088, outperforming many other methods.
  • Qualitative Results: Visual comparisons (Figures 4, 5, 8, and 10) show that FPattNet consistently produces more accurate depth maps, especially in challenging regions with occlusion and textureless surfaces, compared to other state-of-the-art methods. The generated 3D point clouds also exhibit better integrity.

Limitations and Future Work

  • Limitations: While the paper highlights the effectiveness of its approach, the inherent limitations of light field cameras, such as narrow baselines between sub-aperture images, are acknowledged. The computational complexity of processing all views, even with attention, might still be a factor for real-time applications.
  • Future Work: The paper suggests that further research could explore more efficient ways to handle the large amount of data from light field images and potentially investigate more advanced attention mechanisms or fusion strategies. The authors also mention that their method excels in handling occluded and textureless regions, implying that further exploration in these specific challenging areas could be beneficial.

Summary updated

Abstract

A light field camera can capture light information from various directions within a scene, allowing for the reconstruction of the scene. The light field image inherently contains the depth information of the scene, and depth estimations of light field images have become a popular research topic. This paper proposes a depth estimation network of light field images with occlusion awareness. Since light field images contain many views from different viewpoints, identifying the combinations that contribute the most to the depth estimation of the center view is critical to improving the depth estimation accuracy. Current methods typically rely on a fixed set of views, such as vertical, horizontal, and diagonal, which may not be optimal for all scenes. To address this limitation, we propose a novel approach that considers all available views during depth estimation while leveraging an attention mechanism to assign weights to each view dynamically. By inputting all views into the network and employing the attention mechanism, we enable the model to adaptively determine the most informative views for each scene, thus achieving more accurate depth estimation. Furthermore, we introduce a multi-scale feature fusion strategy that amalgamates contextual information and expands the receptive field to enhance the network's performance in handling challenging scenarios, such as textureless and occluded regions.
BibTeX
@inproceedings{xiao2023fpattnet,
  author = {Min Xiao and Chen Lv and Xiaomin Liu},
  title = {{FPattNet: A Multi-Scale Feature Fusion Network with Occlusion Awareness for Depth Estimation of Light Field Images}},
  booktitle = {Sensors},
  year = {2023},
  doi = {10.3390/s23177480},
}
Figure