Attention-Based View Selection Networks for Light-Field Disparity Estimation

Yu-Ju Tsai, Yu-Lun Liu, Ming Ouhyoung, Yung-Yu ChuangAAAI 2020 · OpenReview

Abstract

This paper introduces a novel deep network for estimating depth maps from a light field image. For utilizing the views more effectively and reducing redundancy within views, we propose a view selection module that generates an attention map indicating the importance of each view and its potential for contributing to accurate depth estimation. By exploring the symmetric property of light field views, we enforce symmetry in the attention map and further improve accuracy. With the attention map, our architecture utilizes all views more effectively and efficiently. Experiments show that the proposed method achieves state-of-the-art performance in terms of accuracy and ranks the first on a popular benchmark for disparity estimation for light field images.

Introduction

Light field cameras capture light from multiple directions simultaneously, enabling post-capture refocusing and perspective shifts. However, extracting accurate depth (disparity) maps from this high-dimensional "4D" data is a significant computational challenge. Standard light field images, such as those from Lytro cameras, typically consist of an 81-view grid (9x9). Researchers often face a trade-off: utilizing all views for maximum information at a high computational cost with significant redundancy, or using a subset (like a crosshair) and losing critical diagonal information needed for complex occlusions.

To address this, the LFattNet architecture introduces an attention-based view selection mechanism. By dynamically weighting the importance of different sub-aperture views, the model achieves superior accuracy while managing data redundancy efficiently.

Methodology: The LFattNet Architecture

The LFattNet pipeline transforms raw light field views into high-resolution disparity maps through several specialized stages.

The detailed architecture of LFattNet, featuring feature extraction with SPP, cost volume construction, the attention module, and disparity regression.
  1. Feature Extraction & Spatial Pyramid Pooling (SPP): Each of the 81 views is processed through residual blocks and an SPP module. SPP is vital for capturing multi-scale contextual information, which aids depth estimation in textureless regions.
  2. Cost Volume Construction: Features are shifted across disparity levels to create a 5D cost volume, facilitating pixel-wise comparisons across views.
  3. Attention-Based View Selection: This core innovation generates a 9x9 attention map of weights. These weights inform the subsequent Disparity CNN which views are most reliable for the specific scene.
  4. Disparity Regression: A 3D CNN processes the weighted cost volume, using a "soft argmin" operation to produce a continuous, smooth disparity map.

Exploiting Symmetry in Attention

The researchers explored how geometric constraints on the attention map affect performance. They tested three configurations: Free (81 independent weights), Symmetric (25 weights mirrored across axes), and Radial (15 weights mirrored diagonally).

Comparison of attention map configurations: (a) free, (b) symmetric, and (c) radial. The radial configuration enforces the highest degree of physical symmetry.

Surprisingly, the most constrained version—the Radial map—yielded the best results. By forcing the network to respect the physical symmetry inherent in camera lenses, the model is effectively regularized, preventing it from over-fitting to noise in individual views.

Results and Performance

LFattNet was evaluated on the 4D Light Field Benchmark (Honauer et al. 2016) and other synthetic and real-world datasets. At the time of publication, it ranked #1 on the benchmark across all primary error metrics (MSE*100, Badpix 0.01, 0.03, and 0.07).

Qualitatively, the model excels at preserving sharp edges around thin structures, such as bicycle spokes, and maintaining smoothness in textureless areas. While it faces challenges with non-Lambertian (glossy) surfaces and massive textureless regions, it represents a significant advancement in efficient light field processing.

Conclusion

By integrating an attention mechanism with geometric symmetry constraints, LFattNet demonstrates that "more data" is most effective when paired with "smart selection." This approach sets a new standard for disparity estimation in light field imaging, balancing information density with computational efficiency.

Summary updated

Abstract

This paper introduces a novel deep network for estimating depth maps from a light field image. For utilizing the views more effectively and reducing redundancy within views, we propose a view selection module that generates an attention map indicating the importance of each view and its potential for contributing to accurate depth estimation. By exploring the symmetric property of light field views, we enforce symmetry in the attention map and further improve accuracy. With the attention map, our architecture utilizes all views more effectively and efficiently. Experiments show that the proposed method achieves state-of-the-art performance in terms of accuracy and ranks the first on a popular benchmark for disparity estimation for light field images.
BibTeX
@inproceedings{tsai2020attention,
  author = {Yu-Ju Tsai and Yu-Lun Liu and Ming Ouhyoung and Yung-Yu Chuang},
  title = {{Attention-Based View Selection Networks for Light-Field Disparity Estimation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=X_oXdmcwsL},
}
Figure