Fast and Accurate 3D Measurement Based on Light-Field Camera and Deep Learning

Haoxin Ma, Zhiwen Qian, Tingting Mu, Shengxian Shi2019 · DOI

TL;DR

This paper introduces VommaNet, a novel deep learning architecture for accurate 3D depth estimation from light-field images, specifically tackling the challenging problem of reflective and texture-less surfaces. By effectively integrating multi-scale features and leveraging neighborhood information, VommaNet achieves state-of-the-art performance with comparable runtime to existing methods.

What Problem Does This Paper Address?

Accurate 3D measurement and modeling are crucial for various applications, and light-field cameras offer a promising avenue due to their ability to capture both spatial and angular information. This allows for depth estimation from a single image. However, a significant challenge arises when dealing with reflective and texture-less surfaces. In these regions, traditional depth estimation algorithms struggle because pixels have similar or identical color values, making it difficult to find corresponding points across different views or to infer depth from local image features. Existing methods often fail or produce inaccurate results in these scenarios, limiting the overall utility of light-field cameras for comprehensive 3D reconstruction.

Methods

The core of this work is the VommaNet architecture, a deep learning model designed to overcome the limitations of previous approaches for reflective and texture-less areas. The key innovations include:

  • Multi-Scale Feature Extraction: VommaNet employs a feature pyramid using atrous convolutions with increasing dilation rates (1, 2, 4, 8, 16, 32). This allows the network to capture features at various scales, effectively "stepping back" to consider a larger receptive field. This is crucial for inferring depth in challenging regions by leveraging information from their surroundings.
  • Depthwise Separable Convolutions: To manage computational burden and reduce the number of parameters, the network utilizes depthwise separable convolutions. This is a more efficient alternative to standard convolutions, enabling faster training and inference.
  • Residual Blocks: The architecture incorporates residual blocks to facilitate the training of a deep network. These blocks help mitigate the vanishing gradient problem and allow for the integration of information from shallower layers, improving the flow of gradients and enhancing performance.
  • Combined Loss Function: A weighted sum of three loss functions is used: Mean Absolute Error (MAE) for overall accuracy, a gradient-based loss to penalize errors around edges, and a normal-based loss to improve fine details of the depth map. This comprehensive loss function encourages both pixel-wise accuracy and structural similarity to the ground truth.
  • End-to-End Network: VommaNet is designed as an end-to-end network, taking raw light-field images (specifically, concatenated sub-aperture images) as input and directly outputting a disparity map.

Datasets

The authors utilized two main sources for their data:

  • Benchmark Dataset [43]: This dataset provides light-field images in the form of sub-aperture images (SAIs) along with ground truth depth and disparity maps, typically at a resolution of 512x512 pixels. This dataset includes synthetic scenes.
  • Additional Data: The paper also mentions using additional data, likely from other sources or collected by the authors, to augment their training set.

For training, the authors augmented the data by flipping (up-down, left-right, and both), color inversion, and cropping. They started with 116 pairs of light-field images and ground truth and generated over 104,400 pairs for training. A validation dataset of eight pairs was also used.

Key Insights and Ideas

  • "Stepping Back" for Depth: The fundamental insight is that to accurately estimate depth in reflective or texture-less regions, the network needs to consider a broader context. By analyzing the edges and surrounding areas, the network can infer the depth of these difficult regions.
  • Atrous Convolutions for Receptive Field Expansion: Atrous convolutions are a powerful tool for increasing the receptive field of a convolutional neural network without significantly increasing computational cost or parameter count. This is key to VommaNet's ability to capture global context.
  • Efficiency through Depthwise Separable Convolutions: The adoption of depthwise separable convolutions highlights a practical consideration: achieving high accuracy without prohibitive computational overhead, making the system "fast" as well as accurate.
  • Multi-faceted Loss for Robustness: Combining different loss terms (MAE, gradient, normal) is crucial for capturing not just pixel-wise accuracy but also the structural integrity and fine details of the estimated depth maps, especially in challenging areas.

Key Results

  • Qualitative Performance: VommaNet demonstrated superior or comparable results to state-of-the-art algorithms on real-world light-field images, particularly excelling in reflective and texture-less regions where other methods failed. Examples include preserving sharp outlines of objects and accurately estimating depth on ceramic bowls and skulls.
  • Quantitative Performance (Benchmark): On the benchmark dataset, VommaNet achieved the lowest Mean Squared Error (MSE) of 2.151 × 10⁻², outperforming other methods like [8] (3.968 × 10⁻²) and [31] (2.521 × 10⁻²).
  • Runtime: The network maintained a fast runtime (2.043 seconds) comparable to the fastest existing methods, making it suitable for practical applications.
  • 3D Geometry Accuracy: When tested on standard gauge blocks, the system achieved an average error of 0.3059 mm and a standard deviation of 0.3408 mm, with a runtime of 0.543 seconds. This demonstrates high accuracy for 3D geometry measurement.

Limitations and Future Work

  • Detail Preservation: The authors acknowledge that their network can sometimes blur intricate details within objects, while textures from the background or object surfaces might be preserved.
  • Fixed Input Size: The current network is trained for a specific number of SAIs.
  • Future Work:
    • Recurrent Networks: To handle varying numbers of input images and potentially improve detail preservation, the authors suggest exploring recurrent network architectures.
  • Color Cameras: Replacing the monochrome light-field camera with a color one could further enhance the system's capabilities.
  • High-Resolution Cameras: Advancements in high-resolution light-field cameras could also lead to improved measurement accuracy.

Summary updated

BibTeX
@misc{ma2019fast,
  author = {Haoxin Ma and Zhiwen Qian and Tingting Mu and Shengxian Shi},
  title = {{Fast and Accurate 3D Measurement Based on Light-Field Camera and Deep Learning}},
  year = {2019},
  doi = {10.3390/s19204399},
}
Figure