Three Dimensional Shape Reconstruction via Polarization Imaging and Deep Learning

Xianyu Wu, Penghao Li, Xin Zhang, Jiangtao Chen, Feng HuangSensors 2023 · DOI

TL;DR

This paper introduces a novel deep learning approach, SfP-U²Net, for reconstructing 3D object shapes using polarization imaging. By enhancing the input representation with separated specular and diffuse reflection components and leveraging the U²Net architecture, the method achieves more accurate surface normal estimation and better preservation of object textures compared to previous state-of-the-art techniques.

What Problem Does This Paper Address?

Traditional 3D reconstruction methods like stereo vision and structured light struggle with objects that have smooth surfaces or poor texture, often leading to information loss and inaccurate depth estimation due to overexposure. Polarization imaging, or Shape from Polarization (SfP), offers a promising alternative by utilizing the polarization properties of reflected light, which are less sensitive to lighting conditions and surface texture. However, existing SfP methods, including those based on deep learning, face challenges in accurately recovering fine texture details and estimating surface normals, particularly for objects with both specular and diffuse reflections. This can result in incomplete or inaccurate 3D reconstructions.

Methods

The core of this work is the SfP-U²Net model, which builds upon the U-Net architecture but incorporates the more powerful U²Net encoder-decoder structure. Key methodological contributions include:

  • Enhanced Input Representation: Instead of using raw polarization images directly, the proposed method extracts and utilizes:
  • Separated specular (I_raws) and diffuse (I_rawD) reflection components.
  • An encoded phase angle (φe).
  • A Stokes-vector-based parameter (Ip), which is a modified degree of polarization designed to enhance contrast and reduce background noise.
  • U²Net Architecture: This architecture is chosen for its ability to extract multiscale features and maintain high-resolution feature maps through progressive upsampling and concatenation. This helps in preserving detailed information that might be lost in simpler U-Net architectures.
  • Modified Loss Function: A custom loss function is proposed that focuses more on the normal estimation results by excluding background pixels from the denominator and numerator, thereby improving accuracy.
  • Improved Inference Strategy: The input images are randomly shifted before being split into patches for inference. This ensures that the object is consistently present within the patched regions, leading to a more robust mean normal map.

Datasets

  • DeepSfP Dataset: This dataset, comprising polarization images of 25 objects under various lighting conditions (indoor, cloudy outdoor, sunny outdoor) and orientations, was used for training and validation. It also provides ground truth normal maps derived from structured light scanning.
  • Newly Collected Data: The authors also collected their own polarization images using a custom experimental setup to further validate the proposed method under different real-world conditions (indoor and outdoor).

Key Insights and Ideas

  • Specular and Diffuse Separation is Crucial: Explicitly separating specular and diffuse reflection components provides richer information than raw polarization images, leading to better texture preservation and normal estimation.
  • Stokes-Vector Parameter Enhances Input: The Ip parameter, derived from Stokes vectors, effectively increases the contrast between target and background, helping the network focus on relevant polarization features.
  • U²Net Architecture for Detail Preservation: The U²Net's deeper structure and multi-scale feature extraction are vital for retaining fine texture details, which is a significant limitation of previous U-Net-based SfP methods.
  • Reduced Computational Cost: By using a more informative input representation (fewer channels compared to some alternatives) and the U²Net architecture, the model achieves comparable or better performance with fewer parameters and reduced computation time.

Key Results

  • Reduced Mean Angular Error (MAE): The proposed SfP-U²Net achieved a significantly lower MAE compared to previous methods. For instance, it achieved an MAE of 17.60° on the DeepSfP dataset, outperforming the DeepSfP [21] method (21.76°) and SPW [24] (21.75°).
  • Improved Texture Reconstruction: Qualitative results show that the SfP-U²Net preserves object textures much better, especially in challenging areas like the fine details of a sculpture's face or the legs of a dragon.
  • Reduced Model Size and Computation Time: The SfP-U²Net model has fewer parameters (44.02M vs. 49.59M for SfP-UNet) and significantly reduced computation time (e.g., 0.180s for SfP-U²Net with Input 4 vs. 0.545s for SfP-UNet with Input 2).
  • Performance under Various Conditions: The model demonstrated robust performance under different lighting conditions (indoor, cloudy outdoor) and object orientations.

Limitations and Future Work

  • Limitations:
    • While improved, some detailed texture information was still not fully reconstructed under complex lighting conditions (e.g., cloudy day).
  • The paper notes that the separation of specular and diffuse components, while effective, is based on approximate estimations.
  • Future Work:
    • Develop more accurate methods for separating specular and diffuse reflection components.
  • Explore advanced deep learning techniques to further enhance feature extraction capabilities.
  • Aim for accurate full-frame surface normal estimation using a single network model and original polarization images, potentially by exploring more sophisticated polarization representations.

Summary updated

BibTeX
@article{wu2023three,
  author = {Xianyu Wu and Penghao Li and Xin Zhang and Jiangtao Chen and Feng Huang},
  title = {{Three Dimensional Shape Reconstruction via Polarization Imaging and Deep Learning}},
  journal = {Sensors},
  year = {2023},
  doi = {10.3390/s23104592},
}
Figure