Polarized Object Surface Reconstruction Algorithm Based on RU-GAN Network

Xu Yang, Cai Cheng, Jin Duan, You-Fei Hao, Yong Zhu, Hao ZhangSensors 2023 · DOI

TL;DR

This paper introduces a novel RU-GAN network that leverages polarized images to reconstruct 3D object surfaces, effectively resolving the ambiguity in surface normal vector estimation. By incorporating jump compensation and a specular reflection model, the method achieves higher accuracy and simpler implementation compared to existing techniques.

What Problem Does This Paper Address?

Reconstructing the 3D shape of objects from 2D images is a fundamental challenge in computer vision. While polarization imaging offers rich information about surface properties, it inherently leads to multiple possible solutions for surface normal vectors (up to six), creating ambiguity. Previous attempts to resolve this often relied on additional data like shading or depth maps, making the process complex and less generally applicable. This paper aims to overcome this ambiguity using only polarized images, simplifying the 3D reconstruction pipeline and improving its robustness.

Methods

The core of this work is the proposed RU-GAN network, a Generative Adversarial Network (GAN) with a U-Net architecture enhanced by "jump compensation." This compensation is applied in both the encoder and decoder parts to mitigate information loss during feature extraction and reconstruction.

Key methodological contributions include:

  • Fusion of Six Polarized Surface Normal Vectors: The RU-GAN takes six polarized surface normal vectors (derived from physical models) as input, leveraging the richer information they provide to resolve ambiguity.
  • Jump Compensation: This technique, integrated into the U-Net's encoder and decoder, aims to preserve more detailed information throughout the network, leading to better texture and feature learning.
  • Specular Reflection Model for Reflective Region Correction: To address the issue of specular highlights causing inaccurate normal vector estimations, a model is introduced to optimize the dataset by reducing the impact of these reflective regions. This involves identifying and interpolating over these areas.
  • Content Loss Function: The generator's training incorporates a content loss that includes cosine loss (for normal vector pointing accuracy) and mean squared error loss (for numerical fitting accuracy), ensuring the network learns both geometric and textural details.
  • Energy Function for Height Map Generation: Instead of relying on point cloud data, the paper proposes an energy function that relates surface normal vectors to gradients, enabling height map estimation directly from the reconstructed normals.

Datasets

The paper mentions using the DEEPSFP dataset for training and testing. It also refers to previous works that utilized datasets for normal vector estimation and 3D reconstruction, implying comparisons are made against results obtained on those datasets. No new benchmark datasets are introduced.

Key Insights and Ideas

  • Polarization as a Primary Source for Normal Ambiguity Resolution: The paper strongly advocates for using the inherent information within polarized images to solve the multi-solution problem, rather than relying on external data sources.
  • GANs for Normal Vector Estimation: While GANs are common in image generation, their application to the specific problem of normal vector estimation from polarized data is novel and shown to be effective.
  • Importance of Jump Compensation: The inclusion of jump compensation in the RU-GAN architecture is crucial for preserving fine-grained texture and detail, which is often lost in standard deep learning models.
  • Specular Reflection Handling is Critical: The specular reflection model highlights a practical challenge in real-world imaging and demonstrates a targeted approach to mitigate its negative impact on reconstruction accuracy.
  • Bridging Normals to Height Maps without Point Clouds: The energy function approach offers a more streamlined path to 3D reconstruction by directly inferring height from estimated normals, bypassing the need for explicit point cloud data.

Key Results

  • Improved Accuracy: The proposed RU-GAN network achieves an estimated normal vector accuracy improvement of approximately 20° compared to conventional physical models and about 1.5° compared to recent neural network models (specifically Ba et al. [14]).
  • Enhanced Texture and Detail: Qualitative results show that the reconstructed surfaces exhibit more prominent and clearer textures, especially on complex objects like birds and monsters, compared to previous methods.
  • Reduced Distortion: The method avoids the unnatural color junction issues seen in some traditional methods and the distortion observed in others.
  • Simpler Implementation: The framework requires only a single set of polarized images, making it more practical and less demanding in terms of data acquisition and preprocessing.

Limitations and Future Work

  • Interpolation for Reflective Regions: The paper acknowledges that the quality of interpolation in correcting reflective regions directly impacts the normal vector estimation. Future work will explore fusion interpolation algorithms to further enhance this process.
  • Height Map Refinement: The current height map obtained via the energy function might have issues with excessive height. The authors suggest considering constraints from adjacent points to improve height map accuracy.
  • Generalization to More Complex Scenes: While the paper shows promising results on various objects, further validation on a wider range of complex scenes and lighting conditions would be beneficial.

Summary updated

Abstract

There are six possible solutions for the surface normal vectors obtained from polarization information during 3D reconstruction. To resolve the ambiguity of surface normal vectors, scholars have introduced additional information, such as shading information. However, this makes the 3D reconstruction task too burdensome. Therefore, in order to make the 3D reconstruction more generally applicable, this paper proposes a complete framework to reconstruct the surface of an object using only polarized images. To solve the ambiguity problem of surface normal vectors, a jump-compensated U-shaped generative adversarial network (RU-Gan) based on jump compensation is designed for fusing six surface normal vectors. Among them, jump compensation is proposed in the encoder and decoder parts, and the content loss function is reconstructed, among other approaches. For the problem that the reflective region of the original image will cause the estimated normal vector to deviate from the true normal vector, a specular reflection model is proposed to optimize the dataset, thus reducing the reflective region. Experiments show that the estimated normal vector obtained in this paper improves the accuracy by about 20° compared with the previous conventional work, and improves the accuracy by about 1.5° compared with the recent neural network model, which means the neural network model proposed in this paper is more suitable for the normal vector estimation task. Furthermore, the object surface reconstruction framework proposed in this paper has the characteristics of simple implementation conditions and high accuracy of reconstructed texture.
BibTeX
@inproceedings{yang2023polarized,
  author = {Xu Yang and Cai Cheng and Jin Duan and You-Fei Hao and Yong Zhu and Hao Zhang},
  title = {{Polarized Object Surface Reconstruction Algorithm Based on RU-GAN Network}},
  booktitle = {Sensors},
  year = {2023},
  doi = {10.3390/s23073638},
}
Figure