TL;DR
This paper introduces a novel method for 3D shape reconstruction using thermal polarization (LWIR) images. By leveraging the unique properties of LWIR polarization, which are independent of illumination and material properties, the authors achieve highly accurate and detailed 3D reconstructions, outperforming existing methods, especially for challenging transparent and heterogeneous objects.
What Problem Does This Paper Address?
Reconstructing the 3D shape of objects is a fundamental problem in computer vision with applications ranging from robotics to augmented reality. Traditional methods, like multi-view stereo and shape-from-shading, often struggle with varying illumination conditions and diverse material properties. For instance, multi-view stereo relies on textured surfaces, while structured light and photometric stereo methods make strong assumptions about opaque surfaces and specific lighting. Shape-from-polarization (SfP) in the visible spectrum is particularly sensitive to the complex interplay of specular and diffuse reflections, leading to ambiguities in surface normal estimation. While thermal imaging offers an advantage by being independent of external illumination, existing thermal reconstruction methods still face accuracy limitations. This paper aims to overcome these limitations by exploiting the underutilized thermal polarization cues for robust and detailed 3D shape reconstruction.
Methods
The core of this work lies in the theoretical formulation of a general theory of polarization observation in the Long-Wave Infrared (LWIR) spectrum. The authors demonstrate that LWIR polarization is free from the specular-diffuse ambiguities that plague visible light polarization. This is because LWIR emission is dominant and not mixed with reflected light in the same way as visible light.
The proposed method then utilizes multi-view thermal polarization images, specifically the Angle of Linear Polarization (AoLP), for 3D shape reconstruction. AoLP is chosen because it is invariant to material properties, making it a robust cue. The 3D shape is represented implicitly using a Multi-Layer Perceptron (MLP) that learns the Signed Distance Function (SDF). This MLP is optimized using a differentiable rendering framework, incorporating a tangent space constraint (TSC) loss. This TSC loss leverages the AoLP information to enforce consistency on surface normal estimations across different views. Additional losses, such as silhouette loss and Eikonal regularization, are used to refine the shape representation.
Datasets
The authors prepared a real-world dataset with ground-truth shapes. They captured 7 objects with various shapes and materials, including transparent and low-reflective objects. The objects were placed on a rotating table and captured from about 20-30 views. To obtain ground-truth shapes, a structured light 3D scanner (EinScan-SP V2) was used. For accurate measurements, the objects' surfaces were painted white to make them opaque.
Key Insights and Ideas
- LWIR Polarization's Advantage: The paper convincingly argues that LWIR polarization is inherently more robust for shape reconstruction than visible light polarization due to its independence from external illumination and the absence of specular-diffuse ambiguities.
- AoLP as a Material-Invariant Cue: The consistent and reliable nature of AoLP in LWIR, irrespective of material properties, makes it a powerful and unique cue for shape recovery, especially for challenging materials.
- Implicit Shape Representation with Thermal Polarization: Combining implicit neural representations (SDFs) with thermal polarization cues allows for detailed and accurate reconstruction of complex geometries that are difficult for traditional methods.
- Addressing Transparency in LWIR: The paper highlights that materials transparent in visible light can be opaque in LWIR, opening up new possibilities for reconstructing their shapes.
Key Results
The experimental results demonstrate that the proposed method significantly outperforms existing approaches, including visible multi-view stereo (MVAS) and thermal implicit reconstruction (Thermal IDR), in terms of both shape accuracy (Chamfer distance) and surface normal accuracy (mean angular error).
Qualitatively, the method successfully reconstructs fine details, such as concave parts of transparent vases and bottles, and details on the lid of a container, which are missed by other methods. The paper also shows that visible light polarization methods struggle with noisy AoLP observations, leading to wavy artifacts, while thermal IDR can result in "swollen" shapes due to not explicitly considering surface normals.
Limitations and Future Work
The authors acknowledge several limitations:
- Emission vs. Reflection Dominance: The method assumes that the thermal emission component is sufficiently strong compared to the reflection component. This assumption may not hold in environments with strong ambient thermal sources or when the target object's temperature is close to the environment's.
- Metallic and Rough Surfaces: The method is less effective on metallic or rough surfaces where polarization signals can be unstable or noisy.
- System Limitations: The current capturing system requires time to rotate the polarization filter, which can be a bottleneck.
Future work directions include:
- Decomposing Emission and Reflection: Developing methods to separate emission and reflection components before shape recovery, especially for challenging surfaces.
- Improving System Efficiency: Exploring single-shot polarized LWIR cameras to accelerate thermal polarization analysis.
- Handling Metallic/Rough Surfaces: Investigating techniques to improve performance on these challenging materials.
- Integrating Additional Cues: Combining thermal polarization with other sensing modalities could further enhance robustness.
Summary updated