Polarimetric PatchMatch Multi-View Stereo

Jinyu Zhao, Jumpei Oishi, Yusuke Monno, Masatoshi OkutomiWACV 2024

TL;DR

This paper introduces Polarimetric PatchMatch Multi-View Stereo (PolarPMS), a novel approach that leverages polarization cues to significantly improve the accuracy and completeness of 3D reconstructions from multiple images. By incorporating polarimetric consistency alongside traditional photometric and geometric checks, PolarPMS excels at reconstructing texture-less surfaces, a common challenge for existing methods.

What Problem Does This Paper Address?

Reconstructing accurate 3D models from multiple 2D images is a fundamental problem in computer vision, with applications ranging from robotics to augmented reality. Multi-View Stereo (MVS) algorithms aim to achieve this by finding corresponding points across different views and inferring their 3D positions. PatchMatch MVS methods, known for their balance of accuracy and efficiency, typically rely on photometric consistency (color similarity) to match patches. However, this reliance on texture makes them struggle with texture-less surfaces, leading to incomplete or inaccurate reconstructions. Existing methods often either exclude these challenging regions or employ specialized techniques that may not generalize well to arbitrary objects.

Methods

PolarPMS builds upon the PatchMatch MVS framework by introducing a new cost term for evaluating depth and normal hypotheses: polarimetric consistency. The core idea is that the Angle of Polarization (AoP) of reflected light from a surface is related to the surface's normal direction. This relationship is largely independent of surface texture, making it a valuable cue for texture-less regions.

The method works as follows:

  1. Initialization: Depth and normal for each pixel are randomly initialized.
  2. Hypothesis Generation: For each pixel, seven depth and normal hypotheses are generated (similar to existing PatchMatch MVS methods, including propagation, random generation, and perturbation).
  3. Hypothesis Evaluation: Each hypothesis is evaluated using a cost function that combines:
  4. Photometric Consistency ($F_{pho}$): Standard color similarity between patches in reference and source views.
  5. Geometric Consistency ($F_{geo}$): Measures reprojection errors.
  6. Polarimetric Consistency ($F_{pol}$): This is the novel contribution. It measures the consistency between the observed AoP and the azimuth angle derived from the hypothesized normal. This term helps disambiguate surface normals in texture-less areas.
  7. Depth-Normal Consistency ($F_{dep}$): An additional term to enforce consistency between the estimated normal and the normal calculated from neighboring depth values, leveraging the improved normals from the polarimetric cue.
  8. Selection: The hypothesis with the minimum total cost is selected as the current best estimate.
  9. Iteration: Steps 2-4 are repeated iteratively, scanning pixels in different orders, until convergence or a maximum iteration limit is reached.

The paper also introduces a filtering step before point cloud generation, removing pixels with insufficient polarization cues (low DoP) or weak texture (low pixel intensity variance).

Datasets

The paper evaluates its method using:

  • Synthetic Data: Generated using the Mitsuba 2 renderer with three CG models (Armadillo, Bunny, Dragon) and a polarized plastic material to simulate realistic polarization effects. This allows for precise control over ground truth and material properties.
  • Real Data: A dataset from a previous work [27] containing images of a toy car, a camera, and a statue.

No new datasets were introduced.

Key Insights and Ideas

  • Polarization as a Texture-Independent Cue: The most significant insight is that polarization information (specifically AoP) provides a direct, texture-independent cue for estimating surface normals. This is a crucial breakthrough for MVS in challenging texture-less scenarios.
  • Complementary to Photometric Cues: Polarimetric consistency doesn't replace photometric consistency but complements it. While photometry struggles with uniform surfaces, polarization can still provide reliable information.
  • Cascading Improvements: The improved normal estimation from polarimetric consistency can then be used to further refine depth estimation through depth-normal consistency, creating a synergistic effect.
  • Addressing Texture-less Regions: The method directly tackles the long-standing problem of reconstructing texture-less surfaces, leading to more complete 3D models.

Key Results

  • Improved Accuracy and Completeness: PolarPMS consistently outperforms COLMAP and ACMMP in both accuracy and completeness metrics on synthetic data, particularly for the Bunny model which has significant texture-less areas.
  • Superior Normal Estimation: Numerical evaluations show a significant improvement in normal estimation accuracy compared to COLMAP and ACMMP, with a higher proportion of pixels having very small normal errors.
  • Enhanced Reconstruction of Texture-less Regions: Visual comparisons on both synthetic and real data (toy car, statue) demonstrate PolarPMS's ability to reconstruct details in texture-less regions (e.g., car windows, statue base) that are missed or poorly rendered by other methods.
  • Comparison with Polarimetric MVS: PolarPMS achieves more accurate 3D points than Polarimetric MVS [8] by incorporating multi-view consistency for polarization, whereas Polarimetric MVS uses it for view-by-view propagation.

Limitations and Future Work

  • Inter-reflections: The paper acknowledges that PolarPMS has limitations in reconstructing regions with strong inter-reflections, which exhibit complex polarization behavior.
  • Future Work: The authors suggest adopting a more accurate reflection model to handle these complex inter-reflection scenarios in future work.

Summary updated

BibTeX
@inproceedings{zhao2024polarimetric,
  author = {Jinyu Zhao and Jumpei Oishi and Yusuke Monno and Masatoshi Okutomi},
  title = {{Polarimetric PatchMatch Multi-View Stereo}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure