Robust multi-view stereo without matching

Philippe Lambert, Patrick HébertWorkshops @ ICCV 2009 · DOI

Abstract

Traditional multi-view stereo (MVS) algorithms often struggle with non-Lambertian surfaces and complex occlusions due to their reliance on photoconsistency. This paper presents a robust reconstruction method that avoids explicit image matching. By analyzing the frequency content of light rays passing through 3D points—represented as lumispheres—and employing a novel robust voting scheme, the algorithm identifies optimal "proxy" surfaces. These surfaces are then merged using Radial Basis Functions (RBFs) to create a continuous 3D model. The approach demonstrates high accuracy on standard benchmarks and superior performance on challenging shiny or translucent objects.

Introduction

Most multi-view stereo algorithms rely on the principle of photoconsistency: the assumption that a specific point on an object's surface should appear identical in color and intensity from any viewpoint. While this holds true for matte (Lambertian) surfaces, it fails significantly for shiny, metallic, or translucent objects where the observed color changes based on the viewing angle.

Previous research suggested using a "frequency criterion" to find surfaces by minimizing color variations among neighboring rays. However, direct optimization of this criterion is extremely sensitive to visibility outliers—points that are occluded in some views. Because these outliers often appear in clusters, standard filtering techniques fail, leading algorithms to find "false" surfaces. This paper introduces a reconstruction method robust enough to handle real-world, complex datasets where traditional assumptions break down.

Methodology

The proposed approach consists of two primary stages: determining the optimal depth for each pixel through a robust voting mechanism and merging these individual depth estimates into a global 3D proxy.

The Frequency Criterion and Lumispheres

Instead of matching pixels across images, the algorithm analyzes "lumispheres"—spherical representations of all light rays passing through a 3D point. The goal is to find the depth where the gradient of colors on this lumisphere is minimized. This indicates a point where the rays are most likely originating from a single surface point.

The frequency criterion analyzes light rays passing through a 3D point to find the optimal surface depth.

Robust Voting Scheme

The core innovation of this work is a robust voting mechanism. Rather than simply summing up gradients, which allows a single outlier to skew the average, the algorithm uses a voting scheme where each part of the lumisphere contributes to a multi-modal likelihood distribution. This allows the system to identify a clear consensus depth even when significant portions of the data are corrupted by occlusions or specular reflections.

Comparison between the direct optimization and the proposed robust voting approach, showing how the likelihood peak identifies the optimal depth despite outliers.

Global Surface Reconstruction

To convert scattered depth estimates into a solid 3D model, the authors employ Radial Basis Functions (RBFs). This mathematical framework is highly effective for interpolating data and filling holes in "dataless" regions by using visual hull information as a guide. A RANSAC-type procedure is integrated during the depth-map fitting stage to further prune outliers before the final global merge, ensuring a smooth and regularized surface.

Experimental Results

The algorithm was evaluated on several benchmarks, including the Middlebury MVS datasets (Temple and Dino) and complex non-Lambertian datasets (Bust and Synthetic Buddha).

  • Accuracy: On the Middlebury Temple dataset, the algorithm achieved 0.55mm accuracy and 99.7% completeness, performing comparably to state-of-the-art Lambertian-focused algorithms while making fewer material assumptions.
  • Non-Lambertian Performance: The method successfully reconstructed high-quality proxies for the Bust and Buddha models, where traditional matching-based methods typically produce significant noise or holes due to subsurface scattering and metallic paint.
  • Specular Robustness: Experiments with a glossy galvanized plane confirmed that the algorithm could even reconstruct the "virtual" depths of light sources, demonstrating its utility for high-quality view synthesis.

Discussion and Conclusion

A key insight of this research is the distinction between a physical shape and an "optimal" proxy. For non-Lambertian objects, the proxy surface that captures light behavior (like the depth of a virtual light source reflection) may be more useful for view interpolation than the true geometric surface.

While the method is robust, it currently produces a single continuous surface and requires a high density of images (hundreds) to build reliable lumispheres. Future work may explore volumetric proxies to better represent objects with extreme transparency or complex internal specularities. Despite these requirements, the approach offers a powerful alternative for 3D reconstruction in environments where photoconsistency cannot be guaranteed.

Summary updated

BibTeX
@inproceedings{lambert2009robust,
  author = {Philippe Lambert and Patrick Hébert},
  title = {{Robust multi-view stereo without matching}},
  booktitle = {Workshops @ ICCV},
  year = {2009},
  doi = {10.1109/iccvw.2009.5457419},
}
Figure