Reconstructing Complex Surfaces from Multiple Stereo Views

Pascal Fua

Abstract

We present a framework for 3D surface reconstruction that can be used to model fully 3 dimensional scenes from an arbitrary number of stereo views taken from vastly different viewpoints. This is a key step toward producing 3D world descriptions of complex scenes using stereo and is a very challenging problem: real world scenes tend to contain many 3D objects, they do not usually conform to the 2-1/2D assumption made by traditional algorithms, and one cannot take it for granted that the computed 3D points can easily be clustered into separate groups. By combining a particle based representation, robust fitting, and optimization of an image based objective function, we have been able to reconstruct surfaces without any a priori knowledge of their topology and in spite of the noisiness of the stereo data. Our current implementation goes through three steps—initializing a set of particles from the input 3D data, optimizing their location, and finally grouping them into global surfaces. Using several complex scenes containing multiple objects, we demonstrate its competence and ability to merge information and thus to go beyond what can be done with conventional stereo alone.

Introduction

Traditional stereo reconstruction often relies on the "2.5-D assumption," which models a scene as a single height map where every $(x, y)$ coordinate corresponds to exactly one depth value. While effective for simple terrain, this assumption fails in complex real-world environments characterized by overlapping objects, holes, or multiple distinct bodies. Furthermore, raw stereo data is frequently noisy and incomplete, containing "blunders" or outliers that complicate the grouping of points into meaningful surfaces. This creates a "chicken-and-egg" problem: accurate surface modeling requires clean data, but data cleaning often requires an underlying surface model.

Methodology

The proposed framework addresses these challenges by employing a flexible, particle-based representation instead of rigid global meshes. This approach allows for modeling objects with unknown topologies and complex overlapping structures.

Oriented particle representation, voxel grid initialization, and particle interaction.

1. Initialization via Robust Fitting

The process begins with raw 3-D points generated from standard correlation-based stereo. The reconstruction space is partitioned into voxels (buckets), and a local quadric surface patch—an "oriented particle"—is fitted to the points within each bucket. To handle the inherent noisiness of stereo data, the system utilizes Iterative Reweighted Least Squares (IRLS). This robust statistical technique effectively ignores outliers during the fitting process, ensuring that "blunders" do not skew the initial surface estimates.

Robust fitting mechanism for outlier rejection using Iterative Reweighted Least Squares.

2. Global Optimization

Once initialized, the oriented particles are refined by minimizing a global energy function. This function integrates two critical components:

  • Image-Based Term: Particles are projected back into the original 2-D images to measure the variance of gray levels. Correct placement is indicated by high similarity across all viewpoints, allowing the system to correct errors present in the initial 3-D point cloud.
  • Deformation Term: Modeled as "breakable springs" between neighbors, this term encourages alignment in orientation and position (smoothness) without imposing a fixed topology.

3. Clustering

In the final stage, refined particles are grouped into global surfaces. The clustering metric considers both the spatial distance between particles and the alignment of their tangent planes, allowing the system to distinguish between separate objects even in densely packed scenes.

Results and Discussion

The framework was evaluated on various datasets, including synthetic hemispheres, complex indoor scenes (e.g., a wheel on a box), and aerial imagery. Quantitative validation against ground truth aerial data from Ohio State showed a Root Mean Square (RMS) error of 0.19 meters, equivalent to a disparity error of less than 0.4 pixels.

Notably, the system successfully reconstructed "overhangs" in rock piles, a task typically impossible for 2.5-D algorithms. Even with 40% random noise in the initial data, the robust fitting and optimization stages recovered clean surface shapes. The "breakable springs" concept proved particularly effective, allowing particles to "stick together" if they belong to the same surface but drift apart if they represent different objects or noise.

Conclusion

By representing surfaces as interacting oriented particles and leveraging original image intensities during optimization, this framework overcomes the limitations of traditional stereo reconstruction. While future work is needed to improve hole-filling in featureless regions and refine clustering heuristics for extreme complexity, the current system demonstrates a robust ability to model 3-D worlds without a priori topological knowledge. The particle-based approach is also naturally parallelizable, offering a path toward managing the computational costs of large-scale scene reconstruction.

Summary updated

Abstract

We present a framework for 3D surface reconstruction that can be used to model fully 3 dimensional scenes from an arbitrary number of stereo views. Taken from vastly different viewpoints. This is a key step toward producing 3D world descriptions of complex scenes using stereo and is a very challenging problem: real world scenes tend to contain many 3D objects, they do not usually conform to the 2-1/2D assumption made by traditional algorithms, and one cannot take it for granted that the computed 3D points can easily be clustered into separate groups. By combining a particle based representation, robust fitting, and optimization of an image based objective function, we have been able to reconstruct surfaces without any a priori knowledge of their topology and in spite of the noisiness of the stereo data. Our current implementation goes through three steps-initializing a set of particles from the input 3D data, optimizing their location, and finally grouping them into global surfaces. Using several complex scenes containing multiple objects, we demonstrate its competence and ability to merge information and thus to go beyond what can be done with conventional stereo alone.<
BibTeX
@inproceedings{fua1995reconstructing,
  author = {Pascal Fua},
  title = {{Reconstructing Complex Surfaces from Multiple Stereo Views}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {1995},
  url = {https://openreview.net/forum?id=rJWNigGdZr},
}
Figure