Surface Reconstruction Using 3--D Meshes And Particle Systems

Pascal FuaInfoscience 1995

Abstract

We present a unified framework for 3-D shape reconstruction that allows us to combine multiple information sources—such as stereo and shape-from-shading—derived from images whose vantage points may be very different. A formal integration framework is critical because, in recovering complicated surfaces, the information from a single source is often insufficient to provide a unique answer. We describe two complementary implementations of this paradigm that both rely on deforming a generic object-centered 3-D representation of the surface so as to minimize an objective function. The first implementation models the surface as a triangulated mesh. The second models it as a set of particles that interact with each other through forces that tend to align them. Using several complex scenes, we demonstrate our approach and its ability to merge information and thus to go beyond what can be done with conventional techniques.

Introduction

In computer vision, the "Shape from X" problem (where X represents stereo, shading, or texture) has long been a fundamental challenge. Traditional methods often rely on single information sources and "image-centered" representations, such as depth maps. However, these approaches struggle when dealing with complex surfaces, multiple images from widely different vantage points, or significant self-occlusions and self-shadowing.

This paper introduces a unified framework for 3D surface reconstruction that merges multiple image cues—specifically stereo and shape-from-shading. By employing object-centered representations that exist in 3D space independently of any single camera's perspective, the system can reconstruct complex geometries and topologies that traditional methods cannot handle.

Methodology: Object-Centered Representations

The proposed framework utilizes two complementary object-centered implementations within a single optimization framework: triangulated meshes and oriented particle systems.

Triangulated Meshes

For surfaces with a known topology, such as a human face or a patch of terrain, the system uses a hexagonally connected mesh. The reconstruction is treated as an energy minimization problem where the objective function combines stereo cues, shape-from-shading, and regularization.

Integration of stereo and shading cues using a triangulated mesh.

Stereo cues are calculated by measuring gray-level variance across multiple images for a given surface point. Shape-from-shading estimates albedo variations across facets based on surface normals and light vectors. To ensure smoothness, a "snake-like" optimization technique penalizes high curvature.

Oriented Particle Systems

For scenes with unknown topology or multiple objects (e.g., a pile of rocks), the mesh is replaced with "oriented particles"—small disks that interact via "breakable springs." These forces encourage alignment but allow the surface to disconnect if the data suggests separate objects or holes.

A critical technical contribution is the weighting and normalization scheme. Since stereo and shading data have different units, the authors normalize the gradients of these energy terms at the start of optimization, making the process robust across different scenes without manual retuning.

Experimental Results

The framework was evaluated on several complex real-world datasets, including face triplets from INRIA, a 40-image head sequence from ETH Zurich, and rock pile images featuring complex overhanging geometries.

Face Reconstruction

The mesh-based approach successfully combined stereo data (robust in textured areas like hair) with shape-from-shading (effective in smooth areas like the forehead). This synergy allowed for the creation of highly detailed facial models that capture both coarse structure and fine surface detail.

Face reconstruction results combining stereo and shading information.

Handling Complex Geometries

The particle system successfully reconstructed a pile of rocks, including an overhanging rock. This is a significant result because standard "2.5-D" height maps are mathematically incapable of representing one surface directly above another. Furthermore, the clustering and refinement steps for particles effectively filtered out noise from messy raw stereo data.

Discussion and Conclusion

By modeling surfaces in 3D (object-centered), the system naturally handles occlusions. If a facet is hidden from one camera, the framework simply ignores that camera's contribution for that specific facet. The "breakable springs" in the particle system allow for the representation of complex, non-manifold geometries where influence vanishes as particles move out of alignment.

While the particle system offers greater flexibility, it comes with a higher computational cost. Both methods also require a reasonable initial estimate to avoid local minima. Future work may involve refining shading models to account for complex inter-reflections beyond simple lighting assumptions. Overall, this unified framework demonstrates the power of integrating multiple cues within an object-centered 3D representation.

Summary updated

Abstract

We present a unied framework for 3{D shape reconstruction that allows us to combine multiple information sources|such as stereo and shape-from-shading|derived from images whose vantage points may be very dierent. A formal integration framework is critical because, in recovering complicated surfaces, the information from a single source is often insucient to provide a unique answer. We describe two complementary implementations of this paradigm that both rely on deforming a generic object-centered 3{D representation of the surface so as to minimize an objective function. The rst implementation models the surface as a triangulated mesh. The second models it as a set of particles that interact with each other through forces that tend to align them. Using several complex scenes, we demonstrate our approach and its ability to merge information and thus to go beyond what can be done with conventional techniques. 1
BibTeX
@inproceedings{fua1995surface,
  author = {Pascal Fua},
  title = {{Surface Reconstruction Using 3--D Meshes And Particle Systems}},
  booktitle = {Infoscience},
  year = {1995},
}
Figure