3D Object Reconstruction with Heterogeneous Sensor Data

Li Guan, Jean-Sébastien Franco, Marc Pollefeys2008

TL;DR

This paper introduces a novel framework for 3D object reconstruction by fusing data from heterogeneous sensors: high-resolution camcorders and lower-resolution but faster Time-of-Flight (ToF) range imaging (RIM) cameras. They propose a probabilistic sensor fusion approach using a space occupancy grid and demonstrate its effectiveness in recovering detailed 3D shapes, overcoming limitations of individual sensor types.

What Problem Does This Paper Address?

Reconstructing accurate 3D models of objects is a fundamental problem in computer vision with applications ranging from virtual reality to medical imaging. Traditional methods often rely on single sensor types, each with its own drawbacks. For instance, silhouette-based methods (like visual hulls) can produce complete shapes but struggle with concavities and fine details. Range imaging (RIM) cameras, like the Swiss Ranger 3000, offer depth information at high frame rates but suffer from noisy measurements and low resolution, making them insufficient for detailed reconstruction on their own. This paper addresses the challenge of combining the strengths of these disparate sensors – the detailed shape information from camcorder silhouettes and the direct depth measurements from RIM cameras – to achieve more robust and accurate 3D reconstructions. A key hurdle is the inherent heterogeneity of the data and the noise present in RIM sensors, which requires a sophisticated fusion strategy.

Methods

The core of the paper is a probabilistic sensor fusion framework built upon a space occupancy grid representation of the 3D scene. This framework models the probability of each voxel in the grid being occupied by the object. The key innovations include:

  • Heterogeneous Sensor Calibration: A crucial first step is calibrating the RIM cameras. They propose a simple yet effective depth calibration method that leverages existing geometric camera calibration techniques and the known poses of checkerboard patterns. This calibration corrects for the inherent noise and inaccuracies in RIM depth measurements.
  • Probabilistic Sensor Models: For each sensor type (camcorder and RIM camera), they define probabilistic sensing models.
  • Camcorder Model: This model uses silhouette cues. Since direct silhouette extraction can be sensitive to noise, they treat the silhouette as a latent variable and marginalize it out, making the fusion more robust.
  • RIM Camera Model: This model incorporates the noisy depth measurements. It also uses a latent variable to represent the "front-most surface" along a viewing ray, which is then used to model the depth observation probability.
  • Bayesian Inference: The reconstruction problem is framed as a Bayesian inference problem. Given the sensor observations and their respective models, they infer the posterior probability distribution of the occupancy grid. This probabilistic approach naturally handles sensor noise and variations.
  • Scalable Framework: The proposed framework is designed to be general and scalable, capable of fusing various sensor types (not just silhouettes and depth) as long as their sensing models can be defined.

Datasets

The authors used two sets of data for their experiments:

  • Sensor Network I: Consisted of 3 Canon HG10 camcorders and 1 Swiss Ranger 3100 RIM camera.
  • Sensor Network II: Utilized 9 cameras in total, including SR3100 RIM cameras.

They used checkerboard patterns for geometric and depth calibration. The datasets involved reconstructing static scenes like an office chair with boxes and a sitting person.

Key Insights and Ideas

  • Complementarity of Sensors: The paper strongly demonstrates that camcorder silhouettes and RIM camera depth measurements are highly complementary. Silhouettes provide global shape constraints, while depth data offers local surface details and helps recover concavities that visual hulls miss.
  • Importance of Depth Calibration: The RIM camera depth calibration is shown to be critical. Without it, the reconstructed shapes are significantly less accurate and do not align well with the visual hull. This highlights that raw depth data from consumer-grade RIM sensors is often too noisy for direct use in reconstruction.
  • Probabilistic Fusion for Robustness: Framing the fusion as a Bayesian inference problem using a space occupancy grid makes the system inherently robust to sensor noise. The probabilistic models capture the uncertainty in measurements, leading to more reliable reconstructions.
  • Generality of the Framework: The proposed framework is not limited to the specific sensors used. Its probabilistic nature and reliance on sensor models make it adaptable to other sensor types, suggesting broad applicability in multi-sensor 3D reconstruction.

Key Results

  • The proposed heterogeneous sensor fusion framework significantly improves the quality of 3D object reconstruction compared to using either camcorder silhouettes or RIM camera depth alone.
  • The depth calibration for RIM cameras was shown to be effective, leading to reconstructions that are tangent to the visual hull, indicating accurate recovery of the object's front-most surface.
  • The method successfully reconstructed detailed shapes, including concavities, which are challenging for pure visual hull methods. For example, a reconstruction of a person with a rubber ball showed good agreement with the actual object size.
  • In a challenging scenario with a densely populated scene of 9 people, the fusion approach was able to recover details and concavities that a visual hull alone would fail to capture, resulting in an indistinguishable blob.

Limitations and Future Work

  • Resolution Mismatch: The paper acknowledges the resolution difference between high-res camcorders and low-res RIM cameras. While their framework handles this, they note that missing forearms in some reconstructions were sub-voxel size and could be recovered by increasing volume resolution.
  • Dynamic Scenes: While the paper mentions the potential for real-time reconstruction of dynamic scenes due to high frame rates, the experiments primarily focused on static scenes.
  • Future Work: The authors suggest that their framework can be extended to analyze RIM camera's impulse-response properties more deeply, such as depth measurement variations with respect to infrared light incident angle or material reflectance. They also note that the computation can be parallelized on GPUs, paving the way for faster real-time applications.

Summary updated

Abstract

In this paper, we reconstruct 3D objects with a heterogeneous sensor network of Range Imaging(RIM) sensors and high-res camcorders. With this setup, we first carry out simple but effective depth calibration for the RIM cameras. We then combine the camcorder silhouette cues and RIM camera depth information, for the reconstruction. Our main contribution is the proposal of a sensor fusion framework so that the computation is general, simple and scalable. Although we only discuss the camcorders and RIM cameras in this paper, the proposed framework can be applied to any type of vision sensors. It uses a space occupancy grid as a probabilistic 3D representation of scene contents. After defining sensing models for each type of sensors, the reconstruction is simply a Bayesian inference problem, and can be solved robustly. The experiments show that the recover full 3D closed shapes substantially improved the quality of the noisy RIM sensor measurement.
BibTeX
@misc{guan2008d,
  author = {Li Guan and Jean-Sébastien Franco and Marc Pollefeys},
  title = {{3D Object Reconstruction with Heterogeneous Sensor Data}},
  year = {2008},
}
Figure