Live 3D shape reconstruction, recognition and registration

Carlos Hernández, Frank Perbet, Minh-Tri Pham, George Vogiatzis, Oliver J. Woodford, Atsuto Maki, Björn Stenger, Roberto CipollaWorkshops @ ICCV 2011 · OpenReview

Abstract

We present a video-based system which interactively captures the geometry of a 3D object in the form of a point cloud, then recognizes and registers known objects in this point cloud in a matter of seconds. In order to achieve interactive speed, we exploit both efficient inference algorithms and parallel computation, often on a GPU. The system can be broken down into two distinct phases: geometry capture, and object inference.

Introduction

While 3D scanning and object recognition are well-established fields, performing both simultaneously and interactively remains a significant challenge. Most existing systems focus either on high-fidelity offline reconstruction or fast but limited 2D-to-3D object detection. This work addresses the gap of interactive 3D scene understanding, specifically tackling the computational bottleneck of matching hundreds of thousands of correspondences across video frames and the difficulty of recognizing objects within noisy, sparse point clouds. The goal is a system where a user can move a camera around an object and see a reconstructed, labeled 3D model emerge in real-time.

The live system in use, showing real-time reconstruction and object overlay.

Methodology

The system architecture is divided into two primary phases: Geometry Capture and Object Inference.

Geometry Capture

The geometry capture phase begins with camera pose estimation, utilizing either PTAM (visual SLAM) for general tracking or a known planar pattern for metric-accurate estimation. Once the pose is established, dense reconstruction is performed by searching for correspondences along 1D epipolar lines. To maintain interactive speeds, a probabilistic framework is employed to update the state of each point over time rather than recomputing from scratch. The matching process uses Normalized Cross Correlation (NCC) on $5 \times 5$ windows, implemented entirely on the GPU for massive parallelization.

Object Inference

Once a point cloud is generated, the system performs recognition and registration through several steps:

  1. Voxelization: The point cloud is converted into a $128^3$ voxel volume.
  2. Feature Extraction: A Difference of Gaussians (DoG) detector identifies features across scales, and Principal Component Analysis (PCA) determines local orientation, resulting in a 7D feature pose.
  3. 8D Voting: Features are matched against a database of known objects, with each match casting an 8D vote representing object class and 3D pose.
  4. Minimum-Entropy Hough Transform: A specialized Hough transform filters noise and identifies the most consistent object matches.
Key steps in the object inference pipeline: voxelization, feature extraction, voting, and detection.

Key Insights

A core design principle of the system is the independence of point computation during geometry capture. By avoiding complex regularization, the system can leverage the full power of GPU parallelization. Furthermore, by converting raw point clouds into voxel volumes, the authors successfully adapt 2D computer vision techniques like DoG and PCA to 3D space. Physical constraints, such as object size and ground-plane contact, are used as heuristics to prune false positives and improve robustness.

Results and Discussion

The system achieves recognition and registration in a matter of seconds, maintaining interactive frame rates throughout the reconstruction process. As demonstrated in the figures, the system accurately overlays recognized 3D models directly onto the live video feed and the captured point cloud.

Despite its success, the system currently assumes a static scene for geometry capture. Additionally, while PTAM provides tracking, object inference requires a metric scale, which is currently most reliably provided by a planar pattern. Future work could explore more generalized metric SLAM approaches and more memory-efficient data structures, such as Octrees, to move beyond the current $128^3$ voxel resolution limit.

Summary updated

Abstract

We present a video-based system which interactively captures the geometry of a 3D object in the form of a point cloud, then recognizes and registers known objects in this point cloud in a matter of seconds (fig. 1). In order to achieve interactive speed, we exploit both efficient inference algorithms and parallel computation, often on a GPU. The system can be broken down into two distinct phases: geometry capture, and object inference. We now discuss these in further detail.
BibTeX
@inproceedings{hernandez2011live,
  author = {Carlos Hernández and Frank Perbet and Minh-Tri Pham and George Vogiatzis and Oliver J. Woodford and Atsuto Maki and Björn Stenger and Roberto Cipolla},
  title = {{Live 3D shape reconstruction, recognition and registration}},
  booktitle = {Workshops @ ICCV},
  year = {2011},
  url = {https://openreview.net/forum?id=DoHvFnHm-T},
}
Figure