Abstract
We present a video-based system which interactively captures the geometry of a 3D object in the form of a point cloud, then recognizes and registers known objects in this point cloud in a matter of seconds. In order to achieve interactive speed, we exploit both efficient inference algorithms and parallel computation, often on a GPU. The system can be broken down into two distinct phases: geometry capture, and object inference.
Introduction
While 3D scanning and object recognition are well-established fields, performing both simultaneously and interactively remains a significant challenge. Most existing systems focus either on high-fidelity offline reconstruction or fast but limited 2D-to-3D object detection. This work addresses the gap of interactive 3D scene understanding, specifically tackling the computational bottleneck of matching hundreds of thousands of correspondences across video frames and the difficulty of recognizing objects within noisy, sparse point clouds. The goal is a system where a user can move a camera around an object and see a reconstructed, labeled 3D model emerge in real-time.
Methodology
The system architecture is divided into two primary phases: Geometry Capture and Object Inference.
Geometry Capture
The geometry capture phase begins with camera pose estimation, utilizing either PTAM (visual SLAM) for general tracking or a known planar pattern for metric-accurate estimation. Once the pose is established, dense reconstruction is performed by searching for correspondences along 1D epipolar lines. To maintain interactive speeds, a probabilistic framework is employed to update the state of each point over time rather than recomputing from scratch. The matching process uses Normalized Cross Correlation (NCC) on $5 \times 5$ windows, implemented entirely on the GPU for massive parallelization.
Object Inference
Once a point cloud is generated, the system performs recognition and registration through several steps:
- Voxelization: The point cloud is converted into a $128^3$ voxel volume.
- Feature Extraction: A Difference of Gaussians (DoG) detector identifies features across scales, and Principal Component Analysis (PCA) determines local orientation, resulting in a 7D feature pose.
- 8D Voting: Features are matched against a database of known objects, with each match casting an 8D vote representing object class and 3D pose.
- Minimum-Entropy Hough Transform: A specialized Hough transform filters noise and identifies the most consistent object matches.
Key Insights
A core design principle of the system is the independence of point computation during geometry capture. By avoiding complex regularization, the system can leverage the full power of GPU parallelization. Furthermore, by converting raw point clouds into voxel volumes, the authors successfully adapt 2D computer vision techniques like DoG and PCA to 3D space. Physical constraints, such as object size and ground-plane contact, are used as heuristics to prune false positives and improve robustness.
Results and Discussion
The system achieves recognition and registration in a matter of seconds, maintaining interactive frame rates throughout the reconstruction process. As demonstrated in the figures, the system accurately overlays recognized 3D models directly onto the live video feed and the captured point cloud.
Despite its success, the system currently assumes a static scene for geometry capture. Additionally, while PTAM provides tracking, object inference requires a metric scale, which is currently most reliably provided by a planar pattern. Future work could explore more generalized metric SLAM approaches and more memory-efficient data structures, such as Octrees, to move beyond the current $128^3$ voxel resolution limit.
Summary updated