Robust real-time 3D modeling of static scenes using solely a Time-of-Flight sensor

Johannes Feulner, Jochen Penne, Eva N. K. Kollorz, Joachim HorneggerWorkshops @ CVPR 2009 · DOI

Abstract

Creating 3D models of environments typically requires high-resolution sensors or complex sensor fusion. This paper proposes a method to achieve real-time 3D registration using only a single, low-resolution Time-of-Flight (ToF) camera. By maximizing the correlation between binarized edge features across frames, the system achieves robust tracking without external positioning data.

Introduction

Generating a comprehensive 3D map from a camera with a limited field of view necessitates "registration"—the alignment of multiple overlapping frames into a unified coordinate system. While traditional approaches often fuse low-resolution depth data with high-resolution RGB images or Inertial Measurement Units (IMUs), this work explores the feasibility of using a ToF sensor in isolation.

ToF sensors from the mid-2000s, such as the CSEM SR-3100, presented significant challenges: low lateral resolution (144x176 pixels) and high noise levels. Standard algorithms like Iterative Closest Point (ICP) are often too computationally intensive or prone to failure when applied to such sparse, noisy data.

Methodology

The core of the proposed approach is a registration pipeline that prioritizes feature correlation over raw point-to-point matching.

Registration Pipeline

The process involves detecting features in the current frame and comparing them against projections from a previous frame or a global model.

Overview of the registration pipeline using feature correlation and Levenberg-Marquardt optimization.

As shown in the pipeline above, the algorithm seeks to find the optimal pose $\Pi$ that maximizes the similarity between observed 2D projections and computed projections of 3D features.

Feature Extraction and Correlation

The system utilizes a "structure tensor" to detect edges in both the intensity (amplitude) and depth (z-coordinate) channels.

Visualization of edge detection: (a) raw intensity image, (b) detected depth edges, and (c) combined feature set used for registration.

Instead of minimizing Euclidean distances, the authors maximize the uncentered correlation coefficient (cosine similarity) of these detected edges. This metric is inherently more robust; it does not penalize "empty" regions where no edges are detected, which prevents the optimizer from converging on incorrect local minima in sparse scenes.

Optimization and Robustness

To handle arbitrary camera motion in real-time, several extensions were implemented:

  • Motion Prediction: Estimates the subsequent pose based on constant velocity or acceleration models.
  • Multiresolution Hierarchy: Optimization begins on downsampled, blurred data to capture coarse alignment before refining on higher-resolution details.
  • Random Search: Stochastic perturbations are applied to the estimated pose to help the algorithm escape local optima.
  • Levenberg-Marquardt Optimization: This iterative solver is used to find the best camera pose by minimizing the correlation-based error function.

Experimental Results

The system was evaluated using 13 indoor video sequences captured with a CSEM SR-3100 camera. These sequences included various complex movements such as tilting, panning, and "zigzag" translations. Ground truth was established using high-precision mechanical pan-tilt units.

Key Findings

  • Binary Features: The authors discovered that using binary edge information (presence vs. absence of an edge) yielded better convergence than continuous edge strength values. Continuous values often caused the optimizer to over-focus on high-contrast edges while ignoring subtler, informative features.
  • Accuracy and Speed: The "Extended2" version of the algorithm achieved a mean relative registration error of approximately 45 mm. On a 1.6 GHz Pentium M processor, the system maintained a frame rate of 3 to 4 fps, which is sufficient for interactive 3D modeling.
  • Robustness: The combination of motion prediction and multiresolution hierarchies reduced errors by nearly an order of magnitude in fast-moving sequences compared to a basic registration approach.

Discussion and Limitations

While the system demonstrates the power of ToF-only modeling, it is currently limited to static scenes. Dynamic objects, such as people moving through the frame, would disrupt the registration process. Additionally, like all incremental methods, the system suffers from "drift"—accumulated errors over long sequences that can reach several hundred millimeters.

Future work could involve integrating low-cost IMUs to improve initial pose estimation or exploring loop-closure techniques to mitigate long-term drift. Despite these limitations, the research proves that robust 3D modeling is possible even with the constraints of early-generation ToF hardware.

Summary updated

BibTeX
@inproceedings{feulner2009robust,
  author = {Johannes Feulner and Jochen Penne and Eva N. K. Kollorz and Joachim Hornegger},
  title = {{Robust real-time 3D modeling of static scenes using solely a Time-of-Flight sensor}},
  booktitle = {Workshops @ CVPR},
  year = {2009},
  doi = {10.1109/cvprw.2009.5205204},
}
Figure