Probabilistic Depth Image Registration incorporating Nonvisual Information

Manuel Wüthrich, Peter Pastor, Ludovic Righetti, Aude Billard, Stefan SchaalarXiv 2015 · DOI · arXiv

TL;DR

This paper introduces a novel Bayesian framework for 3D depth image registration that combines visual data with nonvisual motion priors (like robot kinematics). By incorporating "negative space" visibility constraints and outputting a full probability distribution over the 6D transformation rather than a single point estimate, the algorithm achieves highly robust tracking even on sparse, featureless, or poorly aligned point clouds.

What Problem Does This Paper Address?

3D image registration—finding the relative alignment between two overlapping 3D point clouds—is a foundational problem in robotics for tasks like object modeling and tracking. Traditional approaches like Iterative Closest Point (ICP) and feature mapping have significant vulnerabilities:

  1. Local Minima: ICP requires a good initial guess; otherwise, it easily gets stuck in local minima if the object moved quickly between frames.
  2. Feature Dependency: Feature mapping fails on objects with homogeneous textures, uniform colors, or symmetrical geometry.
  3. Wasted Information: Standard algorithms only look at the points on the surface of the object. They completely ignore nonvisual context (e.g., knowing a robot pushed an object across a flat table) and visibility constraints (knowing that certain space is empty because the camera can see through it).

Furthermore, most algorithms output a single "best guess" (point estimate) and a generic fitness score, making it difficult to mathematically propagate directional uncertainty.

Methods

The authors address these gaps by deriving a registration algorithm within a Bayesian framework to compute a posterior distribution $p(\boldsymbol{T}|\boldsymbol{D})$ over the 6-degree-of-freedom rigid-body transformation $\boldsymbol{T}$.

As shown in the variable overview below, the input data $\boldsymbol{D}$ is explicitly split into nonvisual information $\boldsymbol{N}$ and visual information $\boldsymbol{V}$:

Overview of the variables used in the probabilistic framework.

The visual information $\boldsymbol{V}$ is further divided into two components:

  • Patches ($\boldsymbol{P}$): The standard 3D point cloud representing observed object surfaces.
  • Mask ($\boldsymbol{M}$): The "negative space" or line-of-sight between the camera and the object/background. Because the camera has a clear line of sight, no part of the object can exist inside this green frustum zone.
The two types of visual information: surface patches (red) and the visibility mask (green).

To model camera measurement errors cleanly, the authors project Cartesian coordinates into ray coordinates $(w, h, r)$, where $w$ and $h$ represent pixel projections on a virtual image plane and $r$ is the depth along the ray.

Pinhole camera model parametrization using ray coordinates.

Because computing the exact analytical posterior distribution is intractable, the core implementation uses a Monte Carlo sampling approach. The algorithm draws candidate transformations from a sampling distribution dictated by the nonvisual prior, evaluates their likelihood against the visual patches and masks, and computes the mean and covariance matrix of the transformation.

Datasets

The algorithm was evaluated using a custom dataset captured with a Microsoft Kinect camera.

  • Objects: Three items with distinct geometric properties: a box, a flashlight, and a tube.
  • Protocol: Each object was placed on a tabletop and sequentially rotated by roughly $25^\circ$ and translated by a few centimeters 14 times. Exact ground-truth positions and orientations were measured to provide 13 evaluation alignments per object.
The three objects used in the dataset: a box, a flashlight, and a tube.

Key Insights and Ideas

  • Empty Space is Informative: Traditional ICP only penalizes distances between points. The Bayesian formulation here penalizes transformations that project object points into known background or free space. If a proposed transformation places a piece of the object in front of a clear background, the mask constraint quickly drives that sample's likelihood to zero.
  • Priors Tame the Sampling Explosion: Sampling a full 6DOF space randomly is computationally brutal. However, adding simple physical constraints—such as "the object is sliding on a flat table"—collapses the search space, making sampling highly efficient.
  • Directional Uncertainty Matters: By outputting a full covariance matrix, the algorithm can express nuanced certainties (e.g., "we are highly confident about the object's X/Y translation, but uncertain about its rotation around the Z-axis"). This allows standard graph optimization backends (like g2o) to execute precise loop-closure when stitching together full 3D object models.

Key Results

The authors compared their algorithm against standard implementations in the Point Cloud Library (PCL), including ICP, Feature Mapping (FM) using Fast Point Feature Histograms (FPFH), and a combined FM+ICP pipeline.

  • Without Nonvisual Information: Even when assuming no motion priors (using a broad search space), the proposed method achieved significantly lower alignment errors than ICP and feature mapping.
Boxplot comparing alignment errors across different algorithms.
  • With Nonvisual Information: When the "tabletop plane" prior was introduced, the sampling search space shrank dramatically. Computational time plummeted from 30 seconds down to 0.5 seconds per alignment.
  • Robustness to Sparse Data: In extreme cases with highly sparse point clouds where standard PCL pipelines failed entirely (errors $>51^\circ$), the proposed method with the tabletop prior successfully aligned the frames with an error of just $6.3^\circ$.

Limitations and Future Work

The primary bottleneck of the current implementation is computational speed when a motion prior is absent. Drawing and evaluating 100 million samples takes roughly 30 seconds per frame on a CPU, which is far too slow for real-time robotic tracking.

To resolve this, the authors note that the sampling core is inherently parallelizable and could be implemented on a GPU for massive speedups. Additionally, they suggest that feature mapping could be used not as a final alignment tool, but to intelligently guide the initial sampling distribution when nonvisual priors are unavailable. Finally, future work includes testing the algorithm in more complex robotic scenarios, such as tracking objects directly within a robot's gripper.

Summary updated

Abstract

In this paper, we derive a probabilistic registration algorithm for object modeling and tracking. In many robotics applications, such as manipulation tasks, nonvisual information about the movement of the object is available, which we will combine with the visual information. Furthermore we do not only consider observations of the object, but we also take space into account which has been observed to not be part of the object. Furthermore we are computing a posterior distribution over the relative alignment and not a point estimate as typically done in for example Iterative Closest Point (ICP). To our knowledge no existing algorithm meets these three conditions and we thus derive a novel registration algorithm in a Bayesian framework. Experimental results suggest that the proposed methods perform favorably in comparison to PCL implementations of feature mapping and ICP, especially if nonvisual information is available.
BibTeX
@misc{wuthrich2015probabilistic,
  author = {Manuel Wüthrich and Peter Pastor and Ludovic Righetti and Aude Billard and Stefan Schaal},
  title = {{Probabilistic Depth Image Registration incorporating Nonvisual Information}},
  year = {2015},
  eprint = {1504.07857},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1504.07857},
}
Figure