3D Registration for Self-Occluded Objects in Context

Zheng Dang, Fei Wang, Mathieu SalzmannarXiv 2020 · arXiv

Abstract

Estimating the 6D pose of objects from a single 2.5D depth scan in cluttered environments remains a significant challenge in computer vision and robotics. Traditional methods often struggle with local optima or require clean point clouds. This paper introduces a robust, end-to-end deep learning framework that combines instance segmentation with a pose estimation network enhanced by Optimal Transport. By explicitly modeling outliers and handling self-occlusion, the proposed method achieves state-of-the-art "one-shot" registration accuracy and real-time performance on both synthetic and real-world datasets.

Introduction

3D registration—the process of aligning a 3D model to observed points—is fundamental for tasks ranging from robotic manipulation to autonomous navigation. However, real-world scenarios involve "self-occluded objects in context," where sensors only capture partial surfaces (2.5D) amidst significant background clutter. Existing algorithms like Iterative Closest Point (ICP) are prone to failure in these high-outlier environments. This work addresses the gap between complete 3D CAD models and partial, noisy observations by proposing a pipeline that isolates the target object before performing geometric alignment.

Methodology

The proposed architecture follows a two-stage pipeline designed for efficiency and robustness. The first stage utilizes an adapted YOLACT framework for instance segmentation on depth maps, isolating the target object's points from background context like floors and walls.

The two-stage pipeline: Instance Segmentation followed by Pose Estimation.

The second stage performs pose estimation using a network based on Deep Closest Point (DCP-v2). A critical innovation is the integration of an Optimal Transport layer using the Sinkhorn algorithm. Unlike standard SoftMax, this layer includes an "outlier bin," allowing the network to ignore points that lack valid matches due to imperfect segmentation or self-occlusion. Features are extracted using Dynamic Graph CNN (DGCNN) and refined via a Transformer to capture cross-source relationships. To enhance generalization, the authors implemented an on-the-fly rendering strategy using PyTorch3D, exposing the model to a virtually infinite variety of object poses during training.

Experimental Results

The framework was evaluated on ModelNet40, BlenderProc (cluttered synthetic), and TUD-L (real-world) datasets. The method demonstrated superior accuracy compared to traditional handcrafted feature-based methods like TEASER++ and previous learning-based models.

Qualitative results showing depth maps, segmentation masks, and final 3D registration across various object categories.

Key findings include:

  • Robustness to Clutter: The system maintained high precision even when background points constituted over 95% of the scene.
  • Real-Time Performance: The entire pipeline is highly efficient, with the pose estimation stage requiring between 0.03s and 0.11s, making it suitable for live robotics applications.
  • Generalization: Evaluations on the TUD-L dataset proved that the model, trained primarily on synthetic data, generalizes effectively to real-world sensor data.

Conclusion

This work demonstrates that combining learned segmentation with robust geometric matching is essential for practical 3D registration in cluttered scenes. While performance remains tied to the quality of initial segmentation, the inclusion of Optimal Transport significantly mitigates noise and occlusion issues. Future research aims to integrate these stages more tightly and deploy the technology for complex tasks such as non-cooperative satellite rendezvous in space.

Summary updated

Abstract

While much progress has been made on the task of 3D point cloud registration, there still exists no learning-based method able to estimate the 6D pose of an object observed by a 2.5D sensor in a scene. The challenges of this scenario include the fact that most measurements are outliers depicting the object's surrounding context, and the mismatch between the complete 3D object model and its self-occluded observations. We introduce the first deep learning framework capable of effectively handling this scenario. Our method consists of an instance segmentation module followed by a pose estimation one. It allows us to perform 3D registration in a one-shot manner, without requiring an expensive iterative procedure. We further develop an on-the-fly rendering-based training strategy that is both time- and memory-efficient. Our experiments evidence the superiority of our approach over the state-of-the-art traditional and learning-based 3D registration methods.
BibTeX
@misc{dang2020d,
  author = {Zheng Dang and Fei Wang and Mathieu Salzmann},
  title = {{3D Registration for Self-Occluded Objects in Context}},
  year = {2020},
  eprint = {2011.11260},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.11260},
}
Figure