3D Object Detection and Pose Estimation of Unseen Objects in Color\n Images with Local Surface Embeddings

Giorgia Pitteri, Aurélie Bugeau, Slobodan Ilić, Vincent LepetitarXiv 2020 · DOI · arXiv · OpenAlex

Abstract

We present an approach for detecting and estimating the 3D poses of objects in images that requires only an untextured CAD model and no training phase for new objects. Our approach combines Deep Learning and 3D geometry: It relies on an embedding of local 3D geometry to match the CAD models to the input images. For points at the surface of objects, this embedding can be computed directly from the CAD model; for image locations, we learn to predict it from the image itself. This establishes correspondences between 3D points on the CAD model and 2D locations of the input images. However, many of these correspondences are ambiguous as many points may have similar local geometries. We show that we can use Mask-RCNN in a class-agnostic way to detect the new objects without retraining and thus drastically limit the number of possible correspondences. We can then robustly estimate a 3D pose from these discriminative correspondences using a RANSAC-like algorithm. We demonstrate the performance of this approach on the T-LESS dataset, by using a small number of objects to learn the embedding and testing it on the other objects. Our experiments show that our method is on par or better than previous methods.

Introduction

In industrial automation, the ability to quickly adapt to new parts is crucial. Traditional deep learning methods for 6D pose estimation—determining an object's precise position and orientation—often require extensive labeled datasets and significant retraining time for every new object. This paper addresses the "unseen object" challenge: estimating the pose of a novel object using only its untextured CAD model and a standard RGB image. This task is particularly demanding for industrial components, which are frequently textureless, symmetrical, and situated in cluttered, occluded environments.

Methodology: The LSE Pipeline

The proposed method bridges the gap between deep learning and classical 3D geometry by focusing on local geometric features rather than global appearance. The system follows a multi-stage pipeline to establish robust 2D-3D correspondences.

The complete pipeline: LSE prediction and Mask R-CNN segmentation are combined with a CAD database for PnP-based pose estimation.

Local Surface Embeddings (LSEs)

The core innovation is the use of Local Surface Embeddings (LSEs). These are rotation-invariant geometric "fingerprints" derived from the local neighborhood of a 3D point. By decomposing the covariance matrix of local points, the authors create a local coordinate system that remains consistent regardless of the object's global orientation.

Database of textureless CAD models and their computed Local Surface Embeddings (LSEs).

Prediction and Segmentation

  1. LSE Prediction Network: A U-Net architecture is trained on a small set of known objects to predict LSE values for every pixel in an RGB image. Because LSEs represent universal geometric primitives (like specific curves or edges), the network generalizes effectively to unseen surfaces.
  2. Class-Agnostic Segmentation: To manage the search space for correspondences, a Mask R-CNN is employed in a class-agnostic manner. It identifies object boundaries without needing to classify the object, providing a spatial constraint that makes the subsequent pose estimation computationally feasible.

Pose Estimation

Final 6D pose is calculated by matching the predicted 2D LSEs from the image to the pre-computed 3D LSEs of the CAD model. A RANSAC-based Perspective-n-Point (PnP) algorithm is used to robustly filter out ambiguous matches and determine the optimal transformation.

Experimental Results

The approach was evaluated on the T-LESS dataset, which contains 30 textureless, symmetrical industrial objects. The model was trained on a small subset of objects and tested on entirely "unseen" ones.

The method demonstrated superior performance compared to existing zero-shot approaches:

  • Recall: On T-LESS, the proposed method achieved an average recall of 46.7%, significantly outperforming CorNet's 34.7%.
  • Accuracy: Using the Visible Surface Discrepancy (VSD) metric, the method scored 23.27, surpassing MP-Encoder's 20.53.
  • Robustness: Qualitative results showed the system remains effective even when objects are rendered with random textures, confirming its reliance on geometric cues rather than color or texture.

Conclusion

By prioritizing geometry over appearance, this method enables 6D pose estimation for novel objects without retraining. While challenges remain—specifically regarding highly symmetrical objects and unique local geometries not present in the training set—the use of Local Surface Embeddings provides a powerful framework for zero-shot object detection. Future improvements could involve incorporating depth data (RGB-D) or Iterative Closest Point (ICP) refinement to further enhance precision.

Summary updated

Abstract

We present an approach for detecting and estimating the 3D poses of objects\nin images that requires only an untextured CAD model and no training phase for\nnew objects. Our approach combines Deep Learning and 3D geometry: It relies on\nan embedding of local 3D geometry to match the CAD models to the input images.\nFor points at the surface of objects, this embedding can be computed directly\nfrom the CAD model; for image locations, we learn to predict it from the image\nitself. This establishes correspondences between 3D points on the CAD model and\n2D locations of the input images. However, many of these correspondences are\nambiguous as many points may have similar local geometries. We show that we can\nuse Mask-RCNN in a class-agnostic way to detect the new objects without\nretraining and thus drastically limit the number of possible correspondences.\nWe can then robustly estimate a 3D pose from these discriminative\ncorrespondences using a RANSAC- like algorithm. We demonstrate the performance\nof this approach on the T-LESS dataset, by using a small number of objects to\nlearn the embedding and testing it on the other objects. Our experiments show\nthat our method is on par or better than previous methods.\n
BibTeX
@misc{pitteri2020d,
  author = {Giorgia Pitteri and Aurélie Bugeau and Slobodan Ilić and Vincent Lepetit},
  title = {{3D Object Detection and Pose Estimation of Unseen Objects in Color\textbackslash\{\}n Images with Local Surface Embeddings}},
  year = {2020},
  doi = {10.48550/arxiv.2010.04075},
}

Loading related content...

Figure