Learning Better Keypoints for Multi-Object 6DoF Pose Estimation

Yangzheng Wu, Michael GreenspanWACV 2024

TL;DR

This paper introduces KeyGNet, a novel graph network that learns optimal keypoint locations for 6DoF object pose estimation. By learning keypoints that generate similarly distributed votes, KeyGNet significantly improves the accuracy and efficiency of existing pose estimation methods, particularly in challenging multi-object scenarios.

What Problem Does This Paper Address?

Estimating the 6 Degrees of Freedom (6DoF) pose of objects in a 3D scene is a fundamental problem in computer vision with applications ranging from robotics to augmented reality. A popular approach to this problem involves identifying keypoints on an object. However, the selection of these keypoints has traditionally relied on heuristic methods like Farthest Point Sampling (FPS) or selecting corners of bounding boxes. These methods are purely geometry-based and ignore other visual cues like color. Furthermore, the distribution of "votes" that these keypoints generate can be uneven, leading to issues akin to class imbalance during training and hindering performance. This paper addresses the challenge of finding a more effective and data-driven way to select these crucial keypoints, aiming to improve the accuracy and robustness of 6DoF pose estimation methods.

Methods

The core contribution of this paper is KeyGNet, a graph convolutional network designed to learn optimal keypoint locations. Unlike previous methods that relied on hand-crafted heuristics, KeyGNet learns these locations directly from data. The network takes as input a point cloud representing the visible surface of objects and outputs a set of optimized keypoints.

KeyGNet's training is guided by a novel loss function with two main components:

  • Wasserstein Loss ($L_{wass}$): This term encourages the "votes" cast by different points in the scene towards each keypoint to have similar distributions. This is inspired by Wasserstein Generative Adversarial Networks (WGANs) and helps to regularize the learning process, preventing issues similar to class imbalance.
  • Dispersion Loss ($L_{dis}$): This term promotes geometric separation between the learned keypoints, ensuring they are spread out across the object's surface. This is similar in spirit to the FPS method but learned.

The combined loss function, $\alpha L_{wass} + \beta L_{dis}$, is optimized to find a set of keypoints that are both geometrically dispersed and generate similarly distributed votes. The paper also explores dynamic scheduling of the $\alpha$ and $\beta$ weights during training to balance the influence of these two terms.

Datasets

The paper evaluates KeyGNet on several standard datasets for 6DoF pose estimation:

  • BOP (Benchmark for 6D Object Pose Estimation) core datasets: Seven datasets are used, which provide standardized evaluation metrics for various application fields.
  • LINEMOD (LM): A classic dataset for object pose estimation.
  • Occlusion LINEMOD (LMO): A more challenging variant of LINEMOD with occlusions.
  • YCB Video (YCB-V): A dataset featuring objects from the YCB object and model dataset.

The paper does not introduce new datasets but focuses on improving performance on existing ones.

Key Insights and Ideas

  • Learning keypoints is more effective than heuristics: The paper convincingly demonstrates that a data-driven approach to keypoint selection, as implemented by KeyGNet, significantly outperforms traditional heuristic methods like FPS and BBox selection.
  • Vote distribution matters: The core insight is that the similarity in the distribution of votes cast by scene points to each keypoint is crucial for accurate pose estimation. Uneven distributions create training challenges, similar to class imbalance.
  • Wasserstein distance is a powerful tool for distribution similarity: The paper highlights the effectiveness of Wasserstein distance in measuring and enforcing similar vote distributions, even when these distributions do not overlap.
  • Keypoints can bridge the SISO-MIMO gap: KeyGNet's ability to learn a single set of keypoints for all objects in a dataset significantly reduces the performance degradation typically observed when moving from single-object (SISO) to multi-object (MIMO) pose estimation.

Key Results

  • Improved Accuracy: KeyGNet-selected keypoints consistently improved the performance of three state-of-the-art 6DoF PE methods (RCVPose, PVNet, and PVN3D) across all seven BOP core datasets. Improvements in metrics like ADD(S) and BOP AR ranged from 1% to over 21.5%.
  • Reduced SISO-MIMO Gap: The performance gap between SISO and MIMO training was significantly reduced when using KeyGNet keypoints, with some methods even showing slight improvements in MIMO mode.
  • Faster Training: In MIMO scenarios, using KeyGNet keypoints led to reduced training times, with savings of up to 11 hours reported.
  • Robustness to Occlusion: The challenging Occlusion LINEMOD dataset saw a notable improvement of +16.4% on ADD(S) for PVN3D when using KeyGNet keypoints.

Limitations and Future Work

  • Computational Cost: While KeyGNet improves inference speed in MIMO, the training of KeyGNet itself is a separate process that adds to the overall pipeline.
  • Dataset-Specific Keypoints: KeyGNet learns a set of keypoints for a given dataset. While this is effective, it implies that keypoints might need to be re-learned for entirely new object categories or datasets.
  • Future Work: The authors suggest exploring the application of KeyGNet to other keypoint-based computer vision tasks and investigating more advanced graph network architectures for keypoint learning. They also hint at the potential for KeyGNet to be integrated into end-to-end trainable systems.

Summary updated

BibTeX
@inproceedings{wu2024learning,
  author = {Yangzheng Wu and Michael Greenspan},
  title = {{Learning Better Keypoints for Multi-Object 6DoF Pose Estimation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}

Loading related content...

Figure