Multi-View Metal Parts Pose Estimation Based on a Single Camera

Chen Chen, Xin JiangSensors 2024 · DOI

TL;DR

This paper presents a novel multi-view approach for estimating the 6D pose of shiny, textureless metal parts using only a single RGB camera. By intelligently planning subsequent camera viewpoints using ray casting and then fusing pose estimates, the method achieves robust pose recovery without relying on depth sensors.

What Problem Does This Paper Address?

Accurate 6D pose estimation of industrial parts is crucial for tasks like robotic grasping and manipulation. However, metal parts pose a significant challenge due to their reflective surfaces, which often lead to incomplete or noisy depth data from standard RGB-D sensors. Traditional methods struggle with textureless, reflective objects, and even advanced RGB-based approaches can suffer from scale and depth ambiguities from a single viewpoint. This paper tackles the problem of reliably estimating the pose of such challenging objects using only readily available RGB images, by leveraging the power of multiple views.

Methods

The proposed method employs a three-stage pipeline:

  1. Single-View Pose Estimation: The process begins by detecting metal parts in an initial RGB image and predicting their semantic keypoints. Using a network like HRNet, the system then estimates the 6D pose of each detected part relative to the camera.
  2. Next-Best-View Planning: To overcome single-view limitations, the method intelligently plans the next camera viewpoint. It uses ray casting within a simulated environment (based on CAD models) to identify the view that would reveal the most new information about the object. This is achieved by calculating the "additional hit points" – essentially, the newly visible surface area – from potential viewpoints. The viewpoint offering the maximum additional visible area is selected as the "next best view."
  3. Pose Refinement in Different Views: Once the camera is moved to the planned next best view, the pose is estimated again. The system then fuses the pose estimates from all acquired views, weighting them by confidence scores derived from keypoint heatmaps. This multi-view fusion, combined with known camera transformations, refines the overall scene pose and improves robustness, especially in the presence of occlusions or reflective surfaces.

Datasets

The paper utilizes synthetic data for training and validation. Specifically, they generated 50,000 synthetic images for training and 10,000 for validation. These synthetic images were created using Blender, incorporating domain randomization and data augmentation techniques (flipping, brightness adjustment, Gaussian noise) to enhance realism and robustness.

Key Insights and Ideas

  • Ray Casting for Informative Views: The core innovation lies in using ray casting not just for rendering, but for actively planning the most informative next camera viewpoint. This moves beyond simple trajectory following to a more intelligent exploration strategy.
  • Leveraging Confidence Scores: The method's ability to fuse pose estimates from multiple views is significantly enhanced by using semantic keypoint confidence scores. This allows the system to effectively filter out less reliable estimates and focus on more accurate ones.
  • RGB-Only Robustness: The paper demonstrates that high-quality pose estimation for challenging reflective metal parts is achievable without depth sensors, making the approach more accessible and cost-effective for industrial applications.
  • Synthetic Data for Annotation Challenges: The use of synthetic data is a practical solution to the difficulty and cost of annotating keypoints for reflective metal parts in real-world scenarios, while still achieving high accuracy.

Key Results

  • Single-View Accuracy: The method achieved an angle error of 4.09° and a translation error of 4 mm in single-view experiments, resulting in an accuracy of 91.17%.
  • Multi-View Accuracy: For multi-view pose estimation, the accuracy reached 82%.
  • Grasping Success: In grasping experiments using a UR5e robot arm and an Intel Realsense camera, the method achieved a 92% success rate in grasping metal parts.
  • Qualitative Improvements: Figures and experimental results show that the multi-view approach, particularly with the next-best-view planning, significantly improves the number of detectable objects and the overall robustness of pose estimation compared to single-view methods or fixed trajectory approaches.

Limitations and Future Work

  • Reliance on CAD Models: The method requires known CAD models of the metal parts for generating synthetic data and for pose refinement. This limits its applicability to objects for which CAD models are available.
  • Synthetic Data Generation: While effective, the reliance on synthetic data might still have a sim-to-real gap, although domain randomization and augmentation are employed to mitigate this.
  • Future Work: The authors suggest that their method is particularly suitable for metal parts and could be extended to other challenging object categories. Further exploration of more sophisticated next-best-view planning strategies and integration with more advanced object detection and keypoint estimation networks could also be beneficial.

Summary updated

Abstract

Pose estimation of metal parts plays a vital role in industrial grasping areas. It is challenging to obtain complete point clouds of metal parts because of their reflective properties. This study introduces an approach for recovering the 6D pose of CAD-known metal parts from images captured by a single RGB camera. The proposed strategy only requires RGB images without depth information. The core idea of the proposed method is to use multiple views to estimate the metal parts' pose. First, the pose of metal parts is estimated in the first view. Second, ray casting is employed to simulate additional views with the corresponding status of the metal parts, enabling the calculation of the camera's next best viewpoint. The camera, mounted on a robotic arm, is then moved to this calculated position. Third, this study integrates the known camera transformations with the poses estimated from different viewpoints to refine the final scene. The results of this work demonstrate that the proposed method effectively estimates the pose of shiny metal parts.
BibTeX
@inproceedings{chen2024multi,
  author = {Chen Chen and Xin Jiang},
  title = {{Multi-View Metal Parts Pose Estimation Based on a Single Camera}},
  booktitle = {Sensors},
  year = {2024},
  doi = {10.3390/s24113408},
}
Figure