DKPMV: Dense Keypoints Fusion from Multi-View RGB Frames for 6D Pose Estimation of Textureless Objects

Jiahong Chen, Jinghao Wang, Zi Wang, Banglei Guan, Qifeng YuarXiv 2025 · DOI · arXiv

Abstract

Estimating the 6D pose of textureless industrial objects remains a significant challenge for robotic manipulation, particularly when depth data is unreliable. This paper presents DKPMV, a novel pipeline that achieves high-precision 6D pose estimation using only multi-view RGB images. By performing dense keypoint-level fusion and employing a three-stage progressive optimization strategy, DKPMV effectively captures geometric information from shiny or uniform surfaces. Experimental results on the ROBI dataset demonstrate that DKPMV outperforms state-of-the-art RGB-only methods and even surpasses depth-based (RGB-D) approaches in scenarios with noisy depth data.

Introduction

In industrial automation, robotic arms frequently interact with textureless objects—parts made of polished metal or uniform plastic that lack distinct visual patterns. Traditional computer vision algorithms struggle with these objects because they rely on texture for feature matching. While depth sensors (RGB-D) are often used to compensate, they are prone to failure on specular (shiny) surfaces and are often expensive or slow.

Existing multi-view RGB methods typically fuse information at the "pose level" by averaging individual estimates, which ignores the rich geometric redundancy available across views. DKPMV addresses this by shifting the fusion process to the dense keypoint level, providing a more robust foundation for 3D reconstruction and pose alignment.

Methods

The DKPMV pipeline transitions from 2D image coordinates to 3D poses through a series of specialized modules designed to handle the unique constraints of industrial parts.

KeypointNet-SAT with Attention

The process begins with the detection of a dense set of keypoints (typically 256 or 512) on the object in each RGB view. The architecture incorporates an attentional aggregation module that enables keypoints to communicate with neighboring points. This mechanism allows the network to learn spatial constraints and the underlying geometry of the object, resulting in more structured keypoint distributions.

Symmetry-Aware Training (SAT)

Many industrial parts, such as bolts or tubes, exhibit rotational symmetry. To prevent training instability caused by ambiguous poses, the authors utilize a SAT strategy. This approach maps all equivalent symmetric poses to a single "canonical" representation, ensuring the network learns a consistent mapping.

The DKPMV pipeline: from multi-view 2D detection to dense keypoint matching and three-stage pose optimization.

Three-Stage Progressive Optimization

  1. Dense Point Cloud Generation: The system uses RANSAC and SVD-based triangulation to build a 3D point cloud from the 2D keypoint predictions across all views.
  2. Initial Pose Alignment: The reconstructed 3D cloud is aligned with the object's known CAD model using the Umeyama algorithm.
  3. Nonlinear Optimization: The pose is refined by minimizing the global reprojection error across all views, employing a robust loss function to mitigate the impact of outlier keypoints.

Datasets and Evaluation

The researchers evaluated DKPMV using the ROBI dataset, which contains seven highly reflective, textureless industrial objects. Testing was conducted in two environments:

  • Ensenso: High-precision depth data used as a benchmark.
  • RealSense: Low-cost sensor data characterized by significant noise.

The model was trained exclusively on synthetic images, demonstrating a strong ability to generalize from simulation to real-world industrial settings.

Key Results and Insights

DKPMV demonstrates remarkable robustness and accuracy compared to existing baselines. A key finding is that dense redundancy is defensive; the use of hundreds of keypoints allows the system to remain accurate even under partial occlusion or when individual predictions are noisy.

Quantitative comparison of DKPMV against state-of-the-art methods on the ROBI dataset across 4 and 8 views.
  • Superior Accuracy: On the Ensenso test set with 4 views, DKPMV achieved a 93.0% Average Recall (AR) on the ADD metric, outperforming the RGB-D-based MV-3D-KP.
  • Robustness to Noise: In the RealSense environment, where depth data is poor, DKPMV outperformed the next best method by 16.4% in the 4-view setting, proving that sophisticated RGB processing can be more reliable than low-quality depth hardware.
  • Ablation Analysis: Removing the Symmetry-Aware Training (SAT) led to a performance drop of up to 50%, highlighting its necessity for handling industrial parts.

Conclusion and Future Work

DKPMV establishes a new state-of-the-art for multi-view 6D pose estimation of textureless objects. By leveraging dense keypoint fusion and attention mechanisms, it bridges the gap between RGB and RGB-D performance. Future work will focus on modeling the uncertainty of individual keypoint predictions to further enhance fusion resilience and exploring methods to estimate camera extrinsics on the fly for dynamic robotic environments.

Summary updated

BibTeX
@misc{chen2025dkpmv,
  author = {Jiahong Chen and Jinghao Wang and Zi Wang and Ziwen Wang and Banglei Guan and Qifeng Yu},
  title = {{DKPMV: Dense Keypoints Fusion from Multi-View RGB Frames for 6D Pose Estimation of Textureless Objects}},
  year = {2025},
  eprint = {2510.10933},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.10933},
}
Figure