Deep Fusion Transformer Network with Weighted Vector-Wise Keypoints Voting for Robust 6D Object Pose Estimation

Jun Zhou, Kai Chen, Linlin Xu, Qi Dou, Jing Qin

Introduction

Estimating the 6D pose (3D position and orientation) of objects from RGB-D images is a fundamental task in robotics and augmented reality. However, real-world conditions—such as uniform metallic colors, reflective surfaces, and inter-object occlusions—often degrade the performance of standard algorithms. Traditional methods frequently struggle with effective modality integration or rely on slow iterative clustering (like MeanShift) to locate object keypoints. This paper introduces a novel framework that combines a Deep Fusion Transformer (DFTr) for robust RGB-D data integration with a weighted vector-wise voting algorithm for efficient keypoint localization, achieving state-of-the-art accuracy while running up to 2.7x faster than previous leading methods.

Methodology

The proposed framework addresses the limitations of existing pose estimation techniques through two primary innovations integrated into a unified pipeline.

Deep Fusion Transformer (DFTr)

Standard approaches to RGB-D fusion, such as simple concatenation, often fail to capture the complex inter-modality relationships. The DFTr block utilizes a bidirectional cross-attention mechanism to model global semantic correlations. By treating features from both RGB and depth branches as sequences, the network can leverage geometric cues to resolve visual ambiguities (e.g., reflections) and vice versa. This global modeling approach ensures the representation remains robust even in the presence of local sensor noise.

Weighted Vector-Wise Voting

To overcome the speed-accuracy trade-off of iterative clustering methods, the authors propose a weighted vector-wise voting algorithm. The network predicts a unit vector field pointing toward object keypoints. Keypoint localization is then formulated as a global optimization problem, which is solved directly using a weighted least squares approach. This non-iterative method significantly reduces inference time while maintaining high precision.

The Deep Fusion Transformer Network architecture, featuring dual-branch feature extraction, DFTr blocks for cross-modal fusion, and a weighted vector-wise voting module for pose estimation.

Experimental Results

The model was evaluated on several benchmarks, including MP6D, YCB-Video, LineMOD, and Occlusion LineMOD.

State-of-the-Art Accuracy and Efficiency

On the MP6D dataset, which specifically targets metallic parts with high reflectivity, the proposed method outperformed the previous state-of-the-art (FFB6D) by 6.72% in the ADD-S metric and 19.69% in the VSD metric. Furthermore, the framework achieves near real-time performance with an inference time of approximately 48ms per frame, making it up to 2.7x faster than FFB6D.

Robustness to Occlusion

The framework demonstrates exceptional resilience to occlusion. Experimental results show that the method maintains high accuracy even as the percentage of the object's surface that is invisible increases, significantly outperforming iterative baselines like DenseFusion.

Accuracy (ADD-S < 2cm) vs. invisible surface percentage threshold on the YCB-Video dataset, demonstrating the robustness of the proposed method compared to existing baselines.

Conclusion

The integration of global semantic modeling via Transformers and a closed-form linear solver for keypoint voting allows this framework to achieve superior performance in challenging environments. By effectively handling reflections and occlusions while maintaining high computational efficiency, this work provides a robust solution for real-time 6D object pose estimation in unstructured settings.

Summary updated

Abstract

One critical challenge in 6D object pose estimation from a single RGBD image is efficient integration of two different modalities, i.e., color and depth. In this work, we tackle this problem by a novel Deep Fusion Transformer (DFTr) block that can aggregate cross-modality features for improving pose estimation. Unlike existing fusion methods, the proposed DFTr can better model cross-modality semantic correlation by leveraging their semantic similarity, such that globally enhanced features from different modalities can be better integrated for improved information extraction. Moreover, to further improve robustness and efficiency, we introduce a novel weighted vector-wise voting algorithm that employs a non-iterative global optimization strategy for precise 3D keypoint localization while achieving near real-time inference. Extensive experiments show the effectiveness and strong generalization capability of our proposed 3D keypoint voting algorithm. Results on four widely used benchmarks also demonstrate that our method outperforms the state-of-the-art methods by large margins. Code is available at https://github.com/junzastar/DFTr_Voting.
BibTeX
@inproceedings{zhou2023deep,
  author = {Jun Zhou and Kai Chen and Linlin Xu and Qi Dou and Jing Qin},
  title = {{Deep Fusion Transformer Network with Weighted Vector-Wise Keypoints Voting for Robust 6D Object Pose Estimation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure