Introduction
Estimating the 6D pose (3D position and orientation) of objects from RGB-D images is a fundamental task in robotics and augmented reality. However, real-world conditions—such as uniform metallic colors, reflective surfaces, and inter-object occlusions—often degrade the performance of standard algorithms. Traditional methods frequently struggle with effective modality integration or rely on slow iterative clustering (like MeanShift) to locate object keypoints. This paper introduces a novel framework that combines a Deep Fusion Transformer (DFTr) for robust RGB-D data integration with a weighted vector-wise voting algorithm for efficient keypoint localization, achieving state-of-the-art accuracy while running up to 2.7x faster than previous leading methods.
Methodology
The proposed framework addresses the limitations of existing pose estimation techniques through two primary innovations integrated into a unified pipeline.
Deep Fusion Transformer (DFTr)
Standard approaches to RGB-D fusion, such as simple concatenation, often fail to capture the complex inter-modality relationships. The DFTr block utilizes a bidirectional cross-attention mechanism to model global semantic correlations. By treating features from both RGB and depth branches as sequences, the network can leverage geometric cues to resolve visual ambiguities (e.g., reflections) and vice versa. This global modeling approach ensures the representation remains robust even in the presence of local sensor noise.
Weighted Vector-Wise Voting
To overcome the speed-accuracy trade-off of iterative clustering methods, the authors propose a weighted vector-wise voting algorithm. The network predicts a unit vector field pointing toward object keypoints. Keypoint localization is then formulated as a global optimization problem, which is solved directly using a weighted least squares approach. This non-iterative method significantly reduces inference time while maintaining high precision.
Experimental Results
The model was evaluated on several benchmarks, including MP6D, YCB-Video, LineMOD, and Occlusion LineMOD.
State-of-the-Art Accuracy and Efficiency
On the MP6D dataset, which specifically targets metallic parts with high reflectivity, the proposed method outperformed the previous state-of-the-art (FFB6D) by 6.72% in the ADD-S metric and 19.69% in the VSD metric. Furthermore, the framework achieves near real-time performance with an inference time of approximately 48ms per frame, making it up to 2.7x faster than FFB6D.
Robustness to Occlusion
The framework demonstrates exceptional resilience to occlusion. Experimental results show that the method maintains high accuracy even as the percentage of the object's surface that is invisible increases, significantly outperforming iterative baselines like DenseFusion.
Conclusion
The integration of global semantic modeling via Transformers and a closed-form linear solver for keypoint voting allows this framework to achieve superior performance in challenging environments. By effectively handling reflections and occlusions while maintaining high computational efficiency, this work provides a robust solution for real-time 6D object pose estimation in unstructured settings.
Summary updated