Introduction
Estimating the 6D pose of an object—comprising its 3D rotation and 3D translation—is a fundamental task for applications in robotics, autonomous navigation, and augmented reality. Despite significant progress in deep learning-based computer vision, two primary challenges remain: heavy occlusions in cluttered environments and the inherent ambiguity of symmetric objects.
In cluttered scenes, objects are often partially hidden from a single viewpoint, leading to unreliable pose estimates. While multi-view systems can mitigate this by providing additional perspectives, they often struggle with computational efficiency or fail to resolve the mathematical ambiguities presented by symmetric objects (e.g., bowls, bottles, or cylinders). When a model encounters a symmetric object, standard training objectives often force the network to "average" possible orientations, resulting in "blurry" keypoint predictions and inaccurate final poses.
To address these issues, we present SyMFM6D (Symmetry-aware Multi-directional Fusion for Multi-view 6D pose estimation), a framework designed to fuse RGB-D data across multiple views while explicitly accounting for object symmetries during training.
Methodology
The SyMFM6D architecture is built upon three core components: a multi-directional fusion module, a symmetry-aware keypoint detection head, and a global least-squares fitting stage.
Multi-directional Fusion
Existing methods often utilize bi-directional fusion to combine 2D image features with 3D point cloud data. SyMFM6D extends this concept to an arbitrary number of RGB-D views. The system employs a Convolutional Neural Network (CNN) to extract 2D image features and a RandLA-Net to process 3D point cloud features. By passing information back and forth between 2D pixels and 3D points across all available views, the network constructs a rich, joint representation of the entire scene, ensuring that features from one viewpoint can inform and refine features in another.
Symmetry-aware Keypoint Detection
The core innovation of SyMFM6D lies in its training procedure. Rather than forcing the network to predict a single "ground truth" set of keypoints for a symmetric object—which is mathematically ambiguous—the authors precompute the set of all valid rotational symmetries for each object.
During training, a novel loss function calculates the error relative to the closest symmetric transformation. This approach allows the network to converge on a valid, physically consistent pose without the gradient interference caused by equivalent orientations. This "symmetry-awareness" ensures that the predicted 3D keypoints remain sharp and geometrically accurate.
Efficient Pose Fitting
Following keypoint prediction and semantic labeling, the system employs a fast least-squares fitting algorithm. Unlike traditional methods that may process objects sequentially, SyMFM6D predicts keypoints for the entire scene simultaneously. This allows for the rapid calculation of 6D poses for all detected objects in a single pass, significantly reducing the computational overhead in multi-object environments.
Experimental Evaluation
The framework was evaluated against several rigorous benchmarks to test its limits in both single-view and multi-view scenarios:
- YCB-Video: The industry-standard dataset for 6D pose estimation.
- MV-YCB (FixCam & WiggleCam): Multi-view datasets featuring non-symmetric objects. The "WiggleCam" variant specifically tests robustness against noisy camera calibrations and imprecise poses.
- MV-YCB SymMovCam: A new photorealistic synthetic dataset introduced by the authors. It contains 8–16 objects per scene, including highly symmetric ones, viewed from four distinct quadrants to maximize occlusion and symmetry challenges.
Results and Key Insights
SyMFM6D demonstrates state-of-the-art performance across all tested benchmarks:
- State-of-the-Art Accuracy: On the YCB-Video dataset (single-view), the model achieved an ADD-S AUC of 96.8%, surpassing previous leaders like FFB6D (96.6%). On the challenging SymMovCam dataset, SyMFM6D achieved 94.2% ADD-S AUC, vastly outperforming the next best baseline, which reached only 80.6%.
- The Synergy of Symmetry: A notable finding was that the symmetry-aware loss improved accuracy for non-symmetric objects as well. This suggests that handling symmetries correctly encourages the network to learn a more consistent and robust geometric representation of the entire scene.
- Robustness to Calibration Errors: The multi-view fusion architecture acts as a natural stabilizer. In the "WiggleCam" tests, the network successfully compensated for imprecise camera information by aggregating features from multiple perspectives, maintaining high accuracy where single-view or poorly fused models failed.
- Real-time Potential: The system is highly efficient, processing a single RGB-D image in approximately 46ms on an NVIDIA V100 GPU. The pose fitting stage adds only 14ms per object, making the pipeline suitable for near-real-time robotics applications.
Conclusion and Future Work
SyMFM6D provides a robust solution for multi-view 6D pose estimation by combining multi-directional feature fusion with a symmetry-aware training objective. While the current implementation focuses on rotational symmetries, future work will explore reflectional symmetries, which cannot be expressed as simple Euclidean transformations. Additionally, the authors aim to extend the framework to "category-level" estimation, allowing the model to generalize to unseen objects within a known category without requiring specific 3D CAD models for every instance.
Summary updated