SyMFM6D: Symmetry-Aware Multi-Directional Fusion for Multi-View 6D Object Pose Estimation

Fabian Duffhauss, Sebastian Koch, Hanna Ziesche, Ngo Anh Vien, Gerhard Neumann

Introduction

Estimating the 6D pose of an object—comprising its 3D rotation and 3D translation—is a fundamental task for applications in robotics, autonomous navigation, and augmented reality. Despite significant progress in deep learning-based computer vision, two primary challenges remain: heavy occlusions in cluttered environments and the inherent ambiguity of symmetric objects.

In cluttered scenes, objects are often partially hidden from a single viewpoint, leading to unreliable pose estimates. While multi-view systems can mitigate this by providing additional perspectives, they often struggle with computational efficiency or fail to resolve the mathematical ambiguities presented by symmetric objects (e.g., bowls, bottles, or cylinders). When a model encounters a symmetric object, standard training objectives often force the network to "average" possible orientations, resulting in "blurry" keypoint predictions and inaccurate final poses.

To address these issues, we present SyMFM6D (Symmetry-aware Multi-directional Fusion for Multi-view 6D pose estimation), a framework designed to fuse RGB-D data across multiple views while explicitly accounting for object symmetries during training.

Methodology

The SyMFM6D architecture is built upon three core components: a multi-directional fusion module, a symmetry-aware keypoint detection head, and a global least-squares fitting stage.

Multi-directional Fusion

Existing methods often utilize bi-directional fusion to combine 2D image features with 3D point cloud data. SyMFM6D extends this concept to an arbitrary number of RGB-D views. The system employs a Convolutional Neural Network (CNN) to extract 2D image features and a RandLA-Net to process 3D point cloud features. By passing information back and forth between 2D pixels and 3D points across all available views, the network constructs a rich, joint representation of the entire scene, ensuring that features from one viewpoint can inform and refine features in another.

Symmetry-aware Keypoint Detection

The core innovation of SyMFM6D lies in its training procedure. Rather than forcing the network to predict a single "ground truth" set of keypoints for a symmetric object—which is mathematically ambiguous—the authors precompute the set of all valid rotational symmetries for each object.

During training, a novel loss function calculates the error relative to the closest symmetric transformation. This approach allows the network to converge on a valid, physically consistent pose without the gradient interference caused by equivalent orientations. This "symmetry-awareness" ensures that the predicted 3D keypoints remain sharp and geometrically accurate.

Efficient Pose Fitting

Following keypoint prediction and semantic labeling, the system employs a fast least-squares fitting algorithm. Unlike traditional methods that may process objects sequentially, SyMFM6D predicts keypoints for the entire scene simultaneously. This allows for the rapid calculation of 6D poses for all detected objects in a single pass, significantly reducing the computational overhead in multi-object environments.

Experimental Evaluation

The framework was evaluated against several rigorous benchmarks to test its limits in both single-view and multi-view scenarios:

  1. YCB-Video: The industry-standard dataset for 6D pose estimation.
  2. MV-YCB (FixCam & WiggleCam): Multi-view datasets featuring non-symmetric objects. The "WiggleCam" variant specifically tests robustness against noisy camera calibrations and imprecise poses.
  3. MV-YCB SymMovCam: A new photorealistic synthetic dataset introduced by the authors. It contains 8–16 objects per scene, including highly symmetric ones, viewed from four distinct quadrants to maximize occlusion and symmetry challenges.

Results and Key Insights

SyMFM6D demonstrates state-of-the-art performance across all tested benchmarks:

  • State-of-the-Art Accuracy: On the YCB-Video dataset (single-view), the model achieved an ADD-S AUC of 96.8%, surpassing previous leaders like FFB6D (96.6%). On the challenging SymMovCam dataset, SyMFM6D achieved 94.2% ADD-S AUC, vastly outperforming the next best baseline, which reached only 80.6%.
  • The Synergy of Symmetry: A notable finding was that the symmetry-aware loss improved accuracy for non-symmetric objects as well. This suggests that handling symmetries correctly encourages the network to learn a more consistent and robust geometric representation of the entire scene.
  • Robustness to Calibration Errors: The multi-view fusion architecture acts as a natural stabilizer. In the "WiggleCam" tests, the network successfully compensated for imprecise camera information by aggregating features from multiple perspectives, maintaining high accuracy where single-view or poorly fused models failed.
  • Real-time Potential: The system is highly efficient, processing a single RGB-D image in approximately 46ms on an NVIDIA V100 GPU. The pose fitting stage adds only 14ms per object, making the pipeline suitable for near-real-time robotics applications.

Conclusion and Future Work

SyMFM6D provides a robust solution for multi-view 6D pose estimation by combining multi-directional feature fusion with a symmetry-aware training objective. While the current implementation focuses on rotational symmetries, future work will explore reflectional symmetries, which cannot be expressed as simple Euclidean transformations. Additionally, the authors aim to extend the framework to "category-level" estimation, allowing the model to generalize to unseen objects within a known category without requiring specific 3D CAD models for every instance.

Summary updated

Abstract

Detecting objects and estimating their 6D poses is essential for automated systems to interact safely with the environment. Most 6D pose estimators, however, rely on a single camera frame and suffer from occlusions and ambiguities due to object symmetries. We overcome this issue by presenting a novel symmetry-aware multi-view 6D pose estimator called SyMFM6D. Our approach efficiently fuses the RGB-D frames from multiple perspectives in a deep multi-directional fusion network and predicts predefined keypoints for all objects in the scene simultaneously. Based on the keypoints and an instance semantic segmentation, we efficiently compute the 6D poses by least-squares fitting. To address the ambiguity issues for symmetric objects, we propose a novel training procedure for symmetry-aware keypoint detection including a new objective function. Our SyMFM6D network significantly outperforms the state-of-the-art in both single-view and multi-view 6D pose estimation. We furthermore show the effectiveness of our symmetry-aware training procedure and demonstrate that our approach is robust towards inaccurate camera calibration and dynamic camera setups.
BibTeX
@article{duffhauss2023symfm,
  author = {Fabian Duffhauss and Sebastian Koch and Hanna Ziesche and Ngo Anh Vien and Gerhard Neumann},
  title = {{SyMFM6D: Symmetry-Aware Multi-Directional Fusion for Multi-View 6D Object Pose Estimation}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2023},
  url = {https://openreview.net/forum?id=wRsMsclIai},
}
Figure