Introduction
Estimating the 6D pose (3D translation and 3D rotation) of an object from a single RGB image is a foundational task for robotics, augmented reality, and autonomous navigation. Modern deep learning approaches excel at this, but they are notoriously data-hungry. Because annotating 6D poses in real-world 3D space is prohibitively expensive and time-consuming, researchers often train models on virtually generated synthetic data.
However, this introduces a severe "domain gap." Models trained on synthetic images struggle to generalize to real-world images due to differences in lighting, texture, and sensor noise. Existing workarounds either require complex, computationally expensive physically-based rendering, or they still rely on some amount of unlabeled real-world data to bridge the gap. This paper introduces SD-Pose (Semantic Decomposition Pose estimation), a framework that achieves state-of-the-art performance using strictly synthetic training data.
Bridging the Domain Gap via Semantic Decomposition
The core innovation of SD-Pose is the decomposition of an image into multiple semantic representations. This strategy addresses the fundamental trade-off between transferability and robustness.
Highly distilled representations, such as silhouettes, sketches, or edge maps, strip away domain-specific textures and lighting conditions. Because these features look nearly identical in both synthetic and real domains, they possess high transferability. However, they lack the rich details needed to resolve complex poses, such as distinguishing the front from the back of a symmetrically shaped object. Conversely, raw RGB data is robust and detail-rich but highly sensitive to the domain shift between synthetic and real environments.
By extracting a hierarchy of representations—ranging from the most distilled (sketches) to the least distilled (RGB)—SD-Pose ensures that the model can rely on domain-invariant features when textures are misleading, while still utilizing fine details when they are reliable.
The SD-Pose Architecture
SD-Pose is a detection-based framework that processes these multiple representations in parallel. Unlike traditional approaches that map a single RGB input directly to a pose, SD-Pose employs a multi-branch Siamese network architecture.
The pipeline consists of several integrated modules:
- Detection & Zooming: The system first uses a 2D detector (like Mask R-CNN) to locate the object. It then crops and zooms into the object patch, scaling it to a unified resolution. This step is crucial because it ensures that extracted sketches and edges have a consistent style regardless of the object's original scale in the image.
- Cross-semantic Coordinates Net (CCNet): This module processes the different semantic branches (RGB, grayscale, sketch, and edge maps) in parallel to extract features and predict dense 2D-3D coordinates.
- Context-aware Aggregation Net (CANet): This serves as the "brain" of the fusion process. Using an attention mechanism, CANet dynamically scores and fuses features from the different branches based on the specific context of the input image. For instance, if an image is poorly lit, the network learns to prioritize edge or sketch representations over RGB data.
- Coordinates Ensemble Net (CENet): This module takes the coordinate and confidence predictions from all individual branches and the fused CANet branch, ensembling them into a final, highly refined prediction.
- Pose Estimation: The final 6D pose is calculated from these dense coordinates using a standard Perspective-n-Point (PnP) algorithm with RANSAC.
Experimental Results and Performance
The authors evaluated SD-Pose on two standard real-world benchmarks: LineMOD (texture-less objects in cluttered scenes) and Occluded-LineMOD (heavily occluded objects). The model was trained entirely on synthetic data, using both lightweight OpenGL-based rendering and high-quality Blender-based physically rendered data.
On the LineMOD dataset, SD-Pose achieved 56.3% accuracy (ADD metric) when trained on simple OpenGL data, beating the previous synthetic-only state-of-the-art. When trained on higher-quality Blender data, accuracy jumped to 67.3%. On the more challenging Occluded-LineMOD dataset, SD-Pose reached 34.6% accuracy, significantly outperforming previous methods like Self6D (14.6%).
Remarkably, SD-Pose trained only on synthetic data outperformed competing methods that utilize unlabeled real-world RGB-D data during training to bridge the domain gap. This demonstrates that semantic decomposition is a highly effective alternative to domain adaptation techniques that require real-world samples.
Conclusion
SD-Pose offers a robust solution to the synthetic-to-real domain gap in 6D pose estimation. By adaptively fusing multiple levels of semantic representations, the model leverages the strengths of both domain-invariant sketches and detail-rich RGB images. This approach not only sets a new state-of-the-art for synthetic-only training but also suggests a promising direction for other computer vision tasks plagued by domain shifts. Future work will explore applying this multi-level representation strategy to broader tasks beyond pose estimation.
Summary updated