SynPo-Net - Accurate and Fast CNN-Based 6DoF Object Pose Estimation Using Synthetic Training

Yongzhi Su, Jason R. Rambach, Alain Pagani, Didier StrickerSensors 2021 · OpenReview

TL;DR

This paper introduces SynPo-Net, a novel Convolutional Neural Network (CNN) architecture for estimating the 6 Degrees of Freedom (6DoF) pose of objects from single RGB images. By training exclusively on synthetic data and employing a unique domain adaptation technique, SynPo-Net achieves state-of-the-art accuracy and speed, overcoming the limitations of real-world data acquisition for training.

What Problem Does This Paper Address?

Accurately estimating and tracking the 6DoF pose (position and orientation) of objects in images is crucial for applications like robotics and augmented reality. While deep learning methods have shown promise, training these models typically requires vast amounts of real-world data with precise ground truth pose annotations. Acquiring this data is a significant bottleneck due to the difficulty, cost, and limited coverage of possible poses and scenarios. Relying solely on real data also hinders generalization to new environments. This paper tackles the challenge of training robust 6DoF pose estimation models without relying on real-world training data, by leveraging synthetic data and addressing the domain gap between synthetic and real images.

Methods

The core of the proposed solution is SynPo-Net, a CNN architecture specifically designed for pose regression. Key modifications to standard CNN architectures include:

  • Replacing Pooling Layers: Max pooling layers, common in classification tasks, are replaced with convolutional layers. This is because pooling layers reduce spatial resolution and sensitivity to object position, which is detrimental for precise pose regression.
  • Lie Algebra for Rotation Representation: Instead of quaternions or Euler angles, the authors use lie algebra (so(3)) to represent rotations. This offers a compact, unambiguous, and continuous representation that avoids issues like gimbal lock and the need for normalization during optimization, simplifying the regression task.
  • Domain Adaptation via Pencil Filter: To bridge the gap between synthetic and real images, the authors employ a novel domain adaptation technique. Both synthetic and real images are transformed into an intermediate "pencil filter" domain. This process enhances edge information and reduces the influence of volatile color and illumination variations, making the domains more similar and improving the network's ability to generalize.
  • Dynamic Data Augmentation: Augmentations like Gaussian noise, contrast/brightness adjustments, and motion blur are applied dynamically during training, rather than being pre-applied to the dataset. This increases data diversity and improves generalization.

Datasets

  • Training: Synthetic images were generated using 3D models of objects. The backgrounds for these synthetic images were sourced from the PASCAL VOC and IKEA datasets.
  • Evaluation:
    • LINEMOD dataset: A widely used benchmark for object pose estimation.
  • TUD-L dataset: Specifically chosen for its challenging lighting variations.

Key Insights and Ideas

  • Synthetic Data is Sufficient (with the right approach): The paper strongly advocates for the viability of training 6DoF pose estimation models exclusively on synthetic data, provided that the domain gap is effectively addressed.
  • Architecture Matters for Regression: Standard classification CNN architectures are not optimal for pose regression. Modifications like removing pooling layers and using appropriate rotation representations are crucial for improving accuracy.
  • Pencil Filter as a Domain Bridge: Transforming images into a pencil filter domain proves to be an effective strategy for domain adaptation, increasing similarity between synthetic and real images and enabling the use of single-channel input.
  • Lie Algebra Simplifies Rotation Regression: Using lie algebra for rotation representation simplifies the optimization process and avoids common pitfalls associated with other rotation representations.

Key Results

  • State-of-the-Art Accuracy: SynPo-Net significantly outperforms existing methods on the LINEMOD dataset, even surpassing approaches trained with real images, when trained solely on synthetic data. For instance, on the LINEMOD dataset, their mean ADD 10% score is 2.42, which is considerably better than many other methods.
  • High Speed: The proposed network architecture is computationally efficient, achieving a frame processing rate of 65 fps on an RTX 2080 Ti GPU, making it suitable for real-time applications.
  • Domain Adaptation Effectiveness: The pencil filter domain adaptation technique demonstrably reduces the difference in feature map activations between synthetic and real images, leading to improved performance.
  • Ablation Study Validation: An ablation study confirmed that each proposed modification (pencil filter, lie algebra, architectural changes) contributes positively to the overall accuracy.

Limitations and Future Work

  • Limitations:
    • The paper notes that SynPo-Net performs weakly on the "camera" object in the LINEMOD dataset, attributing this to the significant difference between real and synthetic images for this specific object, where domain adaptation is less successful.
  • While the paper focuses on single-object pose estimation, the authors mention that some methods struggle with symmetric objects.
  • Future Work:
    • The authors plan to extend their work to multi-object pose estimation by using SynPo-Net as a backbone network.

Summary updated

Abstract

Estimation and tracking of 6DoF poses of objects in images is a challenging problem of great importance for robotic interaction and augmented reality. Recent approaches applying deep neural networks for pose estimation have shown encouraging results. However, most of them rely on training with real images of objects with severe limitations concerning ground truth pose acquisition, full coverage of possible poses, and training dataset scaling and generalization capability. This paper presents a novel approach using a Convolutional Neural Network (CNN) trained exclusively on single-channel Synthetic images of objects to regress 6DoF object Poses directly (SynPo-Net). The proposed SynPo-Net is a network architecture specifically designed for pose regression and a proposed domain adaptation scheme transforming real and synthetic images into an intermediate domain that is better fit for establishing correspondences. The extensive evaluation shows that our approach significantly outperforms the state-of-the-art using synthetic training in terms of both accuracy and speed. Our system can be used to estimate the 6DoF pose from a single frame, or be integrated into a tracking system to provide the initial pose.
BibTeX
@inproceedings{su2021synpo,
  author = {Yongzhi Su and Jason R. Rambach and Alain Pagani and Didier Stricker},
  title = {{SynPo-Net - Accurate and Fast CNN-Based 6DoF Object Pose Estimation Using Synthetic Training}},
  booktitle = {Sensors},
  year = {2021},
  url = {https://openreview.net/forum?id=DAtUL1dF_OE},
}
Figure