Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains

Bowen Wen, Chaitanya Mitash, Kostas E. BekrisarXiv 2021 · OpenReview

TL;DR

This paper introduces se(3)-TrackNet, a novel deep learning approach for robustly tracking the 6D pose of objects in real-time, even after significant occlusions. By cleverly disentangling feature encoding and leveraging Lie Algebra for 3D orientation, the network achieves state-of-the-art performance by training solely on synthetic data, effectively bridging the sim-to-real gap.

What Problem Does This Paper Address?

Accurate 6D pose tracking (position and orientation in 3D space) is crucial for robots to perform manipulation tasks like grasping and placing objects. While re-estimating an object's pose from scratch for every frame can be fast, it often leads to jerky, incoherent movements. Temporal tracking, which uses information from previous frames, can improve both speed and smoothness. However, existing data-driven tracking methods typically require extensive real-world annotated data, which is expensive and time-consuming to collect. Traditional methods often rely on hand-crafted features and require significant hyperparameter tuning, making them less adaptable to new objects or environments. This paper aims to develop a data-driven tracking system that is robust, fast, and can be trained entirely in simulation, overcoming the need for real-world data.

Methods

The core of the proposed method is se(3)-TrackNet, a neural network designed for robust 6D pose tracking. The key innovations include:

  • Disentangled Feature Encoding: The network processes the previous frame's rendered image and the current real observation through separate feature encoders. This disentanglement helps reduce the "domain shift" between synthetic training data and real-world testing data, making the sim-to-real transfer more effective.
  • Lie Algebra for 3D Orientation: Instead of using traditional representations like Euler angles or quaternions, se(3)-TrackNet utilizes Lie Algebra (specifically, the se(3) manifold) to represent and learn residual pose transformations. This provides a more mathematically sound and continuous representation for optimization.
  • Data-Driven Optimization: The network learns to predict a relative pose transformation ($\Delta T$) between consecutive frames. This transformation is then applied to the previous frame's pose to estimate the current pose. The cost function for this optimization is implicitly learned by the network.
  • Physically Plausible Domain Randomization (PPDR): To generate diverse and realistic synthetic training data, the authors employ PPDR. This method goes beyond simple randomization by simulating physics to ensure objects are placed in plausible configurations, avoiding unrealistic penetrations or floating objects, and then randomizing other visual properties like lighting and textures.

Datasets

  • Training: The network is trained exclusively on synthetic data generated using the PPDR method.
  • Evaluation:
    • YCB-Video Dataset: A standard benchmark for 6D pose tracking, containing RGB-D video sequences of 21 YCB objects.
  • YCBInEOAT Dataset: A newly introduced benchmark by the authors, specifically designed for dynamic object tracking in robotic manipulation scenarios. It includes sequences with various robot end-effectors (vacuum gripper, Robotiq 2F-85, Yale T42 Hand) and different manipulation tasks (pick-and-place, within-hand manipulation, hand-off). This dataset features ground-truth 6D pose annotations for 5 YCB objects.

Key Insights and Ideas

  • Synthetic Data is Sufficient: The most significant insight is that a well-designed network architecture and a robust synthetic data generation pipeline (PPDR) can eliminate the need for real-world training data for 6D pose tracking, a major bottleneck in the field.
  • Disentanglement is Key for Sim-to-Real: Separating the processing of synthetic and real image features is crucial for bridging the domain gap. This allows the network to learn generalizable features from synthetic data that transfer well to real observations.
  • Lie Algebra for Smooth Tracking: Using Lie Algebra for pose representation enables smoother and more accurate learning of incremental pose changes, leading to better long-term tracking.
  • PPDR for Realistic Simulation: The PPDR approach generates more physically plausible synthetic scenes, which likely contributes to the network's improved generalization to real-world scenarios compared to simpler domain randomization techniques.

Key Results

  • State-of-the-Art Accuracy: se(3)-TrackNet achieves consistently robust estimates and outperforms existing methods on the YCB-Video benchmark, even those trained with real data. It achieves high AUC scores for both ADD and ADD-S metrics.
  • Real-Time Performance: The approach runs at a remarkable 90.9 Hz, making it suitable for real-time robotic applications.
  • Robustness to Occlusion: Qualitative results demonstrate the network's ability to recover from complete occlusions without re-initialization.
  • Effective on New Benchmark: The proposed YCBInEOAT dataset presents challenging scenarios (abrupt motions, slippage), and se(3)-TrackNet demonstrates robust long-term tracking performance.
  • Ablation Study Highlights: The ablation study confirms the importance of the physics simulation in PPDR, the depth modality, and the disentangled encoder design for achieving high performance.

Limitations and Future Work

  • Initial Pose Dependency: While the paper focuses on tracking, the initial pose is still provided by a separate single-image pose estimation technique. The robustness to the quality of this initial pose is not extensively explored.
  • Object-Specific Training: The current approach is trained for specific objects. Future work could explore generalization to unseen objects or categories.
  • Domain Gap for Depth: While the network architecture helps, the paper mentions specific pre-processing steps for depth data (Gaussian noise augmentation for synthetic, bilateral filtering for real) to bridge the domain gap. Further research into more advanced domain adaptation techniques for depth could be beneficial.
  • Generalization to More Complex Scenarios: While the YCBInEOAT dataset introduces more dynamic scenarios, further testing on even more complex and cluttered environments with highly deformable objects could be explored.

Summary updated

Abstract

Tracking the 6D pose of objects in video sequences is important for robot manipulation. This work presents se(3)-TrackNet, a data-driven optimization approach for long term, 6D pose tracking. It aims to identify the optimal relative pose given the current RGB-D observation and a synthetic image conditioned on the previous best estimate and the object's model. The key contribution in this context is a novel neural network architecture, which appropriately disentangles the feature encoding to help reduce domain shift, and an effective 3D orientation representation via Lie Algebra. Consequently, even when the network is trained solely with synthetic data can work effectively over real images. Comprehensive experiments over multiple benchmarks show se(3)-TrackNet achieves consistently robust estimates and outperforms alternatives, even though they have been trained with real images. The approach runs in real time at 90.9Hz. Code, data and supplementary video for this project are available at https://github.com/wenbowen123/iros20-6d-pose-tracking
BibTeX
@misc{wen2021data,
  author = {Bowen Wen and Chaitanya Mitash and Kostas E. Bekris},
  title = {{Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains}},
  year = {2021},
  eprint = {2105.14391},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.14391},
}
Figure