Deep 6-DoF Tracking of Unknown Objects for Reactive Grasping

Marc Tuscher, Julian Hörz, Danny Driess, Marc Toussaint

TL;DR

Researchers have developed a robust system that allows robots to track and grasp objects they have never seen before in real-time. By combining Siamese Networks for visual tracking with Iterative Closest Point (ICP) for 3D alignment, the system can react dynamically to moving objects and physical interference, even when the object is partially hidden.

What Problem Does This Paper Address?

In traditional robotic manipulation, robots usually need a precise 3D model or extensive prior training on specific object classes (like "mugs" or "bottles") to interact with them. If an object is moved by a human or obscured by the robot's own gripper, these systems often fail.

This paper tackles reactive grasping of unknown objects. The goal is a system that can:

  1. Track an arbitrary object in 6 degrees of freedom (position and orientation) without a prior model.
  2. Plan a grasp on that object while it is moving.
  3. Adjust the robot's trajectory in real-time to avoid new obstacles or handle sudden perturbations.

Methods

The authors divide the problem into a perception pipeline and a control pipeline:

1. Perception: THOR + G-ICP

  • SiamMask & THOR: The system uses SiamMask (a Siamese Network) to track the object in 2D and generate a segmentation mask. To prevent "tracking drift," they use THOR (Tracking Holistic Object Representations), which maintains a memory of long-term and short-term visual templates of the object.
  • Generalized-ICP (G-ICP): Once the object is segmented in the RGB image, the corresponding depth data is converted into a point cloud. The system uses G-ICP to align a "template" point cloud (captured at the start) to the current live observation, providing the 6-DoF pose.

2. Control: Reactive Grasping & Backstepping

  • Grasp Configuration Observer: Instead of just picking one point, the system optimizes for $N$ different grasp candidates simultaneously. It uses an Inverse Kinematics (IK) solver that balances reaching the target, aligning the gripper, and avoiding collisions.
  • Backstepping: For trajectory planning, the authors implement a "backstepping" technique. If the robot's path to the object becomes blocked by a dynamic obstacle, the robot "steps back" toward its home configuration until a clear path to the moving target is found again.

Datasets

The paper does not rely on a specific new large-scale dataset for training, as the method is designed to be class-agnostic and model-free.

  • Evaluation: The system was tested in a simulated environment against a standard particle-filter-based tracker from the Point Cloud Library (PCL).
  • Real-world Testing: Experiments were conducted using a Franka Emika Panda robot interacting with everyday items: a yellow cuboid, a Pringles can, a measuring tape, and a lashing strap.

Key Insights and Ideas

  • Decoupling Perception and Grasping: By separating the tracking (where is it?) from the grasp calculation (how do I grab it?), the robot can execute complex queries on the scene in real-time.
  • The Power of Templates: Using THOR allows the robot to "remember" what the object looks like from different angles, making it much more robust to rotations than standard trackers.
  • Handling Occlusion: Because the system segments the object first, it ignores "distractor" points (like the robot's own fingers or a human hand) during the 3D alignment phase. This allows it to maintain a stable lock on the object even when more than 50% of it is hidden.

Key Results

  • Stability under Occlusion: In simulations, the proposed THOR + G-ICP method maintained near-zero error while a standard particle filter tracker lost the object entirely when it was partially blocked.
  • Dynamic Handover: The system successfully tracked an object being handed from one robot to another. The particle filter baseline failed during the handover because the two grippers occluded too much of the object.
  • Real-world Reactivity: The robot successfully grasped a Pringles can even while a human was actively moving it around the table, demonstrating true "reactive" behavior.

Limitations and Future Work

  • Pose Definition: While the tracking is model-free, the specific "Grasp Observer" used in this paper assumes the object can be roughly treated as a cuboid for the purpose of aligning the gripper axes.
  • Initialization: The current system relies on background subtraction to find the object initially, which requires a relatively static background (like a tabletop) at the very start of the process.
  • Future Directions: The authors suggest that integrating more advanced deep-learning-based grasp point detection (like GraspNet) could allow the system to handle even more complex, non-geometric shapes more naturally.

Summary updated

Abstract

Robotic manipulation of unknown objects is an important field of research. Practical applications occur in many real-world settings where robots need to interact with an unknown environment. We tackle the problem of reactive grasping by proposing a method for unknown object tracking, grasp point sampling and dynamic trajectory planning. Our object tracking method combines Siamese Networks with an Iterative Closest Point approach for pointcloud registration into a method for 6-DoF unknown object tracking. The method does not require further training and is robust to noise and occlusion. We propose a robotic manipulation system, which is able to grasp a wide variety of formerly unseen objects and is robust against object perturbations and inferior grasping points.
BibTeX
@inproceedings{tuscher2021deep,
  author = {Marc Tuscher and Julian Hörz and Danny Driess and Marc Toussaint},
  title = {{Deep 6-DoF Tracking of Unknown Objects for Reactive Grasping}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=dXjmhvSeH2},
}

Loading related content...

Figure