Abstract
Robotic manipulation of unknown objects is an important field of research. Practical applications occur in many real-world settings where robots need to interact with an unknown environment. We tackle the problem of reactive grasping by proposing a method for unknown object tracking, grasp point sampling and dynamic trajectory planning. Our object tracking method combines Siamese Networks with an Iterative Closest Point approach for pointcloud registration into a method for 6-DoF unknown object tracking. The method does not require further training and is robust to noise and occlusion. We propose a robotic manipulation system, which is able to grasp a wide variety of formerly unseen objects and is robust against object perturbations and inferior grasping points.
Introduction
In traditional robotic manipulation, systems often rely on precise 3D models or extensive prior training on specific object classes. Such systems frequently fail when an object is moved unexpectedly or partially obscured by the robot's own gripper. This research addresses the challenge of reactive grasping of unknown objects, aiming for a system that can track arbitrary objects in 6 degrees of freedom (6-DoF) without prior models, plan grasps on moving targets, and adjust trajectories in real-time to handle physical perturbations.
Methodology
The proposed system architecture is divided into a perception pipeline and a control pipeline.
Perception: THOR + G-ICP
The perception pipeline leverages SiamMask to track objects in 2D and generate segmentation masks. To mitigate "tracking drift," the system incorporates THOR (Tracking Holistic Object Representations), which maintains a memory of long-term and short-term visual templates. Once segmented, depth data is converted into a point cloud, and Generalized-ICP (G-ICP) aligns a template point cloud to the live observation to determine the 6-DoF pose.
Control: Reactive Grasping & Backstepping
The control pipeline utilizes a Grasp Configuration Observer to optimize multiple grasp candidates simultaneously using an Inverse Kinematics (IK) solver. For trajectory planning, a backstepping technique is employed. If the robot's path is obstructed by a dynamic obstacle, it "steps back" toward its home configuration until a clear path to the moving target is re-established.
Experimental Results
The system was evaluated in both simulation and real-world environments using a Franka Emika Panda robot. Testing involved various everyday items such as a yellow cuboid, a Pringles can, and a measuring tape.
Performance and Robustness
The THOR + G-ICP method demonstrated significant stability under occlusion. In simulations, it maintained near-zero error while standard particle-filter-based trackers failed when objects were partially blocked. Real-world experiments confirmed the system's reactivity; the robot successfully grasped a Pringles can even as it was being moved by a human. Furthermore, the system handled dynamic handovers between robots, where high occlusion typically causes baseline methods to lose tracking.
Discussion and Conclusion
A key insight of this work is the decoupling of perception from grasping, allowing for real-time complex scene queries. The use of THOR templates makes the system robust to rotations and occlusions, even when more than 50% of the object is hidden.
While the tracking is model-free, the current grasp observer assumes objects can be approximated as cuboids for gripper alignment. Future work aims to integrate advanced deep-learning-based grasp point detection, such as GraspNet, to handle more complex, non-geometric shapes and reduce reliance on background subtraction for initialization.
Summary updated