Introduction
Robotic clutter removal—the task of clearing a workspace of various objects—remains a significant challenge in robotic manipulation. While grasping isolated objects is a well-studied problem, real-world scenarios often involve tightly packed items where direct grasping is impossible. In such cases, a robot must perform non-prehensile actions, such as pushing, to rearrange objects and create sufficient space for a successful grasp.
Predicting the outcome of a push in a cluttered environment is complex due to unknown physical properties like mass, friction, and center of gravity. Traditional physics-based simulators require these parameters to be known a priori. Conversely, many deep learning approaches attempt to predict future states as raw pixels, often resulting in blurry "ghost" images that are unsuitable for downstream tasks. The Deep Interaction Prediction Network (DIPN) addresses these issues by learning to predict object-level transformations directly from visual input.
Methodology
The proposed system utilizes a modular pipeline that integrates perception, interaction prediction, and grasp evaluation.
Perception and Segmentation
The process begins with scene segmentation using Mask R-CNN. By identifying individual objects as distinct entities rather than a single image blob, the system can reason about the specific movements of each item in the workspace.
The DIPN Architecture
DIPN serves as the "imagination" engine of the robot. Given an initial state observation and a candidate push action, the network predicts a 2D transformation (translation and rotation) for every segmented object. The architecture is composed of two primary modules:
- Direct Transformation Module: Predicts the displacement of the object directly contacted by the robot's end-effector.
- Interactive Transformation Module: Models the "chain reactions" or secondary collisions where the pushed object impacts adjacent items.
By predicting $(x, y, \theta)$ transformations for masks rather than generating new pixels, the resulting "imagined" scenes remain sharp and physically plausible.
Decision Making and Grasping
A separate Grasp Network (GN) evaluates the "graspability" of a scene. During operation, the robot samples multiple potential push actions. For each sample, DIPN generates a synthetic image of the predicted post-push state. The GN then scores these imagined states. If a push is predicted to significantly improve the likelihood of a successful grasp, the robot executes that push; otherwise, it proceeds with a direct grasp attempt.
Experiments and Results
The system was trained in the CoppeliaSim environment and evaluated on a UR-5e manipulator with a Robotiq 2-finger gripper. DIPN demonstrated high sample efficiency, requiring only 100 unlabeled images for segmentation and 500–1500 simulated actions for core network training.
Performance Comparison
DIPN was benchmarked against the Visual Pushing and Grasping (VPG) baseline. Key results include:
- Completion Rate: DIPN achieved a 34% higher completion rate in dense clutter scenarios.
- Grasp Success: The system showed a 20.9% improvement in the success rate of individual grasp attempts.
- Action Efficiency: DIPN cleared workspaces 30.4% more efficiently, requiring fewer total actions than the baseline.
- Prediction Accuracy: The pose prediction error remained below 10% (measured by Intersection-over-Union), indicating that the network's "imagined" outcomes closely matched real-world physics.
Conclusion
DIPN provides a robust framework for predicting complex object interactions in cluttered environments. By focusing on object transformations rather than pixel generation, it enables effective "look-ahead" planning for robotic manipulation. Future work aims to extend this model to 6D poses to handle vertical stacking and more complex object geometries, such as deformable or transparent items.
Summary updated