Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning

Andy Zeng, Shuran Song, Stefan Welker, Johnny Lee, Alberto Rodriguez, Thomas A. Funkhouser

TL;DR

This paper introduces a self-supervised deep reinforcement learning approach, Visual Pushing for Grasping (VPG), that enables robots to learn complex synergies between pushing and grasping actions. By training end-to-end from visual input, VPG discovers how pushing can clear the way for successful grasps and vice-versa, leading to significantly improved picking performance in cluttered and novel environments.

What Problem Does This Paper Address?

Skilled robotic manipulation often relies on a sophisticated interplay between pushing (non-prehensile) and grasping (prehensile) actions. Pushing can rearrange objects to create space for a gripper, while grasping can precisely position objects for more effective pushes. However, most prior research has studied these actions in isolation. Learning to combine them effectively, especially in unstructured environments with novel objects and significant clutter, has been a major challenge. Existing methods often rely on hand-crafted heuristics or model-based approaches that struggle with real-world variability. This paper aims to bridge this gap by learning these synergistic behaviors directly from experience using deep reinforcement learning, without explicit object models or pre-defined rules.

Methods

The core of the VPG system is a model-free deep reinforcement learning framework based on Q-learning. The authors train two fully convolutional networks (FCNs), one for pushing and one for grasping. These networks take a visual representation of the scene (an RGB-D heightmap) as input and output dense pixel-wise predictions of the expected future reward (Q-values) for executing a specific primitive action (push or grasp) at each location and orientation in the scene.

Key methodological aspects include:

  • Joint Learning: The pushing and grasping networks are trained jointly within a single Q-learning framework. This allows the pushing policy to learn actions that are beneficial for future grasps, and the grasping policy to leverage past pushes.
  • Self-Supervised Rewards: The system learns entirely through trial and error. Rewards are primarily derived from successful grasps. Pushes receive a small intrinsic reward if they cause a detectable change in the environment, but their true value is learned through their impact on subsequent grasp success.
  • Pixel-wise Affordances: The FCNs predict Q-values for every pixel in the heightmap, effectively learning a dense map of "affordances" for both pushing and grasping. This allows for fine-grained action selection.
  • Action Parameterization: Actions are defined as primitive motions (e.g., a 10cm push in one of 16 directions, or a parallel-jaw grasp at a specific location and orientation). To handle orientations, the input heightmap is rotated 16 times.
  • State Representation: The state is represented as an RGB-D heightmap, created by projecting 3D point cloud data onto an orthographic plane. This representation captures both color and height information.

Datasets

The paper does not introduce new datasets. Instead, it uses simulated environments and real-world experiments with collections of toy blocks and other office objects. The training and testing scenarios involve varying degrees of clutter, including random arrangements and manually engineered challenging configurations.

Key Insights and Ideas

  • Synergy is Learned, Not Programmed: The most significant insight is that complex pushing-grasping synergies can be learned from scratch through self-supervised reinforcement learning, without explicit programming of these interactions.
  • Grasping Success as the Primary Driver: The system learns to push effectively because pushing actions that lead to successful grasps are implicitly rewarded. This demonstrates how a downstream task (grasping) can effectively supervise an upstream action (pushing).
  • End-to-End Learning is Crucial: Training perception and control policies end-to-end from visual input, rather than separately, is key to achieving these synergistic behaviors.
  • Pixel-wise Predictions Enable Efficiency: The dense, pixel-wise prediction of Q-values allows for efficient computation and direct mapping of actions to visual input, enabling rapid learning.
  • Generalization to Novel Objects: The model-agnostic nature of the learned policies allows them to generalize to objects not seen during training, provided they fall within a similar shape distribution.

Key Results

  • Improved Grasping Success: VPG significantly outperforms grasping-only baselines and a reactive pushing-and-grasping baseline in terms of completion rates and grasp success rates, especially in challenging cluttered scenarios.
  • Efficient Picking: VPG achieves higher action efficiency, meaning it completes picking tasks more succinctly.
  • Rapid Learning: The system can learn effective pushing and grasping policies in a matter of hours of wall-clock time on a real robot (less than 2000 training transitions).
  • Challenging Scenarios: In real-world experiments with difficult arrangements, VPG achieved a 83.3% completion rate compared to 42.9% for the grasping-only baseline.
  • Generalization: The system demonstrated the ability to plan complex pushing motions to de-clutter scenarios with novel objects.

Limitations and Future Work

  • Limited Primitive Expressiveness: The current motion primitives are relatively simple. Future work could explore more dynamic or complex pushing and grasping motions, as well as parallel or sequential combinations of actions.
  • Object Shape Generalization: While VPG generalizes to novel objects, it struggles with significantly different shapes or reflective objects that lack depth data. Training on a wider variety of shapes and further evaluating generalization is suggested.
  • Beyond Pushing and Grasping: The current work focuses on pushing and grasping. Future research could investigate the application of this deep reinforcement learning framework to other multi-step manipulation tasks like rolling, stacking, or levering.
  • Exploration Strategy: The authors note that the intrinsic reward for pushing does not explicitly consider future grasp benefits, and the synergy is learned indirectly. Exploring more direct ways to supervise pushing actions that benefit grasping could be beneficial.

Summary updated

Abstract

Skilled robotic manipulation benefits from complex synergies between non-prehensile (e.g. pushing) and prehensile (e.g. grasping) actions: pushing can help rearrange cluttered objects to make space for arms and fingers; likewise, grasping can help displace objects to make pushing movements more precise and collision-free. In this work, we demonstrate that it is possible to discover and learn these synergies from scratch through model-free deep reinforcement learning. Our method involves training two fully convolutional networks that map from visual observations to actions: one infers the utility of pushes for a dense pixel-wise sampling of end-effector orientations and locations, while the other does the same for grasping. Both networks are trained jointly in a Q-learning framework and are entirely self-supervised by trial and error, where rewards are provided from successful grasps. In this way, our policy learns pushing motions that enable future grasps, while learning grasps that can leverage past pushes. During picking experiments in both simulation and real-world scenarios, we find that our system quickly learns complex behaviors even amid challenging cases of tightly packed clutter, and achieves better grasping success rates and picking efficiencies than baseline alternatives after a few hours of training. We further demonstrate that our method is capable of generalizing to novel objects. Qualitative results (videos), code, pre-trained models, and simulation environments are available at http://vpg.cs.princeton.edu/
BibTeX
@inproceedings{zeng2018learning,
  author = {Andy Zeng and Shuran Song and Stefan Welker and Johnny Lee and Alberto Rodriguez and Thomas A. Funkhouser},
  title = {{Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=HlIJumC2ht-},
}
Figure