Reinforcement Learning Based Pushing and Grasping Objects from Ungraspable Poses

Hao Zhang, Hongzhuo Liang, Lin Cong, Jianzhi Lyu, Long Zeng, Pingfa Feng, Jianwei Zhang

Introduction

Robotic manipulation of flat objects, such as books, phones, or thin plates, presents a significant challenge for standard parallel-jaw grippers. When these objects lie horizontally on a flat surface, they often enter "ungraspable poses"—states where the object is thinner than the gripper's minimum stroke, making it impossible for the robot to get its fingers underneath the item.

While humans intuitively solve this by sliding an object to the edge of a table or using a wall for leverage, existing robotic solutions are often limited. "Push-to-wall" strategies require specific environmental features and frequently fail when objects have beveled or slanted edges that the gripper cannot catch. This paper introduces a more versatile push-to-edge strategy, utilizing a Deep Reinforcement Learning (DRL) framework to enable robots to manipulate objects into graspable positions regardless of edge geometry or the presence of walls.

The PaG (Push-and-Grasp) Framework

The researchers developed a model-free DRL framework named PaG (Push-and-Grasp). The system is designed to synergize the two distinct actions of pushing and grasping into a cohesive policy.

Visual Representation and Policy Optimization

To handle high-dimensional visual input, the system employs a Variational Autoencoder (VAE) pre-trained on 100,000 simulated images. This compresses RGB camera data into a compact 128-dimensional latent vector, which serves as the input for the policy.

The policy itself is optimized using Proximal Policy Optimization (PPO). A key innovation in this work is the use of a single Actor-Critic network with shared layers for both pushing and grasping. By sharing layers, the model learns the underlying structural similarities between the tasks—specifically the requirement for stable contact—allowing the robot to understand the trade-off between moving an object toward the edge and accidentally pushing it off entirely.

Reward Engineering

The reward function ($R$) is carefully constructed to guide the learning process through three components:

  1. Progress Reward ($R_p$): Encourages movement toward the table's edge.
  2. Grasping Reward ($R_g$): Provides a high incentive for a successful lift.
  3. Common Reward ($R_c$): Based on the distance between the gripper and the object's center of mass to ensure stable contact during the pushing phase.

Sim-to-Real Transfer

To bridge the "reality gap" between the Isaac Gym simulator and a physical UR5 robot, the authors utilized CycleGAN. This generative model translates real-world camera images into the simulated style the network was trained on. This approach allows the policy to be deployed on a real robot without the need for time-consuming manual fine-tuning in the physical environment.

Experimental Evaluation

The framework was trained on a dataset of 80 unique objects across 10 categories (circles, squares, hexagons, etc.) with varying sizes. To test the system's robustness, it was evaluated against:

  • Beveled Objects: Shapes with 30° chamfers that typically defeat push-to-wall methods.
  • Irregular Shapes: Letters and large-scale objects with non-overlapping centers of mass.
  • Real-World Objects: 13 daily items including cereal boxes, books, and calculators.

Key Results and Insights

The PaG framework demonstrated significant improvements over multi-network baselines and traditional strategies:

  • Efficiency: The single-network architecture converged 2.5 times faster than approaches using separate networks for pushing and grasping. This confirms that sharing features for contact-based tasks significantly improves data efficiency.
  • Robustness: The model achieved success rates exceeding 80% on beveled objects and maintained high performance on irregular shapes where the center of mass was off-center.
  • Real-World Superiority: In physical tests, PaG maintained success rates between 73% and 93%. In contrast, the "push-to-wall" baseline often dropped to a 0% success rate when faced with beveled edges or environments without a physical wall.

Conclusion and Future Work

The push-to-edge strategy provides a robust solution for picking up flat objects in unstructured environments. By treating pushing and grasping as a unified problem through a single DRL network, the system achieves high efficiency and successful sim-to-real transfer.

Future research will aim to address current limitations, such as the assumption of uniform object density and the challenges posed by extremely complex textures. Additionally, the authors plan to explore "cluttered" scenarios where multiple ungraspable objects are layered or stacked, requiring more sophisticated sequencing of actions.

Summary updated

Abstract

Grasping an object when it is in an ungraspable pose is a challenging task, such as books or other large flat objects placed horizontally on a table. Inspired by human manipulation, we address this problem by pushing the object to the edge of the table and then grasping it from the hanging part. In this paper, we develop a model-free Deep Reinforcement Learning framework to synergize pushing and grasping actions. We first pre-train a Variational Autoencoder to extract high-dimensional features of input scenario images. One Proximal Policy Optimization algorithm with the common reward and sharing layers of Actor-Critic is employed to learn both pushing and grasping actions with high data efficiency. Experiments show that our one network policy can converge 2.5 times faster than the policy using two parallel networks. Moreover, the experiments on unseen objects show that our policy can generalize to the challenging case of objects with curved surfaces and off-center irregularly shaped objects. Lastly, our policy can be transferred to a real robot without fine-tuning by using CycleGAN for domain adaption and outperforms the push-to-wall baseline.
BibTeX
@inproceedings{zhang2023reinforcement,
  author = {Hao Zhang and Hongzhuo Liang and Lin Cong and Jianzhi Lyu and Long Zeng and Pingfa Feng and Jianwei Zhang},
  title = {{Reinforcement Learning Based Pushing and Grasping Objects from Ungraspable Poses}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=6BMWXeW94p},
}
Figure