Learning Pregrasp Manipulation of Objects from Ungraspable Poses

Zhaole Sun, Kai Yuan, Wenbin Hu, Chuanyu Yang, Zhibin LiICRA 2020 · DOI

TL;DR

This paper introduces a novel Deep Reinforcement Learning (DRL) framework for "pregrasp manipulation," enabling robots to push and lift objects into graspable positions when initially ungraspable. Their sim-to-real transfer approach, using a spherical end-effector and a simple reward function, demonstrates robust performance across various object types and support surfaces.

What Problem Does This Paper Address?

Robotic grasping is a fundamental task, but many existing methods struggle when objects are in "ungraspable poses." This occurs when an object is positioned such that no direct grasp is possible, for example, a large, flat box lying flat on a table where the gripper cannot fit underneath. Traditional approaches often rely on evaluating pre-defined graspable positions, failing when none exist. This paper tackles this limitation by proposing "pregrasp manipulation" – a set of actions (pushing and lifting) that proactively create a graspable configuration before the actual grasp attempt. This is inspired by human dexterity, where we often adjust an object's position or orientation to make it easier to pick up.

Methods

The core of this work is a Deep Reinforcement Learning (DRL) framework based on the Soft Actor-Critic (SAC) algorithm. Here's a breakdown of their approach:

  • DRL Framework: They use SAC for its sample efficiency and exploration capabilities, training a policy to learn pregrasp manipulation.
  • State Representation: The robot's state is defined by proprioceptive sensor data (end-effector pose) and external visual information (object pose). Specifically, it includes the distance between the end-effector and the object, their center positions, and pitch orientations.
  • Action Representation: The policy outputs incremental actions for the end-effector's position (Δy, Δz) and pitch orientation (Δpitch). These actions are applied to the current end-effector target command.
  • Reward Design: A simple, yet effective, reward function is used: r = λ₁r_pitch + λ₂r_dist. r_pitch rewards lifting the object higher, while r_dist penalizes contact between the end-effector and the object during pushing. This reward is designed to be independent of the simulation-to-reality gap.
  • Sim-to-Real Transfer: A key contribution is the direct transfer of policies trained in simulation to real-world robots. This is achieved through careful design of the state and action spaces, and by randomizing physics parameters (mass, friction, object position, etc.) during simulation training to enhance robustness.
  • End-Effector: A spherical end-effector is used, offering a larger contact surface and simplifying collision calculations in simulation.

Datasets

The paper does not explicitly mention or create new datasets. Instead, it focuses on training a DRL policy within a simulated environment (PyBullet) and then transferring that learned policy to a real-world robotic setup. The "dataset" for evaluation consists of various real-world objects and support surfaces, as detailed in the experiments section.

Key Insights and Ideas

  • Pregrasp Manipulation is Crucial: The paper convincingly demonstrates that for many real-world scenarios, creating a graspable pose before attempting to grasp is essential, especially for objects in ungraspable configurations.
  • DRL for Dexterous Manipulation: DRL, specifically SAC, proves to be a powerful tool for learning complex, sequential manipulation tasks like pushing and lifting, even with a relatively simple reward function.
  • Sim-to-Real is Achievable: The success of direct sim-to-real transfer without explicit domain randomization or fine-tuning highlights the robustness of their learned policy and the effectiveness of their training methodology.
  • Spherical End-Effectors for Simplicity: The choice of a spherical end-effector, while seemingly simple, contributes to the overall success by simplifying state representation and collision detection, and its broad contact area is beneficial for pushing and lifting.
  • Environment-Independent Reward: The simple, task-focused reward function (lifting height and avoiding collisions) is a significant insight, as it generalizes well across different objects and scenarios without needing object-specific tuning.

Key Results

  • Task Completion: The learned policy successfully generated pregrasp configurations for a variety of objects, with lifting angles consistently above the required threshold for grasping.
  • Robustness: The policy demonstrated robustness to:
  • External Disturbances: It could handle objects dropped from a height and impulse disturbances.
  • Initial Conditions: It performed well across a range of initial object positions and yaw angles.
  • Generalization: The policy, trained on a single object (Object 1) and a vertical wall, generalized effectively to:
  • Different Objects: Various sizes, shapes, and weights (up to 0.78kg).
  • Different Support Surfaces: Inclined surfaces, deformable surfaces (cardboard, sponge), and even gripper fingertips.
  • Sim-to-Real Success: The policy trained in simulation transferred directly to the real robot, performing pregrasp manipulation without further training.

Limitations and Future Work

  • Support Surface Requirement: The method requires a static support surface (e.g., a wall, another object) against which the target object can be pushed. It cannot create clearance in open space.
  • Object Properties: While the method makes no assumptions about object shape or material for grasping, the pregrasp action relies on the object being rigid enough to be pushed and lifted. Deformable objects are tested, but their behavior might be more complex.
  • Limited Dexterity: The current actions are limited to pushing and lifting. More complex pregrasp manipulations (e.g., rotating an object in open space) are not covered.
  • Future Work: The authors suggest integrating their pregrasp manipulation framework into a larger grasping pipeline, enabling end-to-end grasping of objects from initially ungraspable poses. Further exploration of more complex pregrasp actions and handling of highly deformable objects could also be future directions.

Summary updated

BibTeX
@inproceedings{sun2020learning,
  author = {Zhaole Sun and Kai Yuan and Wenbin Hu and Chuanyu Yang and Zhibin Li},
  title = {{Learning Pregrasp Manipulation of Objects from Ungraspable Poses}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  doi = {10.1109/icra40945.2020.9196982},
}
Figure