Pre-and Post-Contact Policy Decomposition for Non-Prehensile Manipulation with Zero-Shot Sim-To-Real Transfer

Minchan Kim, Junhyek Han, Jaehyung Kim, Beomjoon Kim

TL;DR

This paper introduces a Reinforcement Learning (RL) framework that masters complex "non-prehensile" manipulation—like sliding a card or tumbling a box—by decomposing the task into distinct pre-contact and post-contact stages. By combining a novel joint-space action design with an action-scaling curriculum, the system achieves high-performance zero-shot transfer from simulation to a real Franka Emika Panda robot, even for objects with physical properties unseen during training.

What Problem Does This Paper Address?

Most industrial robots are "pick-and-place" experts, but they struggle when grasping is impossible—think of a credit card flat on a table, a box too wide for a gripper, or an object tucked against a wall. These scenarios require non-prehensile manipulation: pushing, dragging, or tumbling an object into place.

Solving this is notoriously difficult for two reasons:

  1. The Exploration Problem: In RL, a robot moving randomly is unlikely to even touch an object, let alone manipulate it purposefully.
  2. The Sim-to-Real Gap: Physics simulators struggle to perfectly model friction and joint dynamics. A policy that works perfectly in a "clean" simulation often fails on a real robot due to hardware intricacies like gear friction or safety-limited joint torques.

Methods

The authors propose a modular system that splits the problem into three parts:

  • Policy Decomposition: Instead of one giant neural network, they use two. The pre-contact policy ($\pi_{pre}$) decides exactly where and how the gripper should first touch the object. The post-contact policy ($\pi_{post}$) then takes over to execute the sequence of pushes or drags needed to reach the goal.
  • Joint-Space Action Design: While many researchers use Operational Space Control (OSC) to move the robot in 3D "Cartesian" space, these authors found OSC has a massive sim-to-real gap. Instead, their policy predicts target end-effector poses and joint gains, which are converted to joint-space targets via Inverse Kinematics (IK). This makes the controller much more robust to real-world hardware friction.
  • Action Scaling Curriculum: To prevent the robot from hitting safety limits (velocity/torque) while still allowing it to explore aggressively early in training, they implemented a curriculum that gradually shrinks the maximum allowed movement "residual" as the robot gets better at the task.
  • Perception Module: They use a 2D keypoint detector trained entirely in simulation using domain randomization and momentum contrast learning to identify object poses from a single RGB camera feed.

Datasets

The researchers evaluated their method across three custom-designed simulation domains in IsaacGym:

  • Card Domain: Sliding and re-orienting a flat card.
  • Bump Domain: Pushing a large box over a physical obstacle and tumbling it.
  • Wall Domain: Lifting and dragging an object against a vertical surface to bypass obstacles.

For real-world evaluation, they tested on a Franka Emika Panda robot using a variety of objects, including "out-of-distribution" items like sponges, tissue boxes, and heavy wooden blocks that were significantly different from the training shapes and masses.

Key Insights and Ideas

  • Not all contacts are equal: The "aha moment" for exploration was realizing that rewarding a robot just for touching an object isn't enough. By using a dedicated pre-contact policy, the robot learns to seek out promising initial contacts that make the subsequent manipulation easier.
  • Joint space > Cartesian space for transfer: The paper argues that Cartesian-space errors are harder for a robot to "correct" than joint-space errors. By moving the control logic into the joint space, they bypassed many of the modeling inaccuracies inherent in complex robot arms.
  • Exploiting the environment: Unlike traditional planners that treat walls or gravity as obstacles to be avoided, this RL approach learns to use them as "fixtures" to help tumble or stabilize objects.

Key Results

  • Zero-Shot Success: The system achieved an 87% success rate across all real-world tasks without any fine-tuning on the physical robot.
  • Superiority over OSC: In real-world tests, the standard OSC-based approach (a common baseline) plummeted to success rates as low as 0% to 13%, highlighting the effectiveness of the authors' joint-space action design.
  • Robust Generalization: The policy successfully manipulated non-rigid objects (like a sponge or a diaper box) and objects with masses 2-3x outside the range seen during training.

Limitations and Future Work

  • Fixed Environments: The current policies are trained for specific environmental geometries (e.g., a specific bump or wall). If the environment changes significantly, the robot needs to be retrained.
  • Shape Sensitivity: While the system generalizes to different masses and frictions, it still relies on 2D keypoints derived from a bounding box. Future work aims to incorporate more complex shape information (like point clouds) to handle highly irregular geometries.
  • Static Camera: The system currently relies on a fixed RGB camera; moving to an eye-in-hand setup could improve flexibility.

Summary updated

Abstract

We present a system for non-prehensile manipulation that require a significant number of contact mode transitions and the use of environmental contacts to successfully manipulate an object to a target location. Our method is based on deep reinforcement learning which, unlike state-of-the-art planning algorithms, does not require apriori knowledge of the physical parameters of the object or environment such as friction coefficients or centers of mass. The planning time is reduced to the simple feed-forward prediction time on a neural network. We propose a computational structure, action space design, and curriculum learning scheme that facilitates efficient exploration and sim-to-real transfer. In challenging real-world non-prehensile manipulation tasks, we show that our method can generalize over different objects, and succeed even for novel objects not seen during training. Project website: https://sites.google.com/view/nonprenehsile-decomposition
BibTeX
@inproceedings{kim2023pre,
  author = {Minchan Kim and Junhyek Han and Jaehyung Kim and Beomjoon Kim},
  title = {{Pre-and Post-Contact Policy Decomposition for Non-Prehensile Manipulation with Zero-Shot Sim-To-Real Transfer}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=NUy9RbC3nN},
}

Loading related content...

Figure