TL;DR
This paper introduces a Reinforcement Learning (RL) framework that masters complex "non-prehensile" manipulation—like sliding a card or tumbling a box—by decomposing the task into distinct pre-contact and post-contact stages. By combining a novel joint-space action design with an action-scaling curriculum, the system achieves high-performance zero-shot transfer from simulation to a real Franka Emika Panda robot, even for objects with physical properties unseen during training.
What Problem Does This Paper Address?
Most industrial robots are "pick-and-place" experts, but they struggle when grasping is impossible—think of a credit card flat on a table, a box too wide for a gripper, or an object tucked against a wall. These scenarios require non-prehensile manipulation: pushing, dragging, or tumbling an object into place.
Solving this is notoriously difficult for two reasons:
- The Exploration Problem: In RL, a robot moving randomly is unlikely to even touch an object, let alone manipulate it purposefully.
- The Sim-to-Real Gap: Physics simulators struggle to perfectly model friction and joint dynamics. A policy that works perfectly in a "clean" simulation often fails on a real robot due to hardware intricacies like gear friction or safety-limited joint torques.
Methods
The authors propose a modular system that splits the problem into three parts:
- Policy Decomposition: Instead of one giant neural network, they use two. The pre-contact policy ($\pi_{pre}$) decides exactly where and how the gripper should first touch the object. The post-contact policy ($\pi_{post}$) then takes over to execute the sequence of pushes or drags needed to reach the goal.
- Joint-Space Action Design: While many researchers use Operational Space Control (OSC) to move the robot in 3D "Cartesian" space, these authors found OSC has a massive sim-to-real gap. Instead, their policy predicts target end-effector poses and joint gains, which are converted to joint-space targets via Inverse Kinematics (IK). This makes the controller much more robust to real-world hardware friction.
- Action Scaling Curriculum: To prevent the robot from hitting safety limits (velocity/torque) while still allowing it to explore aggressively early in training, they implemented a curriculum that gradually shrinks the maximum allowed movement "residual" as the robot gets better at the task.
- Perception Module: They use a 2D keypoint detector trained entirely in simulation using domain randomization and momentum contrast learning to identify object poses from a single RGB camera feed.
Datasets
The researchers evaluated their method across three custom-designed simulation domains in IsaacGym:
- Card Domain: Sliding and re-orienting a flat card.
- Bump Domain: Pushing a large box over a physical obstacle and tumbling it.
- Wall Domain: Lifting and dragging an object against a vertical surface to bypass obstacles.
For real-world evaluation, they tested on a Franka Emika Panda robot using a variety of objects, including "out-of-distribution" items like sponges, tissue boxes, and heavy wooden blocks that were significantly different from the training shapes and masses.
Key Insights and Ideas
- Not all contacts are equal: The "aha moment" for exploration was realizing that rewarding a robot just for touching an object isn't enough. By using a dedicated pre-contact policy, the robot learns to seek out promising initial contacts that make the subsequent manipulation easier.
- Joint space > Cartesian space for transfer: The paper argues that Cartesian-space errors are harder for a robot to "correct" than joint-space errors. By moving the control logic into the joint space, they bypassed many of the modeling inaccuracies inherent in complex robot arms.
- Exploiting the environment: Unlike traditional planners that treat walls or gravity as obstacles to be avoided, this RL approach learns to use them as "fixtures" to help tumble or stabilize objects.
Key Results
- Zero-Shot Success: The system achieved an 87% success rate across all real-world tasks without any fine-tuning on the physical robot.
- Superiority over OSC: In real-world tests, the standard OSC-based approach (a common baseline) plummeted to success rates as low as 0% to 13%, highlighting the effectiveness of the authors' joint-space action design.
- Robust Generalization: The policy successfully manipulated non-rigid objects (like a sponge or a diaper box) and objects with masses 2-3x outside the range seen during training.
Limitations and Future Work
- Fixed Environments: The current policies are trained for specific environmental geometries (e.g., a specific bump or wall). If the environment changes significantly, the robot needs to be retrained.
- Shape Sensitivity: While the system generalizes to different masses and frictions, it still relies on 2D keypoints derived from a bounding box. Future work aims to incorporate more complex shape information (like point clouds) to handle highly irregular geometries.
- Static Camera: The system currently relies on a fixed RGB camera; moving to an eye-in-hand setup could improve flexibility.
Summary updated