Abstract
We present a system for non-prehensile manipulation that requires a significant number of contact mode transitions and the use of environmental contacts to successfully manipulate an object to a target location. Our method is based on deep reinforcement learning which, unlike state-of-the-art planning algorithms, does not require a priori knowledge of the physical parameters of the object or environment such as friction coefficients or centers of mass. The planning time is reduced to the simple feed-forward prediction time on a neural network. We propose a computational structure, action space design, and curriculum learning scheme that facilitates efficient exploration and sim-to-real transfer. In challenging real-world non-prehensile manipulation tasks, we show that our method can generalize over different objects and succeed even for novel objects not seen during training.
Introduction
Most industrial robotic systems are optimized for "pick-and-place" operations. However, many real-world scenarios involve objects that cannot be easily grasped—such as a credit card lying flat on a table, a box too wide for a gripper, or an object tucked tightly against a wall. These scenarios require non-prehensile manipulation: the ability to push, drag, or tumble an object into a target pose without a firm grasp.
Solving non-prehensile tasks is notoriously difficult for two primary reasons. First, the exploration problem in Reinforcement Learning (RL) makes it unlikely for a robot moving randomly to even touch an object, let alone manipulate it purposefully. Second, the sim-to-real gap remains a significant hurdle; physics simulators often struggle to model the nuances of friction and joint dynamics, causing policies that succeed in simulation to fail on real hardware.
Proposed Method
To address these challenges, we introduce a modular RL framework that decomposes the manipulation task and optimizes the control interface for robust transfer.
Policy Decomposition
Instead of training a single monolithic neural network, we decompose the task into two distinct stages:
- Pre-contact policy ($\pi_{pre}$): This policy determines the optimal approach and initial contact point. It learns to seek out promising initial contacts that simplify the subsequent manipulation.
- Post-contact policy ($\pi_{post}$): Once contact is established, this policy takes over to execute the sequence of pushes or drags required to reach the goal.
This decomposition significantly eases the exploration problem by providing a structured transition from reaching to manipulating.
Joint-Space Action Design
While many RL researchers utilize Operational Space Control (OSC) to command the robot in 3D Cartesian space, we found that OSC suffers from a massive sim-to-real gap due to unmodeled hardware intricacies like gear friction and safety-limited joint torques. Our system instead predicts target end-effector poses and joint gains, which are converted to joint-space targets via Inverse Kinematics (IK). By moving the control logic into the joint space, the policy becomes much more robust to the physical realities of the robot arm.
Action Scaling Curriculum
To balance aggressive exploration with hardware safety, we implemented an action-scaling curriculum. Early in training, the robot is allowed larger movement residuals to explore the environment. As the policy improves, the curriculum gradually shrinks these allowed residuals, refining the robot's precision and preventing it from hitting safety limits during real-world execution.
Perception
Our system utilizes a 2D keypoint detector to identify object poses from a single RGB camera feed. This detector is trained entirely in simulation using domain randomization and momentum contrast learning, allowing it to generalize to real-world visual conditions without manual labeling.
Experimental Evaluation
We evaluated our method across three distinct simulation domains in IsaacGym, each representing a unique non-prehensile challenge:
- Card Domain: Sliding and re-orienting a flat card on a surface.
- Bump Domain: Pushing a large box over a physical obstacle and tumbling it.
- Wall Domain: Lifting and dragging an object against a vertical surface to bypass obstacles.
For real-world validation, we deployed the policies on a Franka Emika Panda robot. We tested the system on a variety of objects, including "out-of-distribution" items like sponges, tissue boxes, and heavy wooden blocks that differed significantly in mass and friction from the training set.
Key Results and Insights
- Zero-Shot Success: The system achieved an 87% success rate across all real-world tasks without any fine-tuning on the physical robot.
- Robustness of Joint-Space Control: In head-to-head real-world tests, standard OSC-based approaches plummeted to success rates as low as 0% to 13%. This highlights that joint-space targets are far more effective for transferring complex contact-rich policies from simulation to reality.
- Environmental Exploitation: Unlike traditional planners that treat walls or gravity as obstacles, our RL approach learns to use the environment as a "fixture." The robot successfully uses walls to stabilize objects or the floor to tumble items into the correct orientation.
- Generalization: The policy successfully manipulated non-rigid objects (like a sponge) and objects with masses 2-3x outside the range seen during training, demonstrating the inherent robustness of the learned control laws.
Conclusion and Future Work
This work demonstrates that by decomposing policies and carefully designing the action space, deep RL can master complex non-prehensile manipulation with high reliability. While the current system is trained for specific environmental geometries, future work will focus on incorporating more complex shape information, such as point clouds, to handle highly irregular geometries and dynamic environments. Moving toward an eye-in-hand camera setup will also improve the system's flexibility in unstructured spaces.
Summary updated