Abstract
We approach the problem of high-DOF reaching-and-grasping via learning joint planning of grasp and motion with deep reinforcement learning. To resolve the sample efficiency issue in learning the high-dimensional and complex control of dexterous grasping, we propose an effective representation of grasping state characterizing the spatial interaction between the gripper and the target object. To represent gripper-object interaction, we adopt Interaction Bisector Surface (IBS) which is the Voronoi diagram between two close by 3D geometric objects and has been successfully applied in characterizing spatial relations between 3D objects. We found that IBS is surprisingly effective as a state representation since it well informs the fine-grained control of each finger with spatial relation against the target object. This novel grasp representation, together with several technical contributions including a fast IBS approximation, a novel vector-based reward and an effective training strategy, facilitate learning a strong control model of high-DOF grasping with good sample efficiency, dynamic adaptability, and cross-category generality. Experiments show that it generates high-quality dexterous grasp for complex shapes with smooth grasping motions.
Introduction
Robotic grasping is traditionally decomposed into two distinct stages: grasp synthesis, which identifies a viable final pose, and motion planning, which computes a collision-free trajectory to reach that pose. This decoupled approach often results in sub-optimal performance, as the synthesized grasp might be unreachable or lead to collisions during the approach phase.
Jointly planning both motion and grasp is theoretically superior but computationally daunting, particularly for high-degree-of-freedom (DOF) grippers such as a 24-DOF five-fingered hand. While deep reinforcement learning (RL) offers a potential solution through closed-loop feedback, it typically suffers from poor sample efficiency due to the massive state space. The authors identify the lack of an informative representation of the interaction between the hand and the object as a primary bottleneck.
Methodology: The Interaction Bisector Surface
The core innovation of this work is the use of the Interaction Bisector Surface (IBS) as the primary state representation for a Soft Actor-Critic (SAC) RL model. The IBS is the Voronoi boundary between two 3D objects, representing the "mid-surface" between the gripper and the target.
As the hand moves, the IBS updates dynamically, providing a rich geometric map of the proximity of each finger to the object. To make this feasible for real-time RL, the authors developed a fast grid-based IBS approximation. The framework employs PointNet-based encoders to extract both global features of the interaction and local features segmented by finger proximity.
Technical Enhancements
Beyond the IBS representation, several key components contribute to the model's success:
- Vector-based Q-Representation: Instead of a scalar reward, the critic network outputs a vector. This disentangles standard rewards from finger-specific contact information, allowing the model to learn precise collision avoidance and contact seeking simultaneously.
- Double Replay Buffer: Learning is accelerated using a "bootstrapping" buffer containing imperfect, synthetically generated demonstrations (e.g., linear interpolations) alongside a standard self-exploration buffer.
Key Results and Insights
The proposed method was evaluated on 500 objects from the KIT and GD datasets, with generalization tests on unseen objects from YCB, BigBIRD, and ContactPose.
- Superior Performance: On the YCB dataset, the method achieved a 68.3% average success rate, significantly outperforming baselines like GraspIt! (52.1%) and previous RL approaches (13.6%). When multiple initial configurations were allowed, success rates reached 90.0% to 100.0%.
- Dynamic Adaptability: Because the IBS is updated every frame, the system can grasp moving objects, adapting its trajectory in real-time.
- IBS as a Bridge: The IBS acts as a powerful relational descriptor. It doesn't just describe the hand or the object in isolation; it describes the relationship between them, simplifying the learning of complex control mappings.
- Effective Use of Imperfect Data: The model demonstrates that even "imperfect" demonstrations can effectively guide an RL agent out of sparse-reward regions, provided the state representation is sufficiently informative.
Limitations and Future Work
While the results are promising, several challenges remain. The current reward function prioritizes success over aesthetics, sometimes leading to "unnatural" finger configurations. Additionally, the model struggles with very flat objects due to strict collision avoidance constraints. Future work will focus on bridging the sim-to-real gap, incorporating dynamic information like velocity into the state representation, and improving the naturalness of the generated motions.
Summary updated