Abstract
Dexterous functional tool-use grasping is essential for effective robotic manipulation. However, existing approaches struggle with dataset construction and generalizability across object scales. We propose ScaleADFG, a framework featuring a fully automated dataset construction pipeline and a lightweight grasp generation network. By using an affordance-based algorithm, we synthesize diverse tool-use grasps without human demonstrations, accommodating varied object-hand size ratios. Our dataset includes five object categories with over 1,000 unique shapes and 15 scale variations each, totaling over 60,000 grasps per hand. Experiments show that ScaleADFG significantly improves functional grasp stability and generalizability, enabling effective zero-shot transfer to real-world objects.
Introduction
Functional grasping—the act of picking up a tool specifically to use it—remains a significant challenge in robotics. Unlike universal grasping, functional grasping requires precise hand placement relative to the tool's active parts (e.g., the trigger of a drill). Current research is hindered by three main issues: the high cost of human-demonstrated data, the mismatch between human and robotic hand kinematics, and the sensitivity of models to varying object scales. ScaleADFG addresses these by automating data generation and prioritizing object affordances over human imitation.
Methodology: The ScaleADFG Framework
The ScaleADFG framework is divided into an automated dataset pipeline and a single-stage prediction network.
Automated Dataset Pipeline
The pipeline eliminates the need for manual data collection through three key steps:
- Large-Scale 3D Asset Generation: Using generative models like InstantMesh and TRELLIS, the system converts vast collections of 2D images into high-quality 3D models.
- Affordance Retrieval: Leveraging pre-trained vision models (e.g., DINOv2), the framework automatically identifies functional parts (nozzles, triggers) and grasping parts (handles) on the 3D assets.
- Optimization-based Synthesis: A two-stage optimization process aligns the robotic hand with the object's functional parts and refines the grasp to ensure stability and force closure.
ScaleADFG-Net
The prediction network is a lightweight Conditional Variational Autoencoder (CVAE). By training on a massive, multi-scale dataset, the network learns to generate high-quality grasps in a single forward pass, bypassing the computationally expensive refinement steps required by previous methods.
The ScaleADFG Dataset
The resulting dataset is a major contribution to the field, offering unprecedented scale and diversity across five categories: Drills, Flashlights, Staplers, Lotion Bottles, and Spray Bottles.
Dataset Characteristics
- Shape Diversity: Over 1,000 unique 3D models per category.
- Scale Variability: Each object is scaled across 15 variations, ensuring the model learns to handle different hand-to-object size ratios.
- Multi-Hand Support: Data is generated for both Allegro and Shadow robotic hands.
Key Results and Insights
ScaleADFG demonstrates that affordance-based logic is superior to human imitation for robotic hands. By focusing on the tool's functional requirements, the robot discovers grasps optimized for its own mechanical constraints.
- Simulation Performance: In IsaacSim, the framework achieved a 94.58% success rate, far exceeding the 72.47% of previous state-of-the-art methods.
- Real-World Generalization: On an Aubo I5 arm with an Allegro hand, the system maintained a 76.0% grasp success rate on entirely novel objects, proving its zero-shot transfer capabilities.
- Scale Robustness: Training on a wide spectrum of sizes allows the model to interpolate effectively, handling unseen object scales with high precision.
Conclusion
ScaleADFG provides a scalable solution for teaching robots functional tool use. By automating the creation of multi-scale, affordance-rich datasets, it removes the bottleneck of human demonstrations. While currently focused on rigid objects and simple interactions, future work will explore articulated tools and manipulation in cluttered environments.
Summary updated