ScaleADFG: Affordance-based Dexterous Functional Grasping via Scalable Dataset

Sizhe Wang, Yifan Yang, Yongkang Luo, Daheng Li, Wei Wei, Yan Zhang, Peiying Hu, Y. W. Fu, Haonan Duan, Jia Sun, Peng Wang

Abstract

Dexterous functional tool-use grasping is essential for effective robotic manipulation. However, existing approaches struggle with dataset construction and generalizability across object scales. We propose ScaleADFG, a framework featuring a fully automated dataset construction pipeline and a lightweight grasp generation network. By using an affordance-based algorithm, we synthesize diverse tool-use grasps without human demonstrations, accommodating varied object-hand size ratios. Our dataset includes five object categories with over 1,000 unique shapes and 15 scale variations each, totaling over 60,000 grasps per hand. Experiments show that ScaleADFG significantly improves functional grasp stability and generalizability, enabling effective zero-shot transfer to real-world objects.

Introduction

Functional grasping—the act of picking up a tool specifically to use it—remains a significant challenge in robotics. Unlike universal grasping, functional grasping requires precise hand placement relative to the tool's active parts (e.g., the trigger of a drill). Current research is hindered by three main issues: the high cost of human-demonstrated data, the mismatch between human and robotic hand kinematics, and the sensitivity of models to varying object scales. ScaleADFG addresses these by automating data generation and prioritizing object affordances over human imitation.

Methodology: The ScaleADFG Framework

The ScaleADFG framework is divided into an automated dataset pipeline and a single-stage prediction network.

Overview of the ScaleADFG framework, including automated dataset generation and the grasp prediction network.

Automated Dataset Pipeline

The pipeline eliminates the need for manual data collection through three key steps:

  1. Large-Scale 3D Asset Generation: Using generative models like InstantMesh and TRELLIS, the system converts vast collections of 2D images into high-quality 3D models.
  2. Affordance Retrieval: Leveraging pre-trained vision models (e.g., DINOv2), the framework automatically identifies functional parts (nozzles, triggers) and grasping parts (handles) on the 3D assets.
  3. Optimization-based Synthesis: A two-stage optimization process aligns the robotic hand with the object's functional parts and refines the grasp to ensure stability and force closure.

ScaleADFG-Net

The prediction network is a lightweight Conditional Variational Autoencoder (CVAE). By training on a massive, multi-scale dataset, the network learns to generate high-quality grasps in a single forward pass, bypassing the computationally expensive refinement steps required by previous methods.

The ScaleADFG Dataset

The resulting dataset is a major contribution to the field, offering unprecedented scale and diversity across five categories: Drills, Flashlights, Staplers, Lotion Bottles, and Spray Bottles.

The ScaleADFG dataset showcases significant diversity in object shapes, scales, and robotic hand configurations.

Dataset Characteristics

  • Shape Diversity: Over 1,000 unique 3D models per category.
  • Scale Variability: Each object is scaled across 15 variations, ensuring the model learns to handle different hand-to-object size ratios.
  • Multi-Hand Support: Data is generated for both Allegro and Shadow robotic hands.

Key Results and Insights

ScaleADFG demonstrates that affordance-based logic is superior to human imitation for robotic hands. By focusing on the tool's functional requirements, the robot discovers grasps optimized for its own mechanical constraints.

  • Simulation Performance: In IsaacSim, the framework achieved a 94.58% success rate, far exceeding the 72.47% of previous state-of-the-art methods.
  • Real-World Generalization: On an Aubo I5 arm with an Allegro hand, the system maintained a 76.0% grasp success rate on entirely novel objects, proving its zero-shot transfer capabilities.
  • Scale Robustness: Training on a wide spectrum of sizes allows the model to interpolate effectively, handling unseen object scales with high precision.

Conclusion

ScaleADFG provides a scalable solution for teaching robots functional tool use. By automating the creation of multi-scale, affordance-rich datasets, it removes the bottleneck of human demonstrations. While currently focused on rigid objects and simple interactions, future work will explore articulated tools and manipulation in cluttered environments.

Summary updated

Abstract

Dexterous functional tool-use grasping is essential for effective robotic manipulation of tools. However, existing approaches face significant challenges in efficiently constructing large-scale datasets and ensuring generalizability to everyday object scales. These issues primarily arise from size mismatches between robotic and human hands, and the diversity in real-world object scales. To address these limitations, we propose the ScaleADFG framework, which consists of a fully automated dataset construction pipeline and a lightweight grasp generation network. Our dataset introduce an affordance-based algorithm to synthesize diverse tool-use grasp configurations without expert demonstrations, allowing flexible object-hand size ratios and enabling large robotic hands (compared to human hands) to grasp everyday objects effectively. Additionally, we leverage pre-trained models to generate extensive 3D assets and facilitate efficient retrieval of object affordances. Our dataset comprising five object categories, each containing over 1,000 unique shapes with 15 scale variations. After filtering, the dataset includes over 60,000 grasps for each 2 dexterous robotic hands. On top of this dataset, we train a lightweight, single-stage grasp generation network with a notably simple loss design, eliminating the need for post-refinement. This demonstrates the critical importance of large-scale datasets and multi-scale object variant for effective training. Extensive experiments in simulation and on real robot confirm that the ScaleADFG framework exhibits strong adaptability to objects of varying scales, enhancing functional grasp stability, diversity, and generalizability. Moreover, our network exhibits effective zero-shot transfer to real-world objects. Project page is available at https://sizhe-wang.github.io/ScaleADFG_webpage
BibTeX
@misc{wang2025scaleadfg,
  author = {Sizhe Wang and Yifan Yang and Yongkang Luo and Daheng Li and Wei Wei and Yan Zhang and Peiying Hu and Y. W. Fu and Haonan Duan and Jia Sun and Peng Wang},
  title = {{ScaleADFG: Affordance-based Dexterous Functional Grasping via Scalable Dataset}},
  year = {2025},
  eprint = {2511.09602},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.09602},
}
Figure