TL;DR
This paper introduces a scalable simulation framework that automatically generates diverse reach-to-grasp demonstration data to train biosignals-free prosthetic hands. By leveraging physics-based grasp synthesis and procedurally generated environments, the framework trains imitation learning policies that achieve over 90% grasp success in real-world testing, significantly outperforming models trained on limited real-world datasets.
What Problem Does This Paper Address?
Traditional upper-limb prostheses rely heavily on surface electromyography (sEMG) signals, requiring users to intentionally contract specific muscles to trigger different grasp types. This process is physically and mentally exhausting. To alleviate this, researchers have proposed "biosignals-free" autonomous prostheses that use a wrist-mounted camera to automatically handle low-level grasping, leaving the user responsible only for moving their arm near the object.
However, training these autonomous hands via imitation learning requires massive amounts of human demonstration data. Collecting this data in the real world is a notorious bottleneck: it is labor-intensive, slow, prone to human fatigue, and difficult to scale across diverse objects and environments. This paper tackles this data bottleneck by shifting demonstration collection entirely into a scalable simulation pipeline.
Methods
The authors developed an automated data collection pipeline within NVIDIA Isaac Lab to synthesize high-quality reach-to-grasp demonstrations without human intervention. The framework consists of four main pillars:
- Grasp Synthesis: Instead of hand-crafting grasps, the system uses an optimization algorithm called BoDex to automatically discover physically feasible, contact-rich dexterous grasps for a 6-DoF prosthetic hand (the PSYONIC Ability Hand).
- Trajectory Retargeting: The system takes a baseline database of natural human wrist trajectories and mathematically morphs them so that the final wrist pose perfectly aligns with the synthesized grasp configuration.
- Procedural Scene Generation: To prevent the vision model from overfitting to a single background, the authors used Infinigen Indoors to procedurally generate 10 photorealistic virtual rooms with randomized furniture, lighting, and layouts.
- System Identification: To bridge the sim-to-real gap, an evolutionary algorithm calibrates the simulated joint dynamics (stiffness and damping) to match the physical properties of the real prosthetic hand.
Using this setup, the framework automatically records wrist-view RGB images, proprioceptive joint states, and target actions to build a robust imitation learning dataset.
Datasets
- Google Scanned Objects (GSO): A dataset of high-quality 3D-scanned household items. The authors selected 200 diverse objects for their simulation environment.
- Simulated Demonstration Dataset: The authors automatically collected 2,000 successful reach-to-grasp trials across the 200 objects and 10 procedurally generated rooms.
Key Insights and Ideas
- The Synergy of Diversity: A key finding is that scaling both object shapes and environmental backgrounds simultaneously provides a complementary boost to policy generalization. Training on more rooms makes the model robust to novel backgrounds, while training on more objects improves finger-closure dexterity.
- The Vulnerability of Real-World Data: Models trained on real-world data are often brittle because humans tend to collect data against uniform backgrounds (e.g., a clean white lab table). When the environment changes slightly, these models fail dramatically. Simulation allows for cheap, infinite background randomization, creating much more robust visual representations.
- Action Chunking Matters: Benchmarking showed that transformer-based and central-focused architectures that predict sequences of future actions (like ACT and VTM-VAE) generalize significantly better to unseen objects than diffusion-based alternatives in this specific control paradigm.
Key Results
The authors evaluated their simulation-trained policy against state-of-the-art baselines in both simulation and extensive real-world trials involving 12 human participants (1,800 total trials).
- Sim-to-Real Superiority: In real-world testing across three different scenes, the proposed simulation-trained model achieved an overall 92% grasp success rate.
- Outperforming Real-World Baselines: It significantly outperformed VTM-VAE trained on real-world data, which achieved only a 66% success rate.
- Robustness to Visual Shifts: The real-world baseline completely collapsed (0% success rate) when tested on a novel yellow table due to severe visual domain shift. In contrast, the simulation-trained model maintained high performance (90-95% success) across brown, white, and yellow tables, proving the value of procedural simulation training.
Limitations and Future Work
While highly successful, the framework currently evaluates isolated objects on flat surfaces. The authors note that future work must expand the dataset to handle multi-object clutter, complex obstacle avoidance, and articulated objects (like opening drawers). Additionally, they plan to explore advanced domain adaptation techniques to further minimize the sim-to-real gap and extend the framework to support different types of prosthetic hardware and amputation levels.
Summary updated