Sequential Multi-Object Grasping with One Dexterous Hand

Sicheng He, Zeyu Shangguan, Kuanning Wang, Yongchong Gu, Yuqian Fu, Yanwei Fu, Daniel Seita

Abstract

This paper introduces SeqMultiGrasp, a novel system that enables a robotic arm equipped with a dexterous multi-fingered hand to sequentially grasp and hold two separate objects. By synthesizing constrained single-object grasps, merging them into multi-object configurations, and training a point-cloud-conditioned diffusion model, the authors demonstrate autonomous multi-object manipulation without requiring objects to be pre-clustered. The system achieves a 64.4% success rate in real-world trials, bridging the gap between high-DOF hardware capabilities and autonomous execution.

Introduction

Multi-fingered robotic hands possess high degrees of freedom (DOF), theoretically allowing them to manipulate multiple objects independently—much like humans do when holding keys and a phone simultaneously. However, most robotic grasping research treats these complex hands like simple parallel-jaw grippers, focusing entirely on single-object scenarios.

Existing multi-object grasping research often relies on strong assumptions: objects must be clustered closely for a simultaneous scoop, or a human must manually place objects into the robot's hand. SeqMultiGrasp addresses the harder problem of autonomous, sequential multi-object grasping. The robot must pick up one object, secure it, navigate to a second object located elsewhere, and grasp it without dropping the first.

Methodology

The authors utilize a "divide and conquer" approach to manage the high-DOF control space:

  1. Finger Allocation & Grasp Styles: The grasping sequence is strictly partitioned. The first object is picked up using a Pinch-like Grasp (thumb, index, and middle fingers). The second object is secured using a Side Grasp (ring finger and palm).
  2. Grasp Synthesis & Validation: Building on the Differentiable Force Closure (DFC) algorithm, the system optimizes an energy function to generate single-object grasps constrained to assigned finger subsets. These are validated in the ManiSkill physics simulator to ensure robustness and collision-free configurations.
  3. Merging & Learning: Validated single-object grasps are merged into multi-object hand configurations. To enable real-time execution, a diffusion model is trained on this generated dataset. Conditioned on 3D point clouds extracted via PointNet++, the model quickly proposes multi-object grasp poses for novel scenes.
  4. Execution: The robot executes grasps using a robust "squeeze-and-lift" heuristic paired with CuRobo motion planning, facilitating effective sim-to-real transfer.
Overview of the SeqMultiGrasp pipeline, including contact candidate selection, grasp synthesis via DFC, and the diffusion-based perception model.

Experimental Results

Because no dataset existed for sequential multi-object grasping with the Allegro Hand, the authors curated a custom dataset using objects from YCB, ContactDB, and Omni6DPose benchmarks. For real-world perception, they utilized COLMAP and 2D Gaussian Splatting to reconstruct high-quality scene point clouds.

Performance Metrics

The system was evaluated using a Franka Panda arm and a 16-DOF Allegro Hand across 1,600 simulation trials and 90 real-world trials:

  • Simulation: Using directly synthesized grasps (SG), the system achieved an 82.7% success rate. The diffusion model (LG) achieved 65.8%.
  • Real-World: The system successfully transferred to physical hardware, achieving a 64.4% success rate with synthesized grasps and 56.7% with the diffusion model.
Sequential execution in a real-world environment: (1-2) picking up a small object with a pinch grasp, (3-4) navigating to and securing a second object (soup can) with a side grasp.

Key Insights and Discussion

  • Anatomical Partitioning: Strictly assigning fingers to specific objects bypasses the complexity of overlapping control constraints, making multi-object synthesis computationally tractable.
  • Merging Strategy: Synthesizing highly constrained single-object grasps and merging disjoint kinematic chains is a viable path for generating complex multi-object training data.
  • Heuristics for Dexterity: While Reinforcement Learning is popular, the authors found that a carefully designed squeeze-and-lift heuristic combined with collision-free motion planning transfers to the real world more reliably than learned policies.

Limitations

The current system is limited to rigid objects and a hardcoded capacity of exactly two objects. Furthermore, the real-world perception pipeline (Gaussian Splatting) is computationally expensive, presenting a bottleneck for high-speed real-time robotics. Future work aims to scale dataset diversity and explore sequential grasping for deformable objects.

Summary updated

BibTeX
@inproceedings{he2025sequential,
  author = {Sicheng He and Zeyu Shangguan and Kuanning Wang and Yongchong Gu and Yuqian Fu and Yanwei Fu and Daniel Seita},
  title = {{Sequential Multi-Object Grasping with One Dexterous Hand}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11246620},
}
Figure