Abstract
This paper introduces SeqMultiGrasp, a novel system that enables a robotic arm equipped with a dexterous multi-fingered hand to sequentially grasp and hold two separate objects. By synthesizing constrained single-object grasps, merging them into multi-object configurations, and training a point-cloud-conditioned diffusion model, the authors demonstrate autonomous multi-object manipulation without requiring objects to be pre-clustered. The system achieves a 64.4% success rate in real-world trials, bridging the gap between high-DOF hardware capabilities and autonomous execution.
Introduction
Multi-fingered robotic hands possess high degrees of freedom (DOF), theoretically allowing them to manipulate multiple objects independently—much like humans do when holding keys and a phone simultaneously. However, most robotic grasping research treats these complex hands like simple parallel-jaw grippers, focusing entirely on single-object scenarios.
Existing multi-object grasping research often relies on strong assumptions: objects must be clustered closely for a simultaneous scoop, or a human must manually place objects into the robot's hand. SeqMultiGrasp addresses the harder problem of autonomous, sequential multi-object grasping. The robot must pick up one object, secure it, navigate to a second object located elsewhere, and grasp it without dropping the first.
Methodology
The authors utilize a "divide and conquer" approach to manage the high-DOF control space:
- Finger Allocation & Grasp Styles: The grasping sequence is strictly partitioned. The first object is picked up using a Pinch-like Grasp (thumb, index, and middle fingers). The second object is secured using a Side Grasp (ring finger and palm).
- Grasp Synthesis & Validation: Building on the Differentiable Force Closure (DFC) algorithm, the system optimizes an energy function to generate single-object grasps constrained to assigned finger subsets. These are validated in the ManiSkill physics simulator to ensure robustness and collision-free configurations.
- Merging & Learning: Validated single-object grasps are merged into multi-object hand configurations. To enable real-time execution, a diffusion model is trained on this generated dataset. Conditioned on 3D point clouds extracted via PointNet++, the model quickly proposes multi-object grasp poses for novel scenes.
- Execution: The robot executes grasps using a robust "squeeze-and-lift" heuristic paired with CuRobo motion planning, facilitating effective sim-to-real transfer.
Experimental Results
Because no dataset existed for sequential multi-object grasping with the Allegro Hand, the authors curated a custom dataset using objects from YCB, ContactDB, and Omni6DPose benchmarks. For real-world perception, they utilized COLMAP and 2D Gaussian Splatting to reconstruct high-quality scene point clouds.
Performance Metrics
The system was evaluated using a Franka Panda arm and a 16-DOF Allegro Hand across 1,600 simulation trials and 90 real-world trials:
- Simulation: Using directly synthesized grasps (SG), the system achieved an 82.7% success rate. The diffusion model (LG) achieved 65.8%.
- Real-World: The system successfully transferred to physical hardware, achieving a 64.4% success rate with synthesized grasps and 56.7% with the diffusion model.
Key Insights and Discussion
- Anatomical Partitioning: Strictly assigning fingers to specific objects bypasses the complexity of overlapping control constraints, making multi-object synthesis computationally tractable.
- Merging Strategy: Synthesizing highly constrained single-object grasps and merging disjoint kinematic chains is a viable path for generating complex multi-object training data.
- Heuristics for Dexterity: While Reinforcement Learning is popular, the authors found that a carefully designed squeeze-and-lift heuristic combined with collision-free motion planning transfers to the real world more reliably than learned policies.
Limitations
The current system is limited to rigid objects and a hardcoded capacity of exactly two objects. Furthermore, the real-world perception pipeline (Gaussian Splatting) is computationally expensive, presenting a bottleneck for high-speed real-time robotics. Future work aims to scale dataset diversity and explore sequential grasping for deformable objects.
Summary updated