TL;DR
This paper introduces a novel framework for mobile robots to efficiently and safely hand over objects to humans, even with arbitrary human hand poses. By integrating advanced perception, grasping, and navigation techniques, the system aims to achieve a handover experience as fluid and comfortable as human-to-human interactions.
What Problem Does This Paper Address?
The seamless transfer of objects between robots and humans, known as robot-to-human handover, is a critical yet challenging aspect of human-robot collaboration. Existing methods often struggle with dynamic environments, arbitrary human hand poses, and ensuring a comfortable and safe experience for the human. This paper tackles the problem of enabling a mobile robot to not only locate and grasp an object but also navigate to a human and perform a handover that is fast, safe, and adapts to the human's movements and posture, mimicking the natural fluidity of human interactions.
Methods
The proposed system is broken down into five distinct modules:
- Perception: The robot uses lidar-inertial odometry (LIO) for robust localization and mapping, creating a 2.5D grid map that includes elevation data for navigable zones. It employs the YOLO object detection system to find target objects.
- Grasp Planning: To determine the optimal grasp, the system uses GraspNet to generate feasible grasp poses on the object and Ganhand to predict the human's hand grasp pose. These are then checked for collisions using GraspIt!.
- Navigation: The robot utilizes an exploration planner inspired by frontier-based approaches and a local planner that incorporates obstacle avoidance. It dynamically adjusts its path based on the human's current posture and movements.
- Hand Pose Generation: Using RGB-D camera data, the system leverages MediaPipe for precise hand and finger tracking, generating a detailed mesh model of the human hand. This model is crucial for identifying the optimal handover point.
- Handover Execution: The robot plans and executes motion to deliver the object, adapting its movements in real-time based on the human's hand position and movements, ensuring a collision-free and smooth transfer.
Datasets
- YCB Object and Model Set [29]: Used for training and evaluating grasp generation.
- Matterport3D simulation environment: Used for evaluating the navigation strategy.
- Human hand data: The system utilizes data from real objects grasped by human hands for training.
Key Insights and Ideas
- Adaptive Handover: The system's ability to adapt to arbitrary human hand poses and movements in real-time is a significant advancement, moving beyond fixed handover points.
- Integrated Approach: Combining robust localization, object detection, grasp planning, and dynamic navigation into a single framework is key to achieving efficient and safe handovers.
- Human Preference in Grasping: Explicitly considering the human's hand pose for grasping aims to improve comfort and naturalness.
- 2.5D Grid Map for Navigation: Using a 2.5D grid map with elevation data provides a good balance between computational efficiency and detailed environmental representation for navigation.
Key Results
- High Success Rate: The system achieved a high success rate in handover experiments across different objects (Mug, Can, Scissors), with rates of 100%, 96%, and 100% for the first hand move, and 90%, 86%, and 96% for the second hand move, respectively.
- Comfort and Safety: User studies indicated that the proposed method significantly enhanced comfort satisfaction (4.4/5) compared to a comparative method (3.7/5), closely emulating human-to-human handover comfort (5/5). Safety satisfaction was also rated highly (4.0/5).
- Efficiency: While not explicitly quantified in terms of speedup against specific benchmarks, the system aims for fast and efficient handovers, with a target of completion within 8 seconds.
- Exploration Performance: The proposed exploration strategy, which scatters observation points near large obstacles, showed superior volume coverage and competitive completion times compared to a baseline method in simulation.
Limitations and Future Work
- Global Planner Optimality: The current planning algorithm is not globally optimal, which can lead to longer paths and potential timeouts. Future work aims to address this for improved speed and reliability.
- Occlusion Issues: The robot can fail if the human hand or the robotic arm occludes the camera, preventing accurate pose estimation. Future work will focus on solving these occlusion problems.
- Reactive Strategy: The system currently lacks a reactive strategy to stop if the human hand moves unexpectedly during delivery.
- Sensor Limitations: Depth accuracy of RGB-D cameras in challenging conditions and the effective range for human hand skeleton prediction are noted limitations.
- Head Joint Control: The study does not account for controlling the robot's head joint, which could limit its field of view. Future work may include head joint control to expand the viewing range.
- Computational Resources: While the NVIDIA Jetson Orin platform can handle the computational demands, the paper notes the significant resource requirements for elevation map processing, object detection, and hand mesh reconstruction.
Summary updated