Hierarchical Procedural Framework for Low-latency Robot-Assisted Hand-Object Interaction

Mingqi Yuan, Huijiang Wang, Kai-Fung Chu, Fumiya Iida, Bo Li, Wenjun Zeng

Abstract

Advances in robotics have been driving the development of human-robot interaction (HRI) technologies. However, accurately perceiving human actions and achieving adaptive control remains a challenge in facilitating seamless coordination between human and robotic movements. In this paper, we propose a hierarchical procedural framework to enable dynamic robot-assisted hand-object interaction (HOI). An open-loop hierarchy leverages the RGB-based 3D reconstruction of the human hand, based on which motion primitives have been designed to translate hand motions into robotic actions. The low-level coordination hierarchy fine-tunes the robot's action by using the continuously updated 3D hand models. Experimental validation demonstrates the effectiveness of the hierarchical control architecture. The adaptive coordination between human and robot behavior has achieved a delay of $\leq 0.3$ seconds in the tele-interaction scenario. A case study of ring-wearing tasks indicates the potential application of this work in assistive technologies such as healthcare and manufacturing.

Introduction

Physical Human-Robot Interaction (pHRI) is a critical frontier in robotics, requiring systems that can perceive human intent and react with high precision in real-time. Traditional systems often face two primary bottlenecks: perception and control. Tracking the 21 joints of a human hand in 3D space using standard 2D cameras is computationally intensive and often results in jittery data. Furthermore, translating these coordinates into smooth robotic movements typically introduces significant latency, hindering the natural flow of cooperation. This research addresses these challenges by introducing a hierarchical framework designed for low-latency, adaptive hand-object interaction (HOI).

Proposed Methodology

The framework employs a two-tier hierarchy to bridge the gap between high-level perception and low-level robotic control.

High-Level Perception

The "brain" of the system utilizes MeshGraphormer, a transformer-based architecture, to reconstruct 3D hand meshes and joint coordinates from a single RGB image stream. To interpret these coordinates as intent, a Motion Primitive Model (MPM) based on a Bidirectional Long Short-Term Memory (LSTM) network was developed. This model classifies sequences of hand movements into specific "Motion Primitives" (MPs) such as Keep, Come, Back, and Ring. To ensure stability, the system incorporates moving average and variance-based credibility filters to mitigate noise in the real-time video feed.

Overview of the hierarchical procedural framework, including the perception pipeline and motion primitive model.

Low-Level Coordination

Once a specific primitive (e.g., "Ring") is identified, the system enters a closed-loop cooperation mode. The robot uses the continuously updated 3D hand model to fine-tune the position of a UR5 robotic arm. A key innovation is the Action Alignment mechanism, which calculates the spatial angular deviation between the object (e.g., a ring) and the human finger. This allows the robot to dynamically adjust its end-effector orientation, ensuring the object remains perfectly aligned with the finger's central axis throughout the interaction.

Experimental Results

The framework was validated using a UR5 manipulator equipped with a RobotiQ gripper. The system was trained on a self-collected dataset of 2,000 frames per motion, supplemented by pre-trained weights from the FreiHAND dataset for initial mesh estimation.

Performance Metrics

The experimental results confirmed the system's ability to achieve near-human reaction speeds and high precision:

  • Latency: The command-to-action delay ranged between 0.168s and 0.24s, successfully meeting the $\leq 0.3$s target for seamless interaction.
  • Precision: The system maintained a maximum absolute translational error of 2.48 mm and a rotational error of 1.37 degrees.
Temporal tracking performance of the robot following human hand movements in the ring-wearing task.

Discussion and Future Work

A significant finding was the model's ability to generalize; the MPM successfully recognized gestures from unknown hands despite being trained on data from a single individual. This suggests that 3D joint sequences provide a robust feature space for gesture recognition.

While the current system is limited to four motion primitives and can be affected by total hand occlusion, it provides a modular foundation for more complex assistive tasks. Future research will focus on expanding the primitive library to support tasks like dressing, addressing occlusion through multi-camera setups, and incorporating force feedback to refine interaction dynamics during physical contact.

Summary updated

Abstract

Advances in robotics have been driving the development of human-robot interaction (HRI) technologies. However, accurately perceiving human actions and achieving adaptive control remains a challenge in facilitating seamless coordination between human and robotic movements. In this paper, we propose a hierarchical procedural framework to enable dynamic robot-assisted hand-object interaction (HOI). An open-loop hierarchy leverages the RGB-based 3D reconstruction of the human hand, based on which motion primitives have been designed to translate hand motions into robotic actions. The low-level coordination hierarchy fine-tunes the robot's action by using the continuously updated 3D hand models. Experimental validation demonstrates the effectiveness of the hierarchical control architecture. The adaptive coordination between human and robot behavior has achieved a delay of $\leq 0.3$ seconds in the tele-interaction scenario. A case study of ring-wearing tasks indicates the potential application of this work in assistive technologies such as healthcare and manufacturing.
BibTeX
@misc{yuan2024hierarchical,
  author = {Mingqi Yuan and Huijiang Wang and Kai-Fung Chu and Fumiya Iida and Bo Li and Wenjun Zeng},
  title = {{Hierarchical Procedural Framework for Low-latency Robot-Assisted Hand-Object Interaction}},
  year = {2024},
  eprint = {2405.19531},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.19531},
}
Figure