Abstract
Advances in robotics have been driving the development of human-robot interaction (HRI) technologies. However, accurately perceiving human actions and achieving adaptive control remains a challenge in facilitating seamless coordination between human and robotic movements. In this paper, we propose a hierarchical procedural framework to enable dynamic robot-assisted hand-object interaction. An open-loop hierarchy leverages the computer vision (CV)-based 3D reconstruction of the human hand, based on which motion primitives have been designed to translate hand motions into robotic actions. The low-level coordination hierarchy fine-tunes the robot's action by using the continuously updated 3D hand models. Experimental validation demonstrates the effectiveness of the hierarchical control architecture. The adaptive coordination between human and robot behavior has achieved a delay of $\leq 0.3$ seconds in the tele-interaction scenario. A case study of ring-wearing tasks indicates the potential application of this work in assistive technologies such as healthcare and manufacturing.
Introduction
Physical Human-Robot Interaction (pHRI) is a cornerstone of assistive technology, yet achieving "seamless coordination" remains a significant hurdle. Specifically, in Hand-Object Interaction (HOI) tasks—like a robot helping a person get dressed—the system must overcome two major challenges: perception and control. Accurately tracking and reconstructing a human hand in 3D using only standard RGB cameras is difficult due to noise and occlusions. Furthermore, translating those perceived human movements into fluid, adaptive robotic actions without the lag that typically frustrates human users is a persistent control challenge. Existing solutions often rely on cumbersome wearable sensors or high-latency processing that makes real-time assistance feel "clunky" or unsafe.
Methods
The proposed system utilizes a two-tier hierarchical procedural framework to bridge the gap between human intent and robotic execution.
High-Level Intent Recognition
The system employs MeshGraphormer, a transformer-based architecture, to reconstruct a 3D mesh of the hand from a single RGB image. These 3D coordinates serve as a "universal language," maximizing feature space differences between gestures. These coordinates are fed into a Motion Primitive Model (MPM) powered by a Bidirectional Long Short-Term Memory (Bi-LSTM) network. This model classifies sequences of hand movements into specific "primitives" such as "Come," "Back," or "Ring."
Low-Level Adaptive Control
Upon triggering a task like "Ring-wearing," the system switches to a closed-loop Action Alignment Model. This fine-tunes the robot's UR5 arm and gripper in real-time, adjusting the tool center point (TCP) to match the human finger's orientation and position, even as the person moves or bends their finger. To ensure responsiveness, the framework implements asynchronous processing pipelines and a moving average filter to smooth data oscillations caused by hand tremors.
Datasets
The 3D hand pose estimator was pre-trained on large-scale public datasets. For the MPM, a custom dataset of 1,000 samples was collected, consisting of 3D hand joint coordinate sequences (21 joints) captured over 2,000 frames of RGB data.
Key Results
The framework was validated through a "ring-wearing" case study, demonstrating high precision and low latency.
- Low Latency: The system achieved a maximum control delay of $\leq 0.3$ seconds, with the "come" command responding in just 0.168s.
- High Precision: In dynamic tracking, the robot maintained a maximum absolute translational error of 2.48 mm and a rotational error of 1.37 degrees.
- Adaptive Success: The robot successfully adjusted its orientation to match the human finger's bending angle (up to 18 degrees) and vertical displacement (up to 74.23 mm) in real-time.
A key insight was the model's ability to generalize across different users; a model trained on one person's hand successfully recognized gestures from "unknown" hands, suggesting that geometric relationships of 3D joints are a robust proxy for human intent.
Conclusion
This work presents a robust framework for real-time robot-assisted hand-object interaction. By decoupling high-level intent recognition from low-level adaptive alignment, the system provides responsive and precise assistance. While currently limited by a small set of pre-defined motion primitives, future work aims to expand the primitive library and incorporate interaction dynamics, such as force feedback, for even gentler physical contact. This technology holds significant promise for assistive dressing and other healthcare or manufacturing applications where delicate human-robot coordination is required.
Summary updated