Proactive robot task sequencing through real-time hand motion prediction in human–robot collaboration

Shyngyskhan Abilkassov, Michael Gentner, Almas Shintemirov, Eckehard Steinbach, Mirela PopaImage 2025 · DOI

TL;DR

This paper presents a novel framework that combines advanced deep learning for real-time human hand motion prediction with a specialized heuristic algorithm for proactive robot task sequencing. By accurately forecasting where a human's hands will be, robots can intelligently plan their own actions to avoid collisions and optimize task execution in shared workspaces, leading to safer and more efficient human-robot collaboration.

What Problem Does This Paper Address?

In human-robot collaboration (HRC), especially in dynamic industrial settings, robots need to go beyond simply reacting to their environment. Current reactive systems struggle to anticipate human intentions, leading to inefficiencies and potential safety issues. The core challenge is enabling robots to proactively plan their actions based on a prediction of future human behavior. Specifically, understanding and predicting human hand movements is crucial, as hands are primary tools for interaction and manipulation. This paper tackles the problem of integrating accurate, real-time human hand trajectory forecasting with efficient robot task sequencing to achieve truly proactive HRC.

Methods

The framework has two main components:

  • Hand Motion Prediction: The authors employ a modified UniFormer model, a powerful transformer-based architecture for video understanding. They introduce a multi-task learning loss that jointly predicts future hand positions and the contact delay (the time between a predicted hand position and actual contact). This dual focus on position and timing, along with a classification branch for hand existence, significantly improves prediction accuracy.
  • Task Sequencing: The predicted hand trajectories are fed into a novel task sequencing module. This module formulates the problem as a Dynamic Traveling Salesman Problem with Time Windows (D-TSPTW). To solve this efficiently, they developed Dynamic Variable Neighborhood Search (DynamicVNS), a heuristic algorithm that adapts to changing time windows and node availabilities predicted by the hand motion model. DynamicVNS is designed to be computationally advantageous over standard Variable Neighborhood Search (VNS) for dynamic scenarios.

Datasets

  • Ego4D Dataset: This large-scale egocentric video dataset was used for training and evaluating the hand motion prediction model. The paper specifically leverages the Ego4D Future Hand Prediction benchmark.
  • EgoVLP Pretraining: For initial model pretraining, the authors utilized clip-level annotations from EgoVLP.

Key Insights and Ideas

  • Predicting Contact Delay is Key: Simply predicting hand positions isn't enough. Accurately forecasting when a hand will be at a certain position (contact delay) is critical for robots to make timely decisions and avoid collisions.
  • Multi-task Learning for Robustness: Combining hand position prediction with contact delay regression and hand existence classification within a single model leads to more robust and accurate predictions, even in challenging scenarios.
  • Dynamic TSP for Dynamic Environments: Traditional TSP formulations are insufficient for HRC. Framing the problem as a D-TSPTW, where time windows change based on predicted human actions, is a more appropriate model.
  • Heuristics for Real-Time Planning: Solving complex optimization problems like D-TSPTW in real-time requires efficient heuristics. DynamicVNS offers a significant speedup over existing methods, making it suitable for dynamic HRC.
  • Bridging Forecasting and Planning: The paper successfully demonstrates a seamless integration pipeline, where the output of a deep learning forecasting model directly informs a sophisticated task planning algorithm.

Key Results

  • State-of-the-Art Prediction: The proposed UniFormer model with multi-task learning achieved state-of-the-art performance on the Ego4D Future Hand Prediction benchmark, outperforming previous methods in terms of Mean Key Frame Displacement Error (M.Disp.) and Contact Key Frame Displacement Error (C.Disp.). For example, their model achieved M.Disp. of 43.00 for the left hand and 44.36 for the right hand, significantly better than prior benchmarks.
  • Significant Speedup in Task Sequencing: DynamicVNS demonstrated a substantial speedup compared to the generalized VNS (GVNS) algorithm, particularly with "Random" and "Due time" initialization strategies. For instance, in one comparison, DVNS achieved a mean execution time of 0.326s compared to GVNS's 4.814s.
  • Reduced Execution Time in HRC: Real-world experiments on a UR10e robot showed that the proposed proactive approach significantly reduced task completion times compared to a naive reactive approach. In the "Pick to Box" scenario, execution time dropped from a mean of 45.43s to 41.00s, and in "Block to Block," from 17.57s to 13.94s.
  • Qualitative Success: Visualizations showed the model accurately predicting future hand positions and trajectories, closely following ground truth across diverse scenarios.

Limitations and Future Work

  • Dataset Limitations: While Ego4D is large, the paper notes that models trained on it might still face distribution differences compared to exocentric data.
  • Computational Demands: Although DynamicVNS is efficient, real-time performance in highly complex scenarios might still be challenging.
  • Future Work Suggestions:
    • Develop more advanced travel cost estimations considering robot motion profiles.
  • Incorporate explicit obstacle avoidance modules.
  • Conduct more extensive use-case experiments.
  • Explore end-to-end approaches that directly forecast human intentions for robot task sequencing.

Summary updated

Abstract

Human–robot collaboration (HRC) is essential for improving productivity and safety across various industries. While reactive motion re-planning strategies are useful, there is a growing demand for proactive methods that predict human intentions to enable more efficient collaboration. This study addresses this need by introducing a framework that combines deep learning-based human hand trajectory forecasting with heuristic optimization for robotic task sequencing. The deep learning model advances real-time hand position forecasting using a multi-task learning loss to account for both hand positions and contact delay regression, achieving state-of-the-art performance on the Ego4D Future Hand Prediction benchmark. By integrating hand trajectory predictions into task planning, the framework offers a cohesive solution for HRC. To optimize task sequencing, the framework incorporates a Dynamic Variable Neighborhood Search (DynamicVNS) heuristic algorithm, which allows robots to pre-plan task sequences and avoid potential collisions with human hand positions. DynamicVNS provides significant computational advantages over the generalized VNS method. The framework was validated on a UR10e robot performing a visual inspection task in a HRC scenario, where the robot effectively anticipated and responded to human hand movements in a shared workspace. Experimental results highlight the system’s effectiveness and potential to enhance HRC in industrial settings by combining predictive accuracy and task planning efficiency. • We enhance hand position forecasting with a novel loss, achieving state-of-the-art results. • Our method unifies forecasting and task planning using the Dynamic TSP with Time Windows. • We enable real-time hand motion prediction and seamless integration into robot planning.
BibTeX
@inproceedings{abilkassov2025proactive,
  author = {Shyngyskhan Abilkassov and Michael Gentner and Almas Shintemirov and Eckehard Steinbach and Mirela Popa},
  title = {{Proactive robot task sequencing through real-time hand motion prediction in human–robot collaboration}},
  booktitle = {Image},
  year = {2025},
  doi = {10.1016/j.imavis.2025.105443},
}
Figure