Real-time human progress estimation with online dynamic time warping for collaborative robotics

Davide De Lazzari, Matteo Terreran, Giulio Giacomuzzo, Siddarth Jain, Pietro Falco, Ruggero Carli, Stefano Ghidoni, Diego RomeresFrontiers in Robotics and AI 2025 · DOI

TL;DR

This paper introduces novel methods for real-time estimation of human action progress in collaborative robotics, crucial for seamless human-robot interaction. By adapting Dynamic Time Warping (DTW) with new distance metrics and integrating it into a Reinforcement Learning framework, they significantly improve robot assistance synchronization, leading to reduced waiting times and enhanced user experience.

What Problem Does This Paper Address?

In collaborative robotics, robots need to understand and anticipate human actions to work together effectively. A key challenge is accurately estimating how far along a human is in their current action, especially when human movements vary in speed and style. Existing methods often rely on high-level task recognition or react only after a human explicitly signals a need for help. This can lead to robots being out of sync, causing delays, interruptions, or even safety concerns. The paper tackles this by focusing on estimating progress at the atomic action level, enabling robots to proactively offer assistance at the right moment.

Methods

The core of the paper lies in novel adaptations of Dynamic Time Warping (DTW):

  • Open-end Soft-DTW (OS-DTW<sub>EU</sub>): This is presented as the first real-time application of Open-end Soft-DTW. It uses a "softmin" operation instead of a hard minimum in the DTW algorithm, making it more robust to local minima and better suited for continuous streams of data. The "EU" suffix indicates it uses the Euclidean distance.
  • OS-DTW<sub>WP</sub>: This is a new variant that incorporates a Windowed-Pearson (WP) distance metric. Instead of just looking at absolute positions (like Euclidean distance), WP measures local correlations within sliding windows of the trajectory. This allows it to better capture the shape of movements, even if they differ in speed or scale, making it more resilient to variations in human execution.
  • Proactive Assistance through action-Completion Estimation (PACE) framework: This framework integrates the OS-DTW methods. It uses Reinforcement Learning (RL) to train a robot policy that learns to synchronize its assistance with human actions based on the estimated progress. PACE models the human-robot interaction as a Partially Observable Markov Decision Process (POMDP) and uses a simulated environment for training.

Datasets

  • Human Demonstrations: The study collected data from 5 subjects performing a chair assembly task, with each subject performing the experiment 4 times. This resulted in 21 trajectories per action for tuning and training.
  • Test Dataset: For evaluating the PACE framework, 12 volunteers performed the chair assembly task, generating 24 trajectories per action.
  • Reference Trajectories: For training the DTW algorithms and the PACE policy, reference trajectories were derived from the collected demonstrations.

Key Insights and Ideas

  • The importance of local correlations: The Windowed-Pearson (WP) distance is crucial for capturing the nuances of human motion, especially in tasks with repetitive local patterns like screwing. Relying solely on absolute positions (Euclidean distance) can lead to suboptimal alignments.
  • Softness matters for real-time: Using a "softmin" operation in DTW (Soft-DTW) is essential for handling the continuous, potentially noisy, and variable nature of real-time human motion data.
  • Atomic action progress is key: Estimating progress at the level of individual actions, rather than just high-level tasks, allows for much finer-grained synchronization between humans and robots.
  • RL for proactive, adaptive assistance: Reinforcement Learning is a powerful tool for training robots to make proactive decisions about when and how to assist, based on real-time progress estimates.

Key Results

  • OS-DTW<sub>WP</sub> outperforms OS-DTW<sub>EU</sub> in complex motions: In tasks like screwing, where local patterns are important, OS-DTW<sub>WP</sub> showed significantly better phase estimation accuracy than OS-DTW<sub>EU</sub>.
  • OS-DTW<sub>EU</sub> is effective for consistent tasks: For tasks with stable absolute positions (like rail placing), OS-DTW<sub>EU</sub> performed well.
  • PACE framework reduces idle times: User studies showed that the PACE framework, using OS-DTW<sub>WP</sub>, reduced robot waiting times by nearly two-thirds compared to explicit query systems, without significantly increasing human waiting times.
  • Improved user experience: Participants reported higher levels of interaction fluency, understanding, and overall satisfaction with the PACE system. Subjective measures like NASA-TLX indicated no significant increase in mental or physical strain.

Limitations and Future Work

  • Single reference trajectory: The current PACE framework uses a single reference trajectory for OS-DTW<sub>WP</sub>, overlooking the fact that actions can have multiple valid execution modes.
  • Limited state information: The framework primarily relies on human motion data and could benefit from incorporating visual and contextual cues (e.g., object affordances, environmental state).
  • Fixed action sequences: The current PACE model assumes a fixed action sequence, whereas real-world scenarios often involve dynamic task orders, errors, or mid-action changes.
  • Future Work: The authors suggest extending the framework to handle multimodal sensing, develop more adaptable collaboration strategies for unstructured environments, and explore more robust ways to identify potential failures in progress estimation. They also propose generalizing the approach to other domains beyond industrial assembly.

Summary updated

Abstract

Real-time estimation of human action progress is critical for seamless human-robot collaboration yet remains underexplored. With this paper we propose the first real-time application of Open-end Soft-DTW (OS-DTW<sub>EU</sub>) and introduce OS-DTW<sub>WP</sub>, a novel DTW variant that integrates a Windowed-Pearson distance to effectively capture local correlations. This method is embedded in our Proactive Assistance through action-Completion Estimation (PACE) framework, which leverages reinforcement learning to synchronize robotic assistance with human actions by estimating action completion percentages. Experiments on a chair assembly task demonstrate OS-DTW<sub>WP</sub>'s superiority in capturing local motion patterns and OS-DTW<sub>EU</sub>'s efficacy in tasks presenting consistent absolute positions. Moreover we validate the PACE framework through user studies involving 12 participants, showing significant improvements in interaction fluency, reduced waiting times, and positive user feedback compared to traditional methods.
BibTeX
@inproceedings{lazzari2025real,
  author = {Davide De Lazzari and Matteo Terreran and Giulio Giacomuzzo and Siddarth Jain and Pietro Falco and Ruggero Carli and Stefano Ghidoni and Diego Romeres},
  title = {{Real-time human progress estimation with online dynamic time warping for collaborative robotics}},
  booktitle = {Frontiers in Robotics and AI},
  year = {2025},
  doi = {10.3389/frobt.2025.1623884},
}
Figure