Abstract
Learning fluid and synchronized human-robot interaction (HRI) remains a significant challenge due to the high-dimensional, multimodal, and temporally dependent nature of human movement. This paper introduces Multimodal Interactive Latent Dynamics (MILD), a framework that combines Variational Autoencoders (VAEs) with Hidden Semi-Markov Models (HSMMs) to learn reactive interaction trajectories. By modeling the joint distribution of human and robot movements in a structured latent space, MILD enables robots to generate synchronized, real-time responses to human partners. Experimental results on motion capture and RGB-D datasets demonstrate that MILD significantly outperforms existing state-of-the-art methods in both trajectory accuracy and qualitative synchronization.
Introduction
Effective Human-Robot Interaction (HRI) requires more than just pre-programmed routines; it requires a "dance" of mutual adaptation. Whether performing a handshake, a high-five, or a fistbump, the robot must perceive the human's intent and adjust its own movement speed, height, and timing in real-time.
Traditional Learning from Demonstration (LfD) methods often struggle with the complexity of these interactions. High-dimensional data from sensors like motion capture suits or depth cameras are difficult to process directly, and many deep learning models fail to capture the underlying temporal dynamics. Specifically, standard models often use "uninformed" priors—such as static Gaussian distributions—that do not account for the fact that a movement at time $t$ is inherently dependent on the state at $t-1$. This lack of temporal coherence often results in jerky, disconnected, or poorly timed robot motions.
Methodology
The MILD framework addresses these challenges by bridging the gap between deep representation learning and probabilistic sequence modeling through three core components.
1. Full Covariance Latent Representation
MILD utilizes a Variational Autoencoder (VAE) to compress high-dimensional skeleton data into a compact, 5-dimensional latent space. Unlike standard VAEs that assume independent dimensions (diagonal covariance), MILD predicts full posterior covariance matrices using Cholesky decomposition. This architectural choice allows the model to capture the intricate correlations between different joints—for example, how the movement of a shoulder relates to the trajectory of a wrist during a reaching motion.
2. HSMM as a Latent Prior
To solve the issue of temporal incoherence, MILD replaces the standard static prior with a Hidden Semi-Markov Model (HSMM). While standard Hidden Markov Models (HMMs) model transitions between states, HSMMs explicitly model the duration an agent remains within a specific state. This is crucial for HRI, as it allows the model to understand the "phases" of an interaction, such as the contact duration of a fistbump or the rhythmic cycle of a handshake.
3. Joint Interactive Modeling
MILD learns a joint distribution of both agents simultaneously. During the inference phase, the robot observes the human's trajectory and encodes it into the latent space. The HSMM then "fills in the blanks" for the robot’s corresponding latent trajectory based on the learned interactive dynamics. This latent plan is then decoded back into real-world joint angles for the robot to execute.
Experimental Setup and Datasets
The researchers evaluated MILD using two distinct datasets to test both high-fidelity and real-world performance:
- Butepage et al. Dataset: A high-fidelity dataset featuring Human-Human Interaction (HHI) and Human-Robot Interaction (HRI) captured via motion capture suits. It includes actions such as hand waving, handshaking, and complex "rocket" fistbumps.
- NuiSI (Nuitrack Skeleton Interaction Dataset): A new dataset collected using an Intel RealSense RGB-D camera. This dataset introduces real-world noise and lower-fidelity tracking to test the robustness of the framework during interactions like high-fives and "clap fists."
Results and Discussion
MILD demonstrated superior performance across both quantitative and qualitative metrics compared to the state-of-the-art baseline established by Butepage et al.
Quantitative Accuracy
In predicting the second agent's trajectory, MILD achieved a Mean Squared Error (MSE) ranging from 0.36 cm to 0.67 cm, a significant improvement over the baseline's 0.93 cm to 1.29 cm. When applied to physical hardware using an ABB YuMi robot, MILD’s joint angle predictions were remarkably precise, showing an error of only 0.038 rad for a "shake" action, compared to 0.371 rad for the baseline.
Qualitative Synchronization
Visual analysis of the generated trajectories indicates that MILD produces smoother and more reactive motions. By utilizing the HSMM prior, the latent space becomes a structured map of the interaction's phases, ensuring that the robot stays in sync with the human partner's timing rather than simply following a fixed path.
Zero-Shot Transfer
A key finding of the research is MILD's capacity for zero-shot transfer. The model can learn the fundamental dynamics of an interaction from human-human data and successfully execute those dynamics on a humanoid robot (such as Pepper or YuMi) without additional training, provided the generated skeleton is mapped to the robot's specific degrees of freedom.
Conclusion
MILD provides a robust framework for learning the temporal and spatial dependencies required for natural human-robot interaction. By integrating full covariance VAEs with HSMM priors, the system achieves a level of fluid synchronization previously difficult to attain in LfD.
Future work will focus on integrating automatic action recognition to allow robots to switch between different interaction models (e.g., moving from a wave to a handshake) autonomously. Additionally, the authors aim to incorporate Inverse Kinematics (IK) and force feedback to improve the precision of physical contact during touch-based interactions.
Summary updated