Anticipation in Human-Robot Cooperation: A Recurrent Neural Network Approach for Multiple Action Sequences Prediction

Paul Schydlo, Mirko Rakovic, Lorenzo Jamone, José Santos-Victor

TL;DR

This paper tackles the challenge of robots accurately predicting human actions in cooperative scenarios by developing a recurrent neural network that can anticipate multiple, variable-length action sequences. This is crucial for seamless human-robot collaboration, enabling robots to proactively plan and respond to human intent, leading to improved efficiency and safety.

What Problem Does This Paper Address?

Close collaboration between humans and robots is becoming increasingly vital in fields like advanced manufacturing and assistive applications. For this to be effective, robots need to go beyond simply recognizing current human actions; they must be able to predict future actions and understand human intent. This prediction is heavily reliant on non-verbal cues such as gaze and body posture. While previous research has shown promise in predicting single, fixed-length action sequences using neural networks, two key shortcomings remain: the inability to predict multiple possible future action sequences and the difficulty in handling variable-length action sequences. This paper aims to bridge this gap by developing a model capable of addressing both these limitations.

Methods

The authors propose a recurrent neural network (RNN) architecture, specifically an encoder-decoder model, to address the problem of predicting multiple, variable-length action sequences.

  • Feature Selection: Before prediction, they first quantify the importance of gaze and body pose features using a wrapper-based feature selection method. This involves training an RNN sequence-to-sequence model and evaluating its accuracy with and without specific features to gauge their contribution.
  • Encoder-Decoder Architecture: The core of the prediction model is an encoder-decoder RNN. The encoder condenses past non-verbal cue information (gaze and pose) into a fixed-length context vector using a Long Short-Term Memory (LSTM) cell. This context vector then serves as the initial state for the decoder.
  • Variable-Length Prediction: The decoder, also an LSTM, generates a sequence of discrete action predictions. At each step, it outputs a probability distribution over possible future actions.
  • Multiple Sequence Prediction: To handle multiple future action sequences, the decoder samples actions and feeds them back as input for the next decoding step. To manage the combinatorial explosion of possible sequences, they employ a beam search strategy. This method maintains a set of the top K most probable future action sequences at each decoding step, pruning less likely sequences to keep the search space manageable.

Datasets

The paper utilizes two datasets for evaluation:

  • ACTICIPATE Dataset: This dataset, featuring combined gaze and skeleton data, was used for evaluating the importance of gaze and body pose features. It includes human actor movements for six distinct actions (e.g., "Place Left," "Give Center") and was recorded with motion capture and eye-gaze tracking systems at 120Hz.
  • CAD120 Dataset: This dataset was used for evaluating the multiple action sequence prediction model. It contains human actor skeleton movements for sequences of actions like "pouring" and "eating," offering a more complex scenario with varying action orders. It provides joint position and orientation features at 5Hz.

Key Insights and Ideas

  • Gaze and Pose Synergy: Both gaze and body pose are crucial for predicting human intent. The study found that combining both cues allows for earlier prediction of human actions compared to using body pose alone.
  • Variable-Length Prediction Matters: Training the model on longer future action sequences leads to better prediction accuracy, suggesting that the decoder's ability to manage its internal state is key for handling variable-length predictions.
  • Beam Search for Tractability: The combinatorial complexity of predicting multiple action sequences is a significant challenge. Beam search effectively mitigates this by intelligently pruning the search space, allowing for the estimation of multiple probable futures.
  • Theoretical Link to Reward: The paper theoretically demonstrates how predicting multiple action sequences is essential for estimating the stochastic reward in human-robot cooperation scenarios, where human actions are inherently uncertain.

Key Results

  • Earlier Prediction with Gaze: A model trained on both gaze and body pose cues predicted the correct human action approximately 92 milliseconds earlier than a model trained solely on body pose cues.
  • Impact of Prediction Length: Prediction accuracy was positively correlated with the prediction length the model was trained on.
  • Beam Search Effectiveness: With a beam width of 11, the model could capture 100% of the outcome probability space for one-step ahead predictions. However, for two-step ahead predictions, the same beam width captured only about 75% of the probability space, highlighting the growing complexity.

Limitations and Future Work

  • Limitations: The paper acknowledges that while beam search helps manage complexity, increasing the number of predicted action sequences (beams) increases computational demand. The generalization error is also noted to be influenced by the context vector dimensionality, which acts as a regularizer.
  • Future Work: The authors suggest extending the model by exploring the connection between non-verbal cues and semantic features related to the context. They also propose composing the model with additional information using probabilistic methods and see this work as a strong foundation for implementing joint action scenarios on humanoid robotics platforms like the iCub.

Summary updated

Abstract

Close human-robot cooperation is a key enabler for new developments in advanced manufacturing and assistive applications. Close cooperation require robots that can predict human actions and intent, and understand human non-verbal cues. Recent approaches based on neural networks have led to encouraging results in the human action prediction problem both in continuous and discrete spaces. Our approach extends the research in this direction. Our contributions are three-fold. First, we validate the use of gaze and body pose cues as a means of predicting human action through a feature selection method. Next, we address two shortcomings of existing literature: predicting multiple and variable-length action sequences. This is achieved by introducing an encoder-decoder recurrent neural network topology in the discrete action prediction problem. In addition, we theoretically demonstrate the importance of predicting multiple action sequences as a means of estimating the stochastic reward in a human robot cooperation scenario. Finally, we show the ability to effectively train the prediction model on a action prediction dataset, involving human motion data, and explore the influence of the model's parameters on its performance. Source code repository: https://github.com/pschydlo/ActionAnticipation
BibTeX
@inproceedings{schydlo2018anticipation,
  author = {Paul Schydlo and Mirko Rakovic and Lorenzo Jamone and José Santos-Victor},
  title = {{Anticipation in Human-Robot Cooperation: A Recurrent Neural Network Approach for Multiple Action Sequences Prediction}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2018},
  doi = {10.1109/icra.2018.8460924},
}
Figure