Show, attend and interact: Perceivable human-robot social interaction through neural attention Q-network

Ahmed Hussain Qureshi, Yutaka Nakamura, Yuichiro Yoshikawa, Hiroshi Ishiguro

TL;DR

This paper introduces the Multimodal Deep Attention Recurrent Q-Network (MDARQN), a novel reinforcement learning architecture that enables robots to learn perceivable and socially acceptable human-robot interaction skills. By integrating recurrent attention mechanisms into a deep Q-network, the robot can learn to respond to complex human behaviors in real-world, uncontrolled environments after just 14 days of interaction.

What Problem Does This Paper Address?

Creating robots that can engage in natural and effective social interactions with humans is a significant challenge. A key aspect of this is enabling robots to understand and respond to complex human intentions, which are often conveyed through subtle cues like body language, gaze, and movement. Traditional programming approaches struggle to capture this complexity. While previous work has explored deep reinforcement learning for robots, a crucial missing piece has been the robot's ability to demonstrate its understanding and intentions in a perceivable way, which is vital for building trust and encouraging interaction. This paper aims to bridge this gap by developing a robot that not only learns to interact but also makes its interaction choices understandable to humans.

Methods

The core of this work is the Multimodal Deep Attention Recurrent Q-Network (MDARQN). This architecture builds upon the Multimodal Deep Q-Network (MDQN) by incorporating a recurrent attention model (RAM).

Here's a breakdown of the key components:

  • Multimodal Input: The system processes both grayscale and depth images from the robot's sensors.
  • Dual-Stream Convolutional Neural Networks (CNNs): Two parallel CNN streams independently process the grayscale and depth image data.
  • Recurrent Attention Model (RAM): This is the novel addition. RAM allows the Q-network to focus on specific regions of the input image that are most relevant for decision-making. This not only improves efficiency by reducing computational load but also provides insight into where the robot is "looking" when making a decision.
  • Long Short-Term Memory (LSTM) Network: An LSTM is used within each stream to process sequential information from the attention mechanism and the CNN features, capturing temporal dependencies in the interaction.
  • End-to-End Reinforcement Learning: The entire system is trained using a hit-and-trial method in an uncontrolled real-world environment. The robot learns by interacting with people and receiving rewards based on its actions.
  • Perceivable Actions: The attention mechanism's output is used to guide the robot's physical actions, making its focus and intentions more apparent to humans. For example, the robot might steer its gaze towards a specific person it's attending to.

Datasets

The paper describes training and evaluation conducted over 14 days of real-world interaction with people in public spaces (cafeteria, common rooms, etc.). The robot collected interaction experiences, resulting in approximately 111,504 grayscale and depth frames. Data augmentation (random cropping and mirroring) doubled this dataset. The paper also mentions releasing the depth dataset collected during the experiment.

Key Insights and Ideas

  • Attention for Perceivability: The most significant insight is that adding an attention mechanism to a robot's decision-making process makes its actions more perceivable and thus more socially acceptable. Humans are more willing to interact with robots that signal their focus.
  • Learning from Uncontrolled Environments: The success of learning in a truly uncontrolled, real-world setting for 14 days is a major achievement, demonstrating the robustness of the approach.
  • Human Social Norms: The robot learned to infer human intentions and respond in ways that align with social norms, such as waiting when someone is taking a picture or approaching someone who is looking at it.
  • Dynamic Attention Steering: The attention mechanism not only informs the robot's decision but also actively steers its physical attention (e.g., head movement) to make its interaction more engaging and understandable.
  • Reward Function Sensitivity: The choice of reward function significantly impacts the robot's behavior, with a moderate penalty for unsuccessful handshakes leading to more socially acceptable outcomes than extreme penalties.

Key Results

  • Human-like Social Interaction Skills: After 14 days of training, the robot exhibited human-like social interaction skills, including greeting, looking towards humans, waving, and handshaking.
  • Improved Handshake Ratio: The MDARQN model (specifically MDARQN(Aug) trained on augmented data) achieved a handshake ratio of 0.74, significantly higher than the MDQN model (0.48), indicating greater success in attracting people for interaction.
  • High Accuracy: The MDARQN(Aug) model achieved an accuracy of 95.2% on a test dataset, with a true positive rate of 95.3% and a false positive rate of 3.15%.
  • Socially Acceptable Decisions: The robot learned to make decisions that were deemed socially acceptable by human evaluators in diverse scenarios.
  • Effect of Attention: The attention mechanism was shown to be crucial for perceivable HRSI, leading to increased willingness to interact.

Limitations and Future Work

  • Training Data Requirements: The paper notes that attention-driven neural networks might require more training data to achieve performance comparable to non-attention models.
  • Limited Interaction Memory: The robot currently doesn't remember past interactions, leading to potentially awkward repeated attempts at handshakes.
  • Future Work: The authors plan to:
  • Explore different fusion strategies for multimodal learning.
  • Augment the network with differentiable working memories to enable long-term HRI.
  • Investigate the impact of different fusion strategies on multimodal learning.

Summary updated

Abstract

For a safe, natural and effective human-robot social interaction, it is essential to develop a system that allows a robot to demonstrate the perceivable responsive behaviors to complex human behaviors. We introduce the Multimodal Deep Attention Recurrent Q-Network using which the robot exhibits human-like social interaction skills after 14 days of interacting with people in an uncontrolled real world. Each and every day during the 14 days, the system gathered robot interaction experiences with people through a hit-and-trial method and then trained the MDARQN on these experiences using end-to-end reinforcement learning approach. The results of interaction based learning indicate that the robot has learned to respond to complex human behaviors in a perceivable and socially acceptable manner.
BibTeX
@inproceedings{qureshi2017show,
  author = {Ahmed Hussain Qureshi and Yutaka Nakamura and Yuichiro Yoshikawa and Hiroshi Ishiguro},
  title = {{Show, attend and interact: Perceivable human-robot social interaction through neural attention Q-network}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2017},
  url = {https://openreview.net/forum?id=iiTsDsyR60n},
}
Figure