Intrinsically motivated reinforcement learning for human-robot interaction in the real-world

Ahmed Hussain Qureshi, Yutaka Nakamura, Yuichiro Yoshikawa, Hiroshi Ishiguro

TL;DR

This paper introduces an intrinsically motivated reinforcement learning framework for robots to learn social interaction skills in real-world, uncontrolled environments. By using an action-conditional predictive model to generate intrinsic rewards, the robot successfully acquired human-like social behaviors and made more appropriate decisions than a robot trained with direct task rewards.

What Problem Does This Paper Address?

Social human-robot interaction (HRI) is a growing field, but a significant challenge is teaching robots complex human social skills. Directly instructing robots on these nuanced behaviors is difficult due to their inherent complexity and the vast diversity of human interactions. Traditional reinforcement learning (RL) methods often rely on external, task-specific rewards, which are scarce in HRI scenarios. This scarcity makes it hard for robots to learn and adapt to natural, uncontrolled human interactions. The authors aim to overcome this by enabling robots to learn social skills autonomously through intrinsic motivation, driven by their own predictive models of human behavior.

Methods

The core of the proposed method is an intrinsically motivated deep reinforcement learning framework. This framework consists of two main components:

  • Policy Network (Qnet): This is a deep Q-network that approximates the action-value function. It takes the robot's state (a sequence of images) as input and outputs Q-values for each possible action. Qnet is trained to maximize the expected return, similar to standard DQN.
  • Action-Conditional Prediction Network (Pnet): This network acts as a multi-label classifier. It takes the current state and action as input and predicts the probability of future social events (handshake, eye contact, smile). Pnet is trained to minimize the prediction error between its predictions and the actual observed events.

The intrinsic motivation comes from Pnet. The prediction error of Pnet is used to generate an internal reward signal for Qnet. This means Qnet is rewarded for taking actions that lead to more predictable outcomes according to Pnet. The authors hypothesize that by learning to predict human behavior accurately, the robot will implicitly learn socially appropriate actions.

The training procedure involves two phases:

  1. Data Generation Phase: The robot interacts with the environment using an ε-greedy policy, collecting experience tuples (state, action, event, next state).
  2. Learning Phase: The collected experiences are stored in a replay memory. Both Pnet and Qnet are then trained using mini-batches from this memory. Pnet is trained using Binary Cross Entropy loss, while Qnet is trained using a standard Q-learning objective with the internally generated rewards.

Datasets

The paper utilizes a dataset of human-robot interaction experiences collected in [13]. This dataset consists of 14 days of real-world interactions in public places, totaling 13,938 interaction experiences. These experiences include grayscale and depth images. The authors also collected a new test dataset of 560 interaction experiences gathered by the robot following a random policy for evaluation.

Key Insights and Ideas

  • Intrinsic Motivation as a Driver for Social Skills: The central idea is that by learning to predict human behavior, a robot can implicitly acquire social skills without explicit human-like reward signals. This is a powerful concept for learning in reward-sparse environments.
  • Predictive Model for Social Cues: Pnet's ability to predict key social events (handshake, eye contact, smile) serves as a proxy for understanding human intentions and the success of an interaction.
  • Human Behavior is Predictable: The success of the predictive model suggests that human social behavior, even in complex, uncontrolled environments, exhibits a degree of predictability that can be learned.
  • Intrinsic Rewards Lead to More Human-like Behavior: The results strongly suggest that intrinsically motivated learning leads to more human-like decisions compared to direct reward-based learning, even when the direct reward was for a specific task like successful handshakes.
  • Reward Function Design Matters: The study highlights how different reward functions can significantly impact the robot's learned behavior, with lower penalties for incorrect predictions leading to more socially acceptable outcomes.

Key Results

  • Pnet Performance: Pnet achieved an overall accuracy of 88.9% in predicting all three events (handshake, smile, eye contact) on the test dataset. Individual event prediction accuracies were also high (e.g., Handshake: 97.9%, Smile: 91.9%, Eye Contact: 92.3%).
  • Qnet Performance:
    • Improved Predictability: Qnet significantly improved the total reward accumulated compared to a random policy (66 vs. 17), demonstrating its ability to learn actions that lead to more predictable outcomes.
  • Human-like Behavior: Evaluation by human judges indicated that Qnet's learned behavior was more similar to human social behavior than a baseline model trained with direct rewards. This was quantified by F1-scores, where Qnetim (intrinsically motivated) outperformed Qneta (direct reward).
  • Action Appropriateness: The robot learned to perform appropriate actions based on inferred human intentions, such as waiting when no one is around, looking towards approaching people, waving when people are at a distance, and initiating a handshake when fully engaged.
  • Reward Function Impact: The "neutral" reward function, with a penalty of -0.1 for incorrect predictions, resulted in the most socially acceptable behavior. Penalties of -0.1 or lower were found to be more effective than stricter penalties.

Limitations and Future Work

  • Limited Action Space: The current system is restricted to four actions (wait, look towards human, wave hand, handshake). This limited space can lead to repetitive behavior.
  • Lack of Memory: The robot is oblivious to past interactions with a specific partner, which can result in repetitive actions and a lack of personalized interaction.
  • Future Work:
    • Augmenting with Memory: Incorporating a memory mechanism (e.g., face recognition) to track individuals and avoid repetitive behaviors.
  • Increasing Action Space: Expanding the set of available actions to allow for more nuanced interactions.
  • Mobile Robot Platform: Making the robot mobile to actively approach people for more natural and effective social interactions.
  • Advanced Policy Optimization: Investigating more advanced policy optimization methods for Qnet.
  • Framework for Active Object Recognition: Proposing a framework similar to active object recognition for camera positioning to enhance interaction.

Summary updated

Abstract

For a natural social human-robot interaction, it is essential for a robot to learn the human-like social skills. However, learning such skills is notoriously hard due to the limited availability of direct instructions from people to teach a robot. In this paper, we propose an intrinsically motivated reinforcement learning framework in which an agent gets the intrinsic motivation-based rewards through the action-conditional predictive model. By using the proposed method, the robot learned the social skills from the human-robot interaction experiences gathered in the real uncontrolled environments. The results indicate that the robot not only acquired human-like social skills but also took more human-like decisions, on a test dataset, than a robot which received direct rewards for the task achievement.
BibTeX
@inproceedings{qureshi2018intrinsically,
  author = {Ahmed Hussain Qureshi and Yutaka Nakamura and Yuichiro Yoshikawa and Hiroshi Ishiguro},
  title = {{Intrinsically motivated reinforcement learning for human-robot interaction in the real-world}},
  booktitle = {Neural Networks},
  year = {2018},
  url = {https://openreview.net/forum?id=3PQSQsachq8},
}
Figure