Interactive Reinforcement Learning for Table Balancing Robot

Haein Jeon, Yewon Kim, Bo-Yeong KangACL 2021

TL;DR

This paper introduces an interactive deep reinforcement learning (RL) system that enables robots to learn cooperative tasks, like table balancing, through human voice feedback. By integrating Automatic Speech Recognition (ASR) and sentiment analysis, the system converts natural language feedback into reward signals, significantly improving learning efficiency and performance compared to traditional RL methods.

What Problem Does This Paper Address?

As robots become more integrated into daily life, there's a growing need for intuitive and efficient robot training methods. Traditional Reinforcement Learning (RL) often requires carefully designed reward functions or extensive trial-and-error, which can be time-consuming and difficult for non-experts. Existing Interactive Reinforcement Learning (IARL) approaches, while improving learning, often rely on limited feedback types or require specific hardware interfaces (like mice or remote controls), hindering natural human-robot interaction. The paper addresses this by proposing a system that leverages natural language voice feedback for robot training, making the process more accessible and intuitive.

Methods

The core of the proposed system is a Deep Q-Network (DQN) that learns to perform a cooperative table balancing task. The novelty lies in how human feedback is incorporated:

  • Voice Feedback Integration: Humans provide feedback via voice, which is processed by:
  • Automatic Speech Recognition (ASR): Transcribes spoken words into text. The paper uses Google Cloud Speech-to-Text.
  • Sentiment Analysis: Analyzes the transcribed text to determine the positivity or negativity of the feedback, converting it into a numerical reward value (between -1 and 1). Google Natural Language API is used for this.
  • Reward Shaping: The numerical feedback value is added to the environmental reward, directly influencing the DQN's learning process. This allows the robot to learn optimal policies faster by incorporating human judgment.
  • Two Feedback Models:
    • Consecutive Voice Feedback (Consec-VF): Provides a concentrated burst of feedback early in training to establish an initial learning direction.
  • Periodical Voice Feedback (Prdc-VF): Delivers feedback intermittently throughout the entire training process to ensure consistent reflection of human input.

Datasets

The paper constructed a voice feedback phrase dataset for experiments. This dataset consists of 100 feedback phrases (50 positive, 50 negative) extracted from sentiment lexicons (Hu and Liu, 2004; Nielsen, 2011) and "Classroom English" (Hong and Sohn, 2013). Examples of phrases and their converted numerical values are provided in Table 2.

Key Insights and Ideas

  • Natural Language as a Powerful Feedback Channel: Voice feedback, when processed effectively, can significantly enhance robot learning, making the interaction more natural and less cumbersome than hardware-based methods.
  • Sentiment Analysis Bridges the Gap: Converting nuanced human language into quantifiable reward signals is crucial for integrating voice feedback into RL algorithms.
  • Timing of Feedback Matters: The study highlights that the way feedback is delivered impacts learning. Concentrated early feedback (Consec-VF) appears more effective for achieving optimal policy convergence than intermittent feedback (Prdc-VF).
  • Robustness to Imperfect Recognition: Even with imperfect ASR and sentiment analysis (average recognition rates of 86% and 96% respectively), interactive voice feedback still demonstrably improved the agent's learning performance.

Key Results

  • The proposed interactive deep RL model, particularly the Consec-VF model, achieved significantly higher optimal policy convergence rates compared to the baseline DQN without voice feedback.
  • With the Adam optimizer, the Consec-VF model reached an optimal policy convergence rate of 96%, outperforming the baseline's 73%.
  • The Consec-VF model consistently showed improved learning performance across various optimizers (SGD, Adam, Adagrad, Adadelta) compared to the baseline.
  • The Prdc-VF model showed lower performance, with training loss spikes, suggesting that intermittent feedback might interfere with stable convergence.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that imperfect ASR and sentiment analysis can lead to erroneous reward signals, though experiments showed this did not prevent learning.
  • The Prdc-VF model's lower performance suggests that the timing and frequency of feedback are critical and may require further optimization.
  • Future Work:
    • Incorporating multimodal feedback by utilizing other robot sensors alongside voice.
  • Deepening model optimization techniques to improve learning performance in varying settings.
  • Enabling the robot to learn when to use feedback and when to discard it, or to incorporate text semantics for more nuanced guidance.

Summary updated

BibTeX
@inproceedings{jeon2021interactive,
  author = {Haein Jeon and Yewon Kim and Bo-Yeong Kang},
  title = {{Interactive Reinforcement Learning for Table Balancing Robot}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure