Combining artificial curiosity and tutor guidance for environment exploration

Pierre Fournier, Olivier Sigaud, Mohamed ChétouaniHAL 2017

TL;DR

This paper proposes a novel reinforcement learning framework that combines artificial curiosity with tutor guidance (specifically, gaze-following) to improve an agent's ability to explore and learn tasks in new environments. By carefully balancing these two motivations, the agent achieves faster task discovery and more consistent performance, even in reward-free scenarios.

What Problem Does This Paper Address?

Developing artificial agents that can effectively explore and learn in novel environments is a significant challenge. Traditional reinforcement learning (RL) often relies on externally defined rewards, which are difficult to engineer for every new task or environment. While "artificial curiosity" has emerged as a way for agents to explore autonomously by seeking novelty or uncertainty, and "interactive machine learning" explores how agents can learn from human guidance, these approaches have often been studied in isolation. This paper addresses the gap of effectively combining these two powerful mechanisms: intrinsic motivation for exploration and learning from human social cues, specifically gaze, to accelerate and improve task learning.

Methods

The authors build upon the TEXPLORE-VANIR model-based reinforcement learning framework. Their key innovation is to augment the agent's reward function with two intrinsic motivations:

  • Artificial Curiosity: This component is based on the novelty-seeking drive from TEXPLORE-VANIR, which rewards the agent for encountering novel state-action pairs.
  • Gaze-Following Motivation: This is a new addition. The agent's state is extended to include its gaze position. A new reward signal, $J(s, \sigma_{obs})$, is introduced to encourage the agent's gaze to match the tutor's observed gaze. This is implemented through reward shaping, where the agent receives a bonus if its gaze aligns with the tutor's.

The agent's actions are then driven by a weighted combination of these intrinsic motivations and any external task rewards. The balance between curiosity and gaze-following is controlled by tunable parameters, $\mu$ (for gaze-following) and $\nu$ (for curiosity). The learning process involves building an internal model of the environment and using simulations to update Q-values, with the reward function incorporating both novelty and gaze-following incentives.

Datasets

Not discussed in the paper. The experiments were conducted in a simulated pick-and-place task environment.

Key Insights and Ideas

  • Synergy of Curiosity and Gaze: Combining artificial curiosity with tutor gaze-following is significantly more effective than using either drive in isolation. This suggests that social cues can act as a powerful guide for autonomous exploration.
  • Gaze as a Guidance Signal: The tutor's gaze, even in a simplified form, provides a valuable signal that helps the agent focus its exploration on relevant parts of the environment.
  • Tunable Balance is Crucial: The performance of the agent is highly dependent on the correct weighting of curiosity and gaze-following motivations. An optimal balance leads to faster task discovery and more consistent execution.
  • Reward-Free Learning Potential: The combined approach shows promise for learning tasks even without explicit external rewards, relying solely on intrinsic motivations and tutor guidance. This is a significant step towards more general-purpose learning agents.
  • Curiosity Fades, Gaze Persists: At the beginning of exploration, curiosity plays a dominant role. As the agent discovers more about the environment and the task, the influence of external rewards (if present) or the tutor's gaze becomes more prominent.

Key Results

  • Improved Task Discovery: Agents with a combination of curiosity and gaze-following ($\mu=1$) consistently outperformed agents relying solely on curiosity ($\mu=0$) in discovering the pick-and-place task.
  • Optimal Parameter Setting: The best performance was achieved with $\mu=1$ and $\nu=10$, indicating a strong emphasis on curiosity when guided by gaze.
  • Consistent Performance: In reward-free environments, the combined approach allowed the agent to achieve the task regularly, whereas curiosity alone led to task completion only on rare occasions.
  • Reduced Trial Uncertainty: The combination of motivations reduced the variability in performance across trials, leading to more reliable task execution.

Limitations and Future Work

  • Oversimplified Tutor Gaze: The current tutor gaze is provided directly, not detected. Future work should involve the agent learning to detect the tutor's gaze, which would make the guidance signal more realistic and challenging.
  • Learning Gaze Following: The paper suggests that the agent should learn gaze-following rather than being explicitly programmed to do so.
  • Integrating Goal Discovery and Intent Inference: Future research could integrate automatic goal discovery and tutor intention inference to enable more sophisticated teaching scenarios, potentially leading to subgoal-based rewards and competence-based motivations.
  • Scalability and Sparse Rewards: The authors aim to evaluate the algorithm's scalability in larger state spaces and with more sparse rewards.

Summary updated

Abstract

International audience
BibTeX
@inproceedings{fournier2017combining,
  author = {Pierre Fournier and Olivier Sigaud and Mohamed Chétouani},
  title = {{Combining artificial curiosity and tutor guidance for environment exploration}},
  booktitle = {HAL},
  year = {2017},
}
Figure