An Evaluation Study of Intrinsic Motivation Techniques Applied to Reinforcement Learning over Hard Exploration Environments

Alain Andres, Esther Villar-Rodriguez, Javier Del Ser

TL;DR

This paper provides a comprehensive evaluation of intrinsic motivation (IM) techniques for reinforcement learning (RL) in challenging, sparse-reward environments. By standardizing experimental conditions and analyzing various design choices, the authors reveal that careful selection of IM strategies, weighting mechanisms, and network architectures is crucial for achieving robust performance, rather than solely relying on novel IM algorithms.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents often struggle in environments where rewards are infrequent (sparse rewards). This makes it difficult for agents to learn effective policies, as they receive little feedback on their actions. Intrinsic motivation (IM) techniques aim to solve this by providing agents with an internal "curiosity" reward, encouraging them to explore novel states. While many new IM algorithms have been proposed, their evaluations are often inconsistent. They use different base RL algorithms, neural network architectures, and parameter settings, making it hard to determine if improvements are due to the IM method itself or these other factors. This paper addresses this gap by conducting a fair and systematic evaluation of popular IM techniques, isolating the impact of various design choices on performance in hard exploration tasks.

Methods

The authors investigate three key research questions (RQs) related to IM implementation:

  • RQ1: Intrinsic Reward Coefficient (β) Weighting: They compare static, parametric decay, and adaptive decay strategies for weighting the intrinsic reward against the extrinsic (environmental) reward. This explores how the balance between exploration and exploitation changes over time.
  • RQ2: State Visitation Scaling: They examine the impact of scaling intrinsic rewards based on state visitation counts, specifically comparing episodic counts (rewarding visits to new states within an episode) versus first-time visitation (rewarding a state only the very first time it's encountered).
  • RQ3: Neural Network Architecture: They analyze how different neural network architectures for both the actor-critic agent and the IM modules affect performance and learning efficiency, comparing a "lightweight" design with a more "sophisticated/default" one.

All experiments were conducted using the Proximal Policy Optimization (PPO) RL algorithm as the base.

Datasets

The experiments were performed on procedurally generated RL tasks from the Minimalistic Gridworld Environment (MiniGrid). Specifically, they used three types of scenarios:

  • MultiRoomNXSY (MNXSY): Environments with varying numbers of rooms and sizes, requiring navigation through multiple doors.
  • KeyCorridorS3R3 (KS3R3): A task involving collecting a key to open a door.
  • 0bstructedMaze2Dlh (02D1h): A more complex maze requiring discovery of keys hidden under boxes.

These MiniGrid environments are characterized by sparse rewards and are designed to be computationally efficient, allowing for extensive benchmarking.

Key Insights and Ideas

  • Standardization is Key: The most significant takeaway is that fair comparisons of IM techniques are only possible when controlling for other variables like the base RL algorithm and network architecture. Many reported improvements might be due to these factors rather than the novel IM algorithm itself.
  • Adaptive Decay Wins: Adaptive decay strategies for the intrinsic reward coefficient (β) generally lead to better performance, especially in challenging scenarios, suggesting that dynamically adjusting the exploration-exploitation balance is more effective than static settings.
  • Episodic Scaling Helps: Using episodic state counts for scaling intrinsic rewards generally outperforms simpler "no scaling" or "first-time visit" strategies in hard exploration tasks, indicating the benefit of encouraging broader exploration within an episode.
  • Architecture Matters: The complexity of neural network architectures, particularly for the IM modules, significantly impacts performance. Reducing parameters can lead to failure in challenging tasks, while a well-chosen architecture can improve learning efficiency.
  • RIDE vs. ICM Nuances: While RIDE was initially proposed as an improvement over ICM, the study found that ICM often performs better or comparably when evaluated under the same conditions, especially when using simpler architectures. This highlights the importance of the evaluation setup.

Key Results

  • Weighting Strategies: Adaptive decay strategies (like ad and ad1000) consistently improved performance for COUNTS, RND, and RIDE across various scenarios compared to static weighting (_s). Parametric decay (_pd) also showed significant gains.
  • Scaling Strategies: Episodic scaling (_ep) and first-time visitation scaling (_1st) generally led to better results than no scaling (_noep) in more challenging environments (MN7S8, KS3R3, 02D1h). However, in easier environments (MN7S4, MN10S4), the differences were narrower, and _1st could even be counterproductive.
  • Network Architectures: Reducing the number of parameters in IM modules (_lw_im and _lw_tot) significantly deteriorated performance, leading to failure in challenging scenarios. Using a shared two-headed actor-critic network with lightweight IM modules (_lw_im) was found to be more robust than using entirely lightweight networks (_lw_tot).

Limitations and Future Work

  • Parameter Sensitivity: The authors acknowledge that adaptive decay strategies, while effective, require careful parameterization of their sliding windows. Similarly, parametric decay's effectiveness can be sensitive to its parametrization.
  • Environment Specificity: While MiniGrid is a good benchmark, the findings might not directly generalize to all types of RL environments, especially those with continuous state spaces or different reward structures.
  • Future Work: The authors suggest extending their analysis to more environments and algorithms to obtain more representative results. They also highlight the need for further research into why shared neural network architectures might be beneficial, whether it's due to parameter sharing, common feature spaces, or simply the adoption of different architectures like CNNs.

Summary updated

Abstract

In the last few years, the research activity around reinforcement learning tasks formulated over environments with sparse rewards has been especially notable. Among the numerous approaches proposed to deal with these hard exploration problems, intrinsic motivation mechanisms are arguably among the most studied alternatives to date. Advances reported in this area over time have tackled the exploration issue by proposing new algorithmic ideas to generate alternative mechanisms to measure the novelty. However, most efforts in this direction have overlooked the influence of different design choices and parameter settings that have also been introduced to improve the effect of the generated intrinsic bonus, forgetting the application of those choices to other intrinsic motivation techniques that may also benefit of them. Furthermore, some of those intrinsic methods are applied with different base reinforcement algorithms (e.g. PPO, IMPALA) and neural network architectures, being hard to fairly compare the provided results and the actual progress provided by each solution. The goal of this work is to stress on this crucial matter in reinforcement learning over hard exploration environments, exposing the variability and susceptibility of avant-garde intrinsic motivation techniques to diverse design factors. Ultimately, our experiments herein reported underscore the importance of a careful selection of these design aspects coupled with the exploration requirements of the environment and the task in question under the same setup, so that fair comparisons can be guaranteed.
BibTeX
@inproceedings{andres2022evaluation,
  author = {Alain Andres and Esther Villar-Rodriguez and Javier Del Ser},
  title = {{An Evaluation Study of Intrinsic Motivation Techniques Applied to Reinforcement Learning over Hard Exploration Environments}},
  booktitle = {CD-MAKE},
  year = {2022},
  url = {https://openreview.net/forum?id=MHdbGUd3pM},
}
Figure