TL;DR
This paper provides a comprehensive evaluation of intrinsic motivation (IM) techniques for reinforcement learning (RL) in challenging, sparse-reward environments. By standardizing experimental conditions and analyzing various design choices, the authors reveal that careful selection of IM strategies, weighting mechanisms, and network architectures is crucial for achieving robust performance, rather than solely relying on novel IM algorithms.
What Problem Does This Paper Address?
Reinforcement learning (RL) agents often struggle in environments where rewards are infrequent (sparse rewards). This makes it difficult for agents to learn effective policies, as they receive little feedback on their actions. Intrinsic motivation (IM) techniques aim to solve this by providing agents with an internal "curiosity" reward, encouraging them to explore novel states. While many new IM algorithms have been proposed, their evaluations are often inconsistent. They use different base RL algorithms, neural network architectures, and parameter settings, making it hard to determine if improvements are due to the IM method itself or these other factors. This paper addresses this gap by conducting a fair and systematic evaluation of popular IM techniques, isolating the impact of various design choices on performance in hard exploration tasks.
Methods
The authors investigate three key research questions (RQs) related to IM implementation:
- RQ1: Intrinsic Reward Coefficient (β) Weighting: They compare static, parametric decay, and adaptive decay strategies for weighting the intrinsic reward against the extrinsic (environmental) reward. This explores how the balance between exploration and exploitation changes over time.
- RQ2: State Visitation Scaling: They examine the impact of scaling intrinsic rewards based on state visitation counts, specifically comparing episodic counts (rewarding visits to new states within an episode) versus first-time visitation (rewarding a state only the very first time it's encountered).
- RQ3: Neural Network Architecture: They analyze how different neural network architectures for both the actor-critic agent and the IM modules affect performance and learning efficiency, comparing a "lightweight" design with a more "sophisticated/default" one.
All experiments were conducted using the Proximal Policy Optimization (PPO) RL algorithm as the base.
Datasets
The experiments were performed on procedurally generated RL tasks from the Minimalistic Gridworld Environment (MiniGrid). Specifically, they used three types of scenarios:
- MultiRoomNXSY (MNXSY): Environments with varying numbers of rooms and sizes, requiring navigation through multiple doors.
- KeyCorridorS3R3 (KS3R3): A task involving collecting a key to open a door.
- 0bstructedMaze2Dlh (02D1h): A more complex maze requiring discovery of keys hidden under boxes.
These MiniGrid environments are characterized by sparse rewards and are designed to be computationally efficient, allowing for extensive benchmarking.
Key Insights and Ideas
- Standardization is Key: The most significant takeaway is that fair comparisons of IM techniques are only possible when controlling for other variables like the base RL algorithm and network architecture. Many reported improvements might be due to these factors rather than the novel IM algorithm itself.
- Adaptive Decay Wins: Adaptive decay strategies for the intrinsic reward coefficient (β) generally lead to better performance, especially in challenging scenarios, suggesting that dynamically adjusting the exploration-exploitation balance is more effective than static settings.
- Episodic Scaling Helps: Using episodic state counts for scaling intrinsic rewards generally outperforms simpler "no scaling" or "first-time visit" strategies in hard exploration tasks, indicating the benefit of encouraging broader exploration within an episode.
- Architecture Matters: The complexity of neural network architectures, particularly for the IM modules, significantly impacts performance. Reducing parameters can lead to failure in challenging tasks, while a well-chosen architecture can improve learning efficiency.
- RIDE vs. ICM Nuances: While RIDE was initially proposed as an improvement over ICM, the study found that ICM often performs better or comparably when evaluated under the same conditions, especially when using simpler architectures. This highlights the importance of the evaluation setup.
Key Results
- Weighting Strategies: Adaptive decay strategies (like
adandad1000) consistently improved performance for COUNTS, RND, and RIDE across various scenarios compared to static weighting (_s). Parametric decay (_pd) also showed significant gains. - Scaling Strategies: Episodic scaling (
_ep) and first-time visitation scaling (_1st) generally led to better results than no scaling (_noep) in more challenging environments (MN7S8, KS3R3, 02D1h). However, in easier environments (MN7S4, MN10S4), the differences were narrower, and_1stcould even be counterproductive. - Network Architectures: Reducing the number of parameters in IM modules (
_lw_imand_lw_tot) significantly deteriorated performance, leading to failure in challenging scenarios. Using a shared two-headed actor-critic network with lightweight IM modules (_lw_im) was found to be more robust than using entirely lightweight networks (_lw_tot).
Limitations and Future Work
- Parameter Sensitivity: The authors acknowledge that adaptive decay strategies, while effective, require careful parameterization of their sliding windows. Similarly, parametric decay's effectiveness can be sensitive to its parametrization.
- Environment Specificity: While MiniGrid is a good benchmark, the findings might not directly generalize to all types of RL environments, especially those with continuous state spaces or different reward structures.
- Future Work: The authors suggest extending their analysis to more environments and algorithms to obtain more representative results. They also highlight the need for further research into why shared neural network architectures might be beneficial, whether it's due to parameter sharing, common feature spaces, or simply the adoption of different architectures like CNNs.
Summary updated