TL;DR
This paper tackles the challenge of sparse rewards in reinforcement learning (RL) by introducing a novel intrinsic exploration method called FaSo. FaSo combines "fast" and "slow" curiosity signals to encourage both local and global exploration, enabling agents to discover more complex strategies in challenging environments like Atari games.
What Problem Does This Paper Address?
Reinforcement learning agents often struggle in environments where rewards are infrequent or delayed. This makes it difficult for them to learn effective policies because they don't receive enough feedback to guide their exploration. While intrinsic motivation, or "curiosity," has emerged as a promising solution by allowing agents to generate their own rewards for exploring novel states, current methods tend to focus on local exploration. This can lead to agents getting stuck in local optima, failing to discover broader, more rewarding strategies. The paper aims to bridge this gap by developing an intrinsic exploration method that can effectively explore the environment on a higher, more global level.
Methods
The core of the proposed method, FaSo (Fast and Slow), is an intrinsic reward formulation that decomposes the curiosity signal into two components: a "fast" reward and a "slow" reward. Both rewards are based on the agent's ability to reconstruct an observation given its context, measured by the prediction error.
- Context-Driven Curiosity: The fundamental idea is that an agent is curious about states it cannot easily predict. The "context" of an observation is created by intentionally corrupting or masking parts of it. The agent uses a reconstructor network to try and recreate the original observation from this corrupted context. The larger the reconstruction error, the higher the intrinsic reward.
- Fast Reward: This signal is generated using a context that is only slightly corrupted (e.g., through minor downsampling and upscaling). This encourages the agent to explore novel states that are slightly different from what it has seen before, promoting local exploration.
- Slow Reward: This signal uses a more heavily corrupted context (e.g., more significant downsampling). When contexts are heavily corrupted, similar states in the environment might have very similar contexts. By seeking states that are difficult to reconstruct from these more ambiguous contexts, the agent is incentivized to explore larger, more diverse regions of the state space, leading to global exploration.
- Combined Reward: The final intrinsic reward is a weighted sum of the fast and slow rewards, combined with the extrinsic task reward.
Datasets
The paper evaluates its method on two challenging Atari 2600 games from the Arcade Learning Environment (ALE): Montezuma's Revenge and Private Eye. These games are known for their sparse rewards and require significant exploration to achieve high scores.
Key Insights and Ideas
- High-Level Exploration is Crucial: The paper strongly suggests that for sparse reward environments, simply exploring local novelty isn't enough. Agents need a mechanism to explore on a broader, more strategic level to discover complex goals.
- Decomposing Curiosity: Splitting the intrinsic reward into "fast" and "slow" components is a key innovation. This allows the agent to balance immediate exploration of new states with the long-term goal of understanding the overall environment structure.
- Context Corruption as a Novelty Signal: Using image downsampling and upscaling to create corrupted contexts is an elegant way to generate the necessary input for the reconstructor network, directly linking visual perception to exploration strategy.
- Overcoming Local Optima: The "slow" reward component is specifically designed to pull agents out of local optima by incentivizing exploration of less familiar, larger state spaces.
Key Results
The FaSo method demonstrated superior performance compared to baseline methods on Montezuma's Revenge and Private Eye.
- On Montezuma's Revenge, FaSo achieved a maximum mean score of 8,951, outperforming PPO+ICM (329) and RND (8,152).
- On Private Eye, FaSo achieved a maximum mean score of 11,150, significantly exceeding PPO+ICM (485) and RND (8,666).
- The paper suggests that FaSo's ability to explore a larger number of rooms in Montezuma's Revenge, compared to RND, highlights the effectiveness of its high-level exploration strategy.
Limitations and Future Work
- Dynamic Weighting: The authors suggest that dynamically weighting the fast and slow rewards during training could further improve performance.
- Generalization: They plan to test the approach on different environments and investigate its performance in scenarios where the task reward is entirely absent.
- Context Creation: While image downsampling is used, exploring other methods for context creation might yield further improvements.
Summary updated