TL;DR
This paper introduces "CuRe," a novel curiosity-driven exploration strategy for vision-based reinforcement learning (RL). By training a separate "curious" policy to maximize the error of a state representation learning (SRL) module, CuRe encourages the agent to explore "problematic" states, leading to more diverse and informative training data. This approach significantly improves sample efficiency, training stability, and overall performance across various challenging RL tasks.
What Problem Does This Paper Address?
Vision-based reinforcement learning (RL) aims to enable agents to learn complex tasks directly from raw image observations. While integrating State Representation Learning (SRL) techniques has shown promise in improving sample efficiency, a key challenge remains: the diversity of the data collected during RL training. Standard RL algorithms, driven by maximizing rewards, tend to collect redundant and similar observations, leading to a "comfort zone" for the SRL module. This can result in learned representations that are not robust or generalizable, especially in environments with sparse rewards or complex dynamics. The paper addresses this by proposing a method to actively seek out and learn from states that are "difficult" for the SRL module to represent accurately, thereby enriching the training data and improving the overall learning process.
Methods
The core innovation is the "Curiosity-driven Representations" (CuRe) approach. Instead of solely relying on task rewards, CuRe introduces a secondary objective: maximizing the error of the SRL module. This is achieved by training a separate, "curious" policy alongside the main task policy.
Here's a breakdown:
- SRL Module: The paper considers two popular SRL methods: Regularized Autoencoders (RAEs) and contrastive learning (specifically, InfoNCE loss with instance discrimination). These modules aim to compress observations into meaningful latent representations.
- Curious Policy: A dedicated policy,
Acure, is trained to maximize the SRL error. This policy is incentivized to explore states that the SRL module finds difficult to encode, effectively seeking out "visual discomfort." - Dual-Policy Architecture: The system employs two distinct policies: the main
Ataskpolicy that optimizes for the task reward, and theAcurepolicy for exploration. This separation ensures the task policy remains focused on its objective while the curious policy drives exploration. - Integration: The curious policy's actions are sampled with a probability
pc. When sampled,Acure's actions are used to collect data, which then contributes to the replay buffer. The SRL error generated by these collected states serves as an intrinsic reward for trainingAcure. - Off-Policy RL: The method is designed to work with off-policy RL algorithms (like SAC), allowing for efficient reuse of collected data.
This approach differs from traditional curiosity methods that often rely on predicting future states using dynamics models. CuRe directly leverages the SRL error, making it more directly tied to the quality of the learned representations and avoiding the need for separate dynamics models.
Datasets
The experiments were conducted on six continuous control tasks from the Deepmind Control Suite [37]: reacher_easy, cartpole_swingup, ball_in_cup, finger_spin, finger-turn, and reacher_hard. These tasks were chosen to represent a range of challenges, including contact dynamics and sparse rewards.
Key Insights and Ideas
- "Visual Discomfort" as a Learning Signal: The paper demonstrates that actively seeking out states that are "difficult" for the SRL module (i.e., have high SRL error) is a powerful way to improve representation learning and, consequently, RL performance. This is a shift from traditional exploration that focuses on visiting novel states or maximizing task rewards.
- SRL Error as Intrinsic Reward: Directly using the SRL error as an intrinsic reward for a separate exploration policy is an effective way to guide the agent towards more informative states. This bypasses the need for complex dynamics models often used in curiosity-driven exploration.
- Synergistic Interplay: The curious policy and the SRL module form a symbiotic relationship. The SRL error guides the curious policy's exploration, and the diverse data collected by the curious policy, in turn, helps the SRL module learn better representations. This is likened to an adversarial setting, similar to GANs.
- Improved Generalization: By encouraging exploration of a wider range of states, including those that might be rare or difficult to reach with a purely reward-driven policy, CuRe likely leads to more robust and generalizable state representations.
Key Results
- Significantly Higher SRL Error Visitation: The curious policy consistently visited states with SRL errors that were approximately two orders of magnitude higher than those visited by random policies or standard SAC-AE policies. This directly validates the core hypothesis of seeking "discomfort."
- Improved Performance Across Tasks: In all tested environments, the CuRe-enhanced algorithms (
sac_ae+cureandcurl+cure) outperformed their baseline counterparts. - Enhanced Sample Efficiency and Stability: CuRe stabilized training and reduced reward variance, particularly in more complex tasks like
reacher_hardandfinger_turn, where baselines struggled to achieve high rewards. For simpler tasks, it accelerated convergence. - Agnostic to SRL Method: The benefits of CuRe were observed when integrated with both RAEs and contrastive learning methods, demonstrating its flexibility.
- Pretraining Benefits: Pretraining the SRL encoder using data collected by a CuRe-based policy (without task reward) led to better downstream RL performance than random pretraining or no pretraining, especially in challenging environments.
Limitations and Future Work
- Computational Cost: While the paper doesn't explicitly detail computational overhead, training a separate policy and computing SRL errors at each step likely adds to the overall training time.
- Hyperparameter Tuning: The effectiveness of CuRe might depend on the choice of hyperparameters, such as
pc(curious exploration probability) andβ(weight of the SRL error). - Future Work: The authors suggest exploring the transfer learning capabilities of their architecture, scaling it to multi-modal tasks, and conducting real-world experiments. They also note that studying the effect of the environment's initial state distribution mechanism on performance is an interesting avenue for future research.
Summary updated