The Challenge of Exploration in High-Dimensional RL
Exploration remains a fundamental hurdle in Reinforcement Learning (RL), particularly when agents must learn from high-dimensional observations like raw pixels. In environments with sparse rewards, agents cannot rely solely on the environment's feedback to learn; they must be intrinsically motivated to explore.
Traditional exploration methods often rely on "intrinsic rewards" based on prediction errors—rewarding the agent for visiting states where its internal dynamics model fails. A more principled and robust approach is state entropy maximization, which encourages the agent to visit all states as uniformly as possible.
However, calculating state entropy in high-dimensional spaces is computationally expensive. Existing methods typically require learning a compressed representation of the environment via auxiliary tasks, such as contrastive learning or inverse dynamics prediction. These auxiliary tasks introduce significant drawbacks: they require extra hyperparameters, cause training instability due to a constantly shifting representation space, and add substantial computational overhead.
The authors of this paper ask a compelling question: Can we estimate state entropy effectively without the baggage of additional representation learning procedures?
Methods: The RE3 Pipeline
To solve this, the authors introduce RE3 (Random Encoders for Efficient Exploration). RE3 is a remarkably simple, compute-efficient exploration method that uses a fixed, randomly initialized encoder to estimate state entropy.
The core pipeline operates as follows:
- Fixed Random Encoder: The system utilizes a standard Convolutional Neural Network (CNN) architecture initialized with random weights. Crucially, these weights are never updated during training.
- Embedding Space: High-dimensional observations (pixels) are passed through this fixed encoder to generate low-dimensional embeddings.
- k-NN Entropy Estimation: The entropy of the state distribution is estimated by measuring the distance between a new state's embedding and its $k$-nearest neighbors ($k$-NN) currently stored in the replay buffer.
- Intrinsic Reward Calculation: The intrinsic reward $r^i$ for a given state $s$ is defined as the log-distance to its $k$-th nearest neighbor: $r^i(s) = \log(||y - y^{k-NN}||_2 + 1)$.
- Reward Scheduling: The agent optimizes for a total reward that sums the environment's extrinsic reward and the intrinsic reward. The intrinsic reward is scaled by a weight $\beta_t$ that decays exponentially over time, allowing the agent to transition from pure exploration to exploitation.
To maximize computational efficiency, the low-dimensional embeddings are stored directly in the replay buffer, eliminating the need for redundant forward passes during policy updates.
Key Insights: The Power of Inductive Bias
The most striking insight from RE3 is that a randomly initialized CNN provides a highly meaningful representation space for exploration.
Because of their architectural design (local receptive fields, shared weights, pooling), CNNs possess a strong "inductive bias." They naturally group visually similar images together in the embedding space, even without any weight optimization. This inherent structure is entirely sufficient to distinguish between novel and familiar states for the sake of exploration.
Furthermore, RE3 benefits from stability through stasis. Because the random encoder is fixed, the distance between any two states in the embedding space remains constant throughout the entire training process. In contrast, methods that learn representations "on the fly" provide a moving target for the RL agent, which can destabilize the policy.
By skipping the optimization of auxiliary losses, RE3 achieves state-of-the-art exploration performance with a fraction of the computational cost (FLOPs) required by representation-learning methods.
Experimental Results
The authors evaluated RE3 across three major RL benchmarks, demonstrating its versatility and efficiency:
- DeepMind Control Suite: On visual control tasks with sparse rewards, RE3 excelled. For instance, in the Cheetah Run Sparse task, RE3 combined with the RAD algorithm achieved an average return of ~600, whereas standard RAD and DrQ failed to solve the task entirely.
- MiniGrid: On the notoriously difficult DoorKey-8x8 gridworld task, A2C combined with RE3 achieved a return of 0.49. Competing exploration methods like RND (Random Network Distillation) or ICM (Intrinsic Curiosity Module) failed to achieve non-trivial returns when paired with A2C.
- Atari 2600: RE3 successfully improved the performance of the Rainbow algorithm across several hard-exploration games, notably increasing the human-normalized score on Montezuma’s Revenge.
- Compute Efficiency: Across these tasks, RE3 required roughly half the Floating Point Operations (FLOPs) compared to methods that actively learn representations for state entropy, while achieving better or comparable results.
RE3 also proved highly effective for unsupervised pre-training, where an agent explores an environment without any extrinsic rewards. The behaviors learned during this phase significantly speed up adaptation when a specific task is later introduced.
Limitations and Future Work
While highly effective, RE3 has a few limitations. In environments with extreme visual complexity, a purely random encoder might struggle; the authors note that a pre-trained encoder (e.g., on ImageNet) might be necessary for such domains. Additionally, RE3 is task-agnostic and explores uniformly. In environments with "distractor" objects that change randomly but are irrelevant to the goal, RE3 might waste time exploring those visual distractions.
Future work could investigate how different neural architectures, such as Vision Transformers, affect the quality of the random embedding space. There is also potential in exploring how state entropy could be used to explicitly guide policy actions, rather than just serving as a passive reward bonus.
Summary updated