State Entropy Maximization with Random Encoders for Efficient Exploration

Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, Kimin Lee

Abstract

Recent exploration methods have proven to be a recipe for improving sample-efficiency in deep reinforcement learning (RL). However, efficient exploration in high-dimensional observation spaces still remains a challenge. This paper presents Random Encoders for Efficient Exploration (RE3), an exploration method that utilizes state entropy as an intrinsic reward. In order to estimate state entropy in environments with high-dimensional observations, we utilize a k-nearest neighbor entropy estimator in the low-dimensional representation space of a convolutional encoder. In particular, we find that the state entropy can be estimated in a stable and compute-efficient manner by utilizing a randomly initialized encoder, which is fixed throughout training. Our experiments show that RE3 significantly improves the sample-efficiency of both model-free and model-based RL methods on locomotion and navigation tasks from DeepMind Control Suite and MiniGrid benchmarks. We also show that RE3 allows learning diverse behaviors without extrinsic rewards, effectively improving sample-efficiency in downstream tasks.

Introduction

In Reinforcement Learning (RL), agents often struggle to explore environments with high-dimensional observations, such as pixels, especially when extrinsic rewards are sparse. To overcome this, intrinsic rewards are often employed to encourage the visitation of novel states. A principled approach to this is state entropy maximization, which rewards the agent for spreading its visits uniformly across the state space.

However, calculating entropy directly from high-dimensional pixel data is computationally prohibitive. Existing state-of-the-art methods typically address this by learning low-dimensional representations through auxiliary tasks, such as next-state prediction or contrastive learning. While effective, these auxiliary tasks introduce significant complexity, requiring careful hyperparameter tuning and imposing a heavy computational burden. Furthermore, the non-stationary nature of a learning representation can lead to unstable intrinsic reward signals. RE3 challenges the necessity of learning these representations for effective exploration.

The RE3 Framework

RE3 (Random Encoders for Efficient Exploration) is a simple and compute-efficient exploration method that utilizes a fixed, randomly initialized convolutional encoder to estimate state entropy.

The RE3 architecture: a fixed random encoder provides intrinsic rewards based on state entropy, while a separate RL encoder optimizes the policy.

The framework consists of two primary components:

  1. Fixed Random Encoder: RE3 uses a standard convolutional neural network (CNN) with weights that are randomly initialized and frozen throughout training. This leverages the "Deep Image Prior"—the concept that CNN architectures themselves possess a powerful inductive bias for extracting meaningful features.
  2. k-Nearest Neighbor (k-NN) Entropy Estimator: The agent maps pixel observations into a low-dimensional vector space using the random encoder. The intrinsic reward is then calculated based on the distance to the $k$ closest vectors in the replay buffer: $$r^i(s_t) = \log(||y_t - y_t^{k\text{-NN}}||_2 + 1)$$ A larger distance indicates a more novel state, resulting in a higher intrinsic reward. To balance exploration and exploitation, the weight of this intrinsic reward is exponentially decayed over time.

Why Random Encoders Work

A key finding of this work is that representation learning is not strictly necessary for effective exploration. Even with random weights, a CNN can distinguish between semantically different scenes.

Visualization of nearest neighbors in the representation space. Random encoders preserve semantic similarity comparable to the true state space.

As shown in the visualization above, states that are neighbors in the random encoder's representation space are visually and semantically similar, mirroring the behavior of the true state space. Because the encoder is fixed, the distance between any two states remains constant, providing a stable and reliable reward signal. This stability, combined with the lack of gradient updates for the exploration module, makes RE3 approximately 2x more compute-efficient than methods that learn representations online.

Experimental Evaluation

The authors evaluated RE3 across several challenging benchmarks, comparing it against established methods like Random Network Distillation (RND) and the Intrinsic Curiosity Module (ICM).

  • DeepMind Control Suite: In visual locomotion tasks with sparse rewards, RE3 showed dramatic improvements. For instance, on the Cheetah Run Sparse task, RE3 combined with RAD achieved an average return of ~600, while standard baselines failed to solve the task.
  • MiniGrid: In procedurally generated environments like DoorKey-8x8, RE3 achieved a return of 0.49, significantly outperforming ICM and RND, which failed to achieve non-trivial returns.
  • Atari (ALE): RE3 improved the performance of the Rainbow agent on Montezuma's Revenge, nearly doubling the score compared to the baseline.

Conclusion

RE3 demonstrates that a fixed random encoder provides a surprisingly effective and stable basis for state entropy estimation in high-dimensional RL. By bypassing the complexities of online representation learning, RE3 offers a compute-efficient and robust solution for exploration. Future work may explore the use of ensembles of random encoders for more complex backgrounds or investigate how different network architectures, such as Vision Transformers, might further enhance these "random" features.

Summary updated

Abstract

Recent exploration methods have proven to be a recipe for improving sample-efficiency in deep reinforcement learning (RL). However, efficient exploration in high-dimensional observation spaces still remains a challenge. This paper presents Random Encoders for Efficient Exploration (RE3), an exploration method that utilizes state entropy as an intrinsic reward. In order to estimate state entropy in environments with high-dimensional observations, we utilize a k-nearest neighbor entropy estimator in the low-dimensional representation space of a convolutional encoder. In particular, we find that the state entropy can be estimated in a stable and compute-efficient manner by utilizing a randomly initialized encoder, which is fixed throughout training. Our experiments show that RE3 significantly improves the sample-efficiency of both model-free and model-based RL methods on locomotion and navigation tasks from DeepMind Control Suite and MiniGrid benchmarks. We also show that RE3 allows learning diverse behaviors without extrinsic rewards, effectively improving sample-efficiency in downstream tasks. Source code and videos are available at https://sites.google.com/view/re3-rl.
BibTeX
@inproceedings{seo2021state,
  author = {Younggyo Seo and Lili Chen and Jinwoo Shin and Honglak Lee and Pieter Abbeel and Kimin Lee},
  title = {{State Entropy Maximization with Random Encoders for Efficient Exploration}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=SYUZvB0HzGq},
}
Figure