DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards

Shanchuan Wan, Yujin Tang, Yingtao Tian, Tomoyuki Kaneko

Abstract

Exploration is a fundamental aspect of reinforcement learning (RL), and its effectiveness is a deciding factor in the performance of RL algorithms, especially when facing sparse extrinsic rewards. Recent studies have shown the effectiveness of encouraging exploration with intrinsic rewards estimated from novelties in observations. However, there is a gap between the novelty of an observation and an exploration, as both the stochasticity in the environment and the agent's behavior may affect the observation. To evaluate exploratory behaviors accurately, we propose DEIR, a novel method in which we theoretically derive an intrinsic reward with a conditional mutual information term that principally scales with the novelty contributed by agent explorations, and then implement the reward with a discriminative forward model. Extensive experiments on both standard and advanced exploration tasks in MiniGrid show that DEIR quickly learns a better policy than the baselines. Our evaluations on ProcGen demonstrate both the generalization capability and the general applicability of our intrinsic reward.

Introduction

In Reinforcement Learning (RL), the ability of an agent to explore its environment effectively often determines whether it will master a task or fail to find a solution. In environments with sparse extrinsic rewards—where a signal is only received upon completion—agents rely on "intrinsic rewards" or artificial curiosity to drive discovery. However, a significant challenge remains: the "Noisy TV" problem. Standard curiosity-driven agents can become trapped by environmental stochasticity (like a flickering screen), perceiving random noise as novelty worth investigating.

To address this, we introduce DEIR (Discriminative-model-based Episodic Intrinsic Reward). DEIR is designed to decouple environmental noise from an agent’s actual exploratory progress by mathematically ensuring that intrinsic rewards are scaled by the agent's own contribution to the observed novelty.

The DEIR Framework

1. Conditional Mutual Information (CMI) Scaling

The core innovation of DEIR is the use of a conditional mutual information term to scale intrinsic rewards. Rather than simply measuring the distance between two consecutive observations, DEIR asks: How much did the agent's specific action contribute to this change? By scaling novelty with CMI, the framework suppresses rewards for changes caused by random environmental noise and amplifies rewards for states reached through intentional agent behavior.

2. Discriminative Forward Model

Traditional forward models attempt to predict the next state pixel-by-pixel, a process that is computationally expensive and often inaccurate in high-dimensional or noisy spaces. DEIR instead utilizes a discriminative model inspired by contrastive learning. This model learns to distinguish between genuine trajectories and "fake" ones (randomly sampled past observations). This approach allows the agent to capture the underlying structure of the task without the overhead of full state reconstruction.

3. Reward Calculation

The intrinsic reward in DEIR is calculated as the ratio of the squared distance between observation embeddings to the distance between trajectory embeddings. This ensures that the agent is incentivized to reach states that are not only visually novel but also represent a significant departure from previous action sequences.

Experimental Evaluation

Benchmarks

DEIR was evaluated across two rigorous RL benchmarks:

  • MiniGrid: A grid-world suite featuring 20 exploration tasks. The researchers introduced "Advanced" variants to test robustness, including reduced view sizes (3x3), Gaussian noise, and invisible obstacles.
  • ProcGen: A suite of 16 procedurally generated games designed to evaluate an agent's ability to generalize to unseen levels.

Key Results

  • Efficiency: In the complex ObstructedMaze-Full task within MiniGrid, DEIR achieved state-of-the-art performance using only 70% of the training frames required by previous leading methods like NovelD.
  • Robustness to Noise: DEIR demonstrated remarkable stability in noisy environments. While baseline methods saw performance drops of up to 75% when Gaussian noise was introduced, DEIR’s performance remained consistent.
  • Generalization: On ProcGen's Hard mode, DEIR consistently outperformed or matched top-tier baselines such as NGU and ICM, proving that its learned representations are robust across randomized environments.

Discussion and Future Work

DEIR represents a significant step toward more "intelligent" curiosity in RL. By focusing on agent-caused novelty, it avoids the pitfalls of environmental stochasticity that plague earlier methods.

However, several avenues for future research remain. Currently, DEIR utilizes an episodic reward structure that resets every game; combining this with "lifelong" rewards could further enhance long-term exploration. Additionally, while DEIR has proven effective in discrete action spaces, adapting the CMI term for continuous action spaces (such as robotics) is a priority. Finally, the ability to distinguish self-caused novelty holds great promise for multi-agent systems, where the movements of other agents often act as a complex form of environmental noise.

Summary updated

Abstract

Exploration is a fundamental aspect of reinforcement learning (RL), and its effectiveness is a deciding factor in the performance of RL algorithms, especially when facing sparse extrinsic rewards. Recent studies have shown the effectiveness of encouraging exploration with intrinsic rewards estimated from novelties in observations. However, there is a gap between the novelty of an observation and an exploration, as both the stochasticity in the environment and the agent's behavior may affect the observation. To evaluate exploratory behaviors accurately, we propose DEIR, a novel method in which we theoretically derive an intrinsic reward with a conditional mutual information term that principally scales with the novelty contributed by agent explorations, and then implement the reward with a discriminative forward model. Extensive experiments on both standard and advanced exploration tasks in MiniGrid show that DEIR quickly learns a better policy than the baselines. Our evaluations on ProcGen demonstrate both the generalization capability and the general applicability of our intrinsic reward. Our source code is available at https://github.com/swan-utokyo/deir.
BibTeX
@inproceedings{wan2023deir,
  author = {Shanchuan Wan and Yujin Tang and Yingtao Tian and Tomoyuki Kaneko},
  title = {{DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=x78Pi3cGQX},
}
Figure