DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards

Shanchuan Wan, Yujin Tang, Yingtao Tian, Tomoyuki Kaneko

Abstract

Exploration is a fundamental aspect of reinforcement learning (RL), and its effectiveness is a deciding factor in the performance of RL algorithms, especially when facing sparse extrinsic rewards. Recent studies have shown the effectiveness of encouraging exploration with intrinsic rewards estimated from novelty in observations. However, there is a gap between the novelty of an observation and an exploration, as both the stochasticity in the environment and the agent's behavior may affect the observation. To evaluate exploratory behaviors accurately, we propose DEIR, a novel method in which we theoretically derive an intrinsic reward with a conditional mutual information term that principally scales with the novelty contributed by agent explorations, and then implement the reward with a discriminative forward model. Extensive experiments on both standard and advanced exploration tasks in MiniGrid show that DEIR quickly learns a better policy than the baselines. Our evaluations on ProcGen demonstrate both the generalization capability and the general applicability of our intrinsic reward.

Introduction

In reinforcement learning (RL), agents often struggle when external rewards are sparse, wandering for thousands of steps without feedback. To mitigate this, researchers use intrinsic rewards (curiosity) to reward the agent for encountering novel states. However, a major pitfall is the "Noisy TV" problem: an agent might become fixated on random environmental noise (like static on a screen) because every frame is technically novel. Standard methods often fail to distinguish between novelty caused by environmental stochasticity and novelty resulting from the agent's purposeful exploration. DEIR addresses this by ensuring rewards are tied directly to the agent's actions.

Methodology

The core of DEIR is a theoretically grounded intrinsic reward based on Conditional Mutual Information (CMI).

Discriminative Forward Model

Instead of a generative approach that attempts to predict high-dimensional next states exactly, DEIR uses a discriminative model. It trains a neural network to distinguish between genuine transitions and "fake" transitions sampled from past observations. This makes the system robust to pixel-level noise that typically degrades generative models.

Architecture and Episodic Memory

The model utilizes a CNN for visual processing and a Gated Recurrent Unit (GRU) to maintain trajectory history. This temporal context is essential for Partially Observable Markov Decision Processes (POMDPs). The agent maintains a buffer of embeddings from the current episode, and the intrinsic reward is calculated by comparing the current state to this history, encouraging the agent to visit areas it hasn't seen within that specific run.

Overview of the DEIR architecture, featuring a discriminative forward model and episodic history for intrinsic reward calculation.

Experimental Results

DEIR was evaluated on MiniGrid and ProcGen benchmarks.

Efficiency and Robustness

In the complex ObstructedMaze-Full task, DEIR reached state-of-the-art performance using only 70% of the training frames required by the previous best method, NovelD. Furthermore, DEIR demonstrated exceptional robustness to noise. While traditional baselines like ICM and RND failed in environments with added Gaussian noise, DEIR maintained consistent performance across variants like reduced view sizes and invisible obstacles.

Exploration coverage comparison between DEIR and the best-performing baseline across various noise conditions in MiniGrid.

Generalization

On the ProcGen benchmark, DEIR outperformed or matched SOTA baselines across 16 procedurally generated games, proving its scalability to complex, high-dimensional visual environments. Ablation studies confirmed that both the CMI scaling and the discriminative model are critical components for its success.

Conclusion

DEIR provides a robust framework for exploration by mathematically decoupling environmental randomness from agent-driven discovery. By leveraging a discriminative forward model and trajectory embeddings, it overcomes the Noisy TV problem and excels in partially observable environments. Future work includes extending DEIR to continuous action spaces and exploring its utility in multi-agent settings.

Summary updated

Abstract

Exploration is a fundamental aspect of reinforcement learning (RL), and its effectiveness is a deciding factor in the performance of RL algorithms, especially when facing sparse extrinsic rewards. Recent studies have shown the effectiveness of encouraging exploration with intrinsic rewards estimated from novelties in observations. However, there is a gap between the novelty of an observation and an exploration, as both the stochasticity in the environment and the agent's behavior may affect the observation. To evaluate exploratory behaviors accurately, we propose DEIR, a novel method in which we theoretically derive an intrinsic reward with a conditional mutual information term that principally scales with the novelty contributed by agent explorations, and then implement the reward with a discriminative forward model. Extensive experiments on both standard and advanced exploration tasks in MiniGrid show that DEIR quickly learns a better policy than the baselines. Our evaluations on ProcGen demonstrate both the generalization capability and the general applicability of our intrinsic reward.
BibTeX
@inproceedings{wan2023deir,
  author = {Shanchuan Wan and Yujin Tang and Yingtao Tian and Tomoyuki Kaneko},
  title = {{DEIR: Efficient and Robust Exploration through Discriminative-Model-Based Episodic Intrinsic Rewards}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=x78Pi3cGQX},
}
Figure