Abstract
Exploration is a fundamental aspect of reinforcement learning (RL), and its effectiveness is a deciding factor in the performance of RL algorithms, especially when facing sparse extrinsic rewards. Recent studies have shown the effectiveness of encouraging exploration with intrinsic rewards estimated from novelty in observations. However, there is a gap between the novelty of an observation and an exploration, as both the stochasticity in the environment and the agent's behavior may affect the observation. To evaluate exploratory behaviors accurately, we propose DEIR, a novel method in which we theoretically derive an intrinsic reward with a conditional mutual information term that principally scales with the novelty contributed by agent explorations, and then implement the reward with a discriminative forward model. Extensive experiments on both standard and advanced exploration tasks in MiniGrid show that DEIR quickly learns a better policy than the baselines. Our evaluations on ProcGen demonstrate both the generalization capability and the general applicability of our intrinsic reward.
Introduction
In reinforcement learning (RL), agents often struggle when external rewards are sparse, wandering for thousands of steps without feedback. To mitigate this, researchers use intrinsic rewards (curiosity) to reward the agent for encountering novel states. However, a major pitfall is the "Noisy TV" problem: an agent might become fixated on random environmental noise (like static on a screen) because every frame is technically novel. Standard methods often fail to distinguish between novelty caused by environmental stochasticity and novelty resulting from the agent's purposeful exploration. DEIR addresses this by ensuring rewards are tied directly to the agent's actions.
Methodology
The core of DEIR is a theoretically grounded intrinsic reward based on Conditional Mutual Information (CMI).
Discriminative Forward Model
Instead of a generative approach that attempts to predict high-dimensional next states exactly, DEIR uses a discriminative model. It trains a neural network to distinguish between genuine transitions and "fake" transitions sampled from past observations. This makes the system robust to pixel-level noise that typically degrades generative models.
Architecture and Episodic Memory
The model utilizes a CNN for visual processing and a Gated Recurrent Unit (GRU) to maintain trajectory history. This temporal context is essential for Partially Observable Markov Decision Processes (POMDPs). The agent maintains a buffer of embeddings from the current episode, and the intrinsic reward is calculated by comparing the current state to this history, encouraging the agent to visit areas it hasn't seen within that specific run.
Experimental Results
DEIR was evaluated on MiniGrid and ProcGen benchmarks.
Efficiency and Robustness
In the complex ObstructedMaze-Full task, DEIR reached state-of-the-art performance using only 70% of the training frames required by the previous best method, NovelD. Furthermore, DEIR demonstrated exceptional robustness to noise. While traditional baselines like ICM and RND failed in environments with added Gaussian noise, DEIR maintained consistent performance across variants like reduced view sizes and invisible obstacles.
Generalization
On the ProcGen benchmark, DEIR outperformed or matched SOTA baselines across 16 procedurally generated games, proving its scalability to complex, high-dimensional visual environments. Ablation studies confirmed that both the CMI scaling and the discriminative model are critical components for its success.
Conclusion
DEIR provides a robust framework for exploration by mathematically decoupling environmental randomness from agent-driven discovery. By leveraging a discriminative forward model and trajectory embeddings, it overcomes the Noisy TV problem and excels in partially observable environments. Future work includes extending DEIR to continuous action spaces and exploring its utility in multi-agent settings.
Summary updated