Introduction
Model-Based Reinforcement Learning (MBRL) is recognized for its sample efficiency, achieved by learning a "world model" to simulate environment interactions. However, a persistent challenge in MBRL is the tendency for agents to converge on local optima, ceasing exploration once a moderately successful strategy is identified.
To address this, researchers often turn to Posterior Sampling for Reinforcement Learning (PSRL), which utilizes Thompson Sampling. Instead of optimizing for a single "best" model, the agent samples from a posterior distribution of models. While effective in theory, maintaining a true posterior for high-dimensional image inputs is computationally prohibitive. EVaDE (Event-based Variational Distributions for Exploration) proposes a solution by introducing specialized variational layers that provide the necessary inductive biases for object-based domains like Atari games.
Methods: The EVaDE Architecture
EVaDE introduces three novel convolutional layers that employ Gaussian multiplicative dropout to approximate variational distributions over the reward function. These layers are integrated into the reward model of the SimPLe (Simulated Policy Learning) architecture to induce "reward hallucinations" that drive exploration.
Specialized Noisy Layers
- Noisy Event Interaction Layer: This layer uses $m \times m \times c$ filters to perturb the interactions between different input channels (representing distinct objects). This encourages the agent to explore novel object-to-object interactions.
- Noisy Event Weighting Layer: By randomly scaling specific feature maps, this layer prevents the agent from over-optimizing for a single event type (e.g., small, frequent rewards) and pushes it toward discovering high-value milestones.
- Noisy Event Translation Layer: This layer introduces spatial perturbations to object positions. This "fuzziness" helps agents navigate narrow passages where high precision is required, preventing them from getting stuck in sequences where minor positional errors lead to failure.
The authors demonstrate via Theorem 1 that these layers do not restrict the network's representational capacity, as each can learn to function as an identity mapping if the data supports it.
Experimental Evaluation
The method was rigorously tested on the Atari 100K benchmark, a standard for evaluating sample efficiency where agents are limited to 100,000 environment steps (approximately two hours of gameplay).
Key Results
EVaDE-SimPLe demonstrated superior performance across several metrics:
- Performance Gain: Achieved a Mean Human-Normalized Score (HNS) of 0.682, representing a 30% improvement over the vanilla SimPLe baseline and a 79% improvement over the contrastive learning method CURL.
- Benchmark Dominance: EVaDE-SimPLe secured the highest score in 11 out of 26 games in the suite.
- Robustness: Using the Inter-Quartile Mean (IQM) metric, which mitigates the impact of outliers, EVaDE outperformed the baseline by 68%.
Ablation studies confirmed that while each layer contributes to performance, their combination offers the most consistent exploration benefits across diverse game dynamics.
Discussion and Future Work
EVaDE demonstrates that exploration can be effectively driven by structured noise that respects the logic of the environment. By perturbing the reward model in ways that mimic meaningful game events, the agent is naturally incentivized to explore the state space.
Future research directions include adapting these layers for continuous control tasks where "events" are less discrete and integrating EVaDE into search-based architectures like EfficientZero to enhance Monte Carlo Tree Search (MCTS) exploration. Despite the significant computational cost of benchmarking—requiring over six months of GPU time—the results suggest that event-based variational distributions are a powerful tool for sample-efficient RL.
Summary updated