EVaDE : Event-Based Variational Thompson Sampling for Model-Based Reinforcement Learning

Siddharth Aravindan, Dixant Mittal, Wee Sun Lee

Introduction

Model-Based Reinforcement Learning (MBRL) is recognized for its sample efficiency, achieved by learning a "world model" to simulate environment interactions. However, a persistent challenge in MBRL is the tendency for agents to converge on local optima, ceasing exploration once a moderately successful strategy is identified.

Breakout environment examples illustrating the object-based nature of the Atari 100K benchmark.

To address this, researchers often turn to Posterior Sampling for Reinforcement Learning (PSRL), which utilizes Thompson Sampling. Instead of optimizing for a single "best" model, the agent samples from a posterior distribution of models. While effective in theory, maintaining a true posterior for high-dimensional image inputs is computationally prohibitive. EVaDE (Event-based Variational Distributions for Exploration) proposes a solution by introducing specialized variational layers that provide the necessary inductive biases for object-based domains like Atari games.

Methods: The EVaDE Architecture

EVaDE introduces three novel convolutional layers that employ Gaussian multiplicative dropout to approximate variational distributions over the reward function. These layers are integrated into the reward model of the SimPLe (Simulated Policy Learning) architecture to induce "reward hallucinations" that drive exploration.

The three proposed EVaDE layers: (a) Noisy Event Interaction, (b) Noisy Event Weighting, and (c) Noisy Event Translation.

Specialized Noisy Layers

  1. Noisy Event Interaction Layer: This layer uses $m \times m \times c$ filters to perturb the interactions between different input channels (representing distinct objects). This encourages the agent to explore novel object-to-object interactions.
  2. Noisy Event Weighting Layer: By randomly scaling specific feature maps, this layer prevents the agent from over-optimizing for a single event type (e.g., small, frequent rewards) and pushes it toward discovering high-value milestones.
  3. Noisy Event Translation Layer: This layer introduces spatial perturbations to object positions. This "fuzziness" helps agents navigate narrow passages where high precision is required, preventing them from getting stuck in sequences where minor positional errors lead to failure.

The authors demonstrate via Theorem 1 that these layers do not restrict the network's representational capacity, as each can learn to function as an identity mapping if the data supports it.

Experimental Evaluation

The method was rigorously tested on the Atari 100K benchmark, a standard for evaluating sample efficiency where agents are limited to 100,000 environment steps (approximately two hours of gameplay).

Key Results

EVaDE-SimPLe demonstrated superior performance across several metrics:

  • Performance Gain: Achieved a Mean Human-Normalized Score (HNS) of 0.682, representing a 30% improvement over the vanilla SimPLe baseline and a 79% improvement over the contrastive learning method CURL.
  • Benchmark Dominance: EVaDE-SimPLe secured the highest score in 11 out of 26 games in the suite.
  • Robustness: Using the Inter-Quartile Mean (IQM) metric, which mitigates the impact of outliers, EVaDE outperformed the baseline by 68%.

Ablation studies confirmed that while each layer contributes to performance, their combination offers the most consistent exploration benefits across diverse game dynamics.

Discussion and Future Work

EVaDE demonstrates that exploration can be effectively driven by structured noise that respects the logic of the environment. By perturbing the reward model in ways that mimic meaningful game events, the agent is naturally incentivized to explore the state space.

Future research directions include adapting these layers for continuous control tasks where "events" are less discrete and integrating EVaDE into search-based architectures like EfficientZero to enhance Monte Carlo Tree Search (MCTS) exploration. Despite the significant computational cost of benchmarking—requiring over six months of GPU time—the results suggest that event-based variational distributions are a powerful tool for sample-efficient RL.

Summary updated

Abstract

Posterior Sampling for Reinforcement Learning (PSRL) is a well-known algorithm that augments model-based reinforcement learning (MBRL) algorithms with Thompson sampling. PSRL maintains posterior distributions of the environment transition dynamics and the reward function to procure posterior samples that are used to generate data for training the controller. Maintaining posterior distributions over all possible transition and reward functions for tasks with high dimensional state and action spaces is intractable. Recent works show that dropout used in conjunction with neural networks induce variational distributions that can approximate these posteriors. In this paper, we propose Event-based Variational Distributions for Exploration (EVaDE), variational distributions that are useful for MBRL, especially when the underlying domain is object-based. We leverage the general domain knowledge of object-based domains to design three types of event-based convolutional layers to direct exploration, namely the noisy event interaction layer, the noisy event weighting layer and the noisy event translation layer respectively. These layers rely on Gaussian dropouts and are inserted in between the layers of the deep neural network model to help facilitate variational Thompson sampling. We empirically show the effectiveness of EVaDE equipped Simulated Policy Learning (SimPLe) on a randomly selected suite of Atari games, where the number of agent environment interactions is limited to 100K.
BibTeX
@inproceedings{aravindan2024evade,
  author = {Siddharth Aravindan and Dixant Mittal and Wee Sun Lee},
  title = {{EVaDE : Event-Based Variational Thompson Sampling for Model-Based Reinforcement Learning}},
  booktitle = {ACML Conference Track},
  year = {2024},
  url = {https://openreview.net/forum?id=OYlNXMMSXB},
}
Figure