EVaDE : Event-Based Variational Thompson Sampling for Model-Based Reinforcement Learning

Siddharth Aravindan, Dixant Mittal, Wee Sun Lee

TL;DR

EVaDE introduces object-aware variational dropout layers designed specifically to improve exploration in model-based reinforcement learning. By injecting structured noise into object interactions, weights, and spatial translations within a learned reward model, it significantly boosts sample efficiency in low-data environments like the Atari 100K benchmark.

What Problem Does This Paper Address?

Model-Based Reinforcement Learning (MBRL) is highly prized for its sample efficiency. By learning a "world model" (a simulator of the environment's transitions and rewards), an agent can practice entirely in imagination. To explore effectively, agents often use Posterior Sampling for Reinforcement Learning (PSRL)—a form of Thompson Sampling where the agent samples a plausible world model from a posterior distribution and optimizes its policy against it.

However, maintaining a true posterior distribution over complex, high-dimensional visual world models is mathematically intractable. While standard neural network dropout can be used to approximate these posteriors via variational inference, generic dropout injects unstructured noise. In object-based domains (like most video games), rewards depend heavily on specific object positions and interactions (which the authors call events). Unstructured noise fails to drive targeted exploration toward these meaningful events, often leaving agents trapped in local optima.

Methods

The authors propose EVaDE (Event-based Variational Distributions for Exploration). Instead of generic dropout, they introduce three specialized convolutional layers that leverage domain knowledge of object-based environments. These layers use Gaussian multiplicative dropout to perturb the reward model in structurally meaningful ways:

  1. Noisy Event Interaction Layer: Uses $m \times m \times c$ filters to combine input channels locally. Assuming channels represent different objects, this layer randomly alters how combinations of objects interacting at the same spatial location are rewarded.
  2. Noisy Event Weighting Layer: Uses $1 \times 1$ convolutional filters to randomly scale (up-weight or down-weight) entire feature channels, changing the perceived importance of specific detected events.
  3. Noisy Event Translation Layer: Shifts feature maps slightly in spatial directions (top, bottom, left, right). This helps the agent explore alternate trajectories through "narrow passages" where precise spatial positioning determines success or failure.
The three types of EVaDE filters: (a) interaction, (b) weighting, and (c) translation.

The authors integrate these layers into the reward network of SimPLe (Simulated Policy Learning), an iterative MBRL framework. This allows the agent to plan and optimize its policy using diverse, structurally perturbed reward models sampled from the induced variational posterior.

Datasets

The method was evaluated on the Atari 100K test suite, a standard benchmark consisting of 26 Atari games. In this benchmark, the agent is strictly limited to 100,000 interactions with the real environment—a highly restrictive, low-data regime designed to test sample efficiency.

Key Insights and Ideas

  • Exploration Doesn't Require Perfect Posteriors: A core insight of the paper is that for exploration purposes, a variational distribution doesn't need to perfectly match the true posterior. As long as the distribution successfully perturbs the policy enough to break out of local optima, any model inaccuracies will eventually be corrected by new data from the real environment.
  • Injecting Inductive Biases into Noise: Instead of just using object-oriented priors to help a model learn faster, EVaDE uses them to help the model explore smarter. By structuring the noise around object interactions and translations, the agent imagines variations of the world that are actually worth exploring.
  • Zero Representational Cost: The authors prove mathematically (Theorem 1) that adding these EVaDE layers does not hinder the network's capacity to learn the true underlying model. Because each layer can easily learn to represent the identity function, they provide exploration benefits during training without capping the model's ultimate accuracy.

Key Results

  • Superior Sample Efficiency: EVaDE-SimPLe achieved a mean Human Normalized Score (HNS) of 0.682, outperforming the popular low-data baseline CURL (0.381) by 79% and vanilla SimPLe (0.525) by 30%.
  • Top Performance: It achieved the highest score in 11 out of the 26 games when compared against strong baselines like Data-Efficient Rainbow and OTRainbow.
  • Beating Humans: The EVaDE-equipped agent surpassed average human performance levels in 5 games: Asterix, Boxing, CrazyClimber, Freeway, and Krull.
  • Ablation Validation: Ablation studies confirmed that while each individual layer improves exploration, their combination yields the highest overall performance and robustness across different game dynamics.

Limitations and Future Work

  • Reward-Only Perturbations: EVaDE currently only applies structured noise to the reward function model, leaving the transition dynamics model deterministic. Extending this structured variational approach to transition dynamics could further enhance exploration in environments with complex physics.
  • Integration with Modern Search: While EVaDE was evaluated on top of SimPLe, the current state-of-the-art methods on the Atari 100K benchmark (such as EfficientZero) rely heavily on Monte Carlo Tree Search (MCTS). Integrating EVaDE's structured reward posteriors into search-based planners remains an open and highly promising direction for future research.

Summary updated

Abstract

Posterior Sampling for Reinforcement Learning (PSRL) is a well-known algorithm that augments model-based reinforcement learning (MBRL) algorithms with Thompson sampling. PSRL maintains posterior distributions of the environment transition dynamics and the reward function, which are intractable for tasks with high-dimensional state and action spaces. Recent works show that dropout, used in conjunction with neural networks, induces variational distributions that can approximate these posteriors. In this paper, we propose Event-based Variational Distributions for Exploration (EVaDE), which are variational distributions that are useful for MBRL, especially when the underlying domain is object-based. We leverage the general domain knowledge of object-based domains to design three types of event-based convolutional layers to direct exploration. These layers rely on Gaussian dropouts and are inserted between the layers of the deep neural network model to help facilitate variational Thompson sampling. We empirically show the effectiveness of EVaDE-equipped Simulated Policy Learning (EVaDE-SimPLe) on the 100K Atari game suite.
BibTeX
@inproceedings{aravindan2024evade,
  author = {Siddharth Aravindan and Dixant Mittal and Wee Sun Lee},
  title = {{EVaDE : Event-Based Variational Thompson Sampling for Model-Based Reinforcement Learning}},
  booktitle = {ACML Conference Track},
  year = {2024},
  url = {https://openreview.net/forum?id=OYlNXMMSXB},
}
Figure