TL;DR
EVaDE (Event-based Variational Distributions for Exploration) improves exploration in model-based reinforcement learning by incorporating object-centric inductive biases into reward model posteriors. By inserting three specialized, Gaussian-dropout-based convolutional layers into the reward network, EVaDE enables targeted variational Thompson sampling. Evaluated on the data-constrained Atari 100K benchmark, EVaDE-SimPLe boosts the mean human-normalized score by 30% over its base algorithm.
What Problem Does This Paper Address?
In data-sparse environments where agents can only take a limited number of actions (e.g., 100K steps), effective exploration is crucial. Posterior Sampling for Reinforcement Learning (PSRL) tackles exploration via Thompson sampling—drawing plausible transition and reward models from a posterior distribution. However, maintaining exact posteriors in deep neural networks with pixel inputs is intractable.
While techniques like dropout can induce variational approximations of the posterior, standard variational distributions do not take advantage of domain structure. In visually rich, object-based domains (like classic video games), rewards typically stem from specific "events"—such as collisions or interactions between objects. Without an inductive bias reflecting this structure, policies often converge to suboptimal behaviors or fixate on easily discovered local rewards.
Methods
EVaDE approximates PSRL by using variational inference over deterministic reward models. Using Gaussian multiplicative dropout ($\tilde{\theta} = \theta(1 + \sigma \epsilon)$, where $\epsilon \sim \mathcal{N}(0, 1)$), the variances $\sigma^2$ are learned jointly with parameters $\theta$. The authors design three modular convolutional layers representing distinct exploration primitives:
- Noisy Event Interaction Layer ($m \times m \times c$): Combines channels across small spatial patches. If individual channels detect object locations, perturbing these filters alters the perceived value of multi-object collocations, prompting the agent to seek out varied interactions.
- Noisy Event Weighting Layer ($1 \times 1 \times c$): Uses channel-wise trainable scale factors to randomly increase or decrease the reward weight of specific detected events, helping the agent escape local optima caused by overemphasizing a single event.
- Noisy Event Translation Layer ($(2n+1) \times (2n+1)$): Constrained filters (trainable only along the center row and column) that softly translate spatial activations horizontally or vertically. This perturbs "narrow passage" trajectories—sequences of delicate, highly localized actions—forcing the policy to find alternative paths.
The authors prove (Theorem 1) that because these layers can represent the identity function (under stride 1 and non-shrinking dimensions), adding them does not hinder the model's capacity to represent the true underlying function.
EVaDE is integrated into SimPLe (Simulated Policy Learning). Notably, the perturbations are applied exclusively to the reward head while the transition dynamics model remains deterministic and shared.
┌─────────────────────────┐
│ Input Video Frames │
└────────────┬────────────┘
│
┌─────────────┴─────────────┐
│ Base Dynamics Model │
└──────┬─────────────┬──────┘
│ │
(Shared Features) │ │ (Next-Frame Predictions)
▼ ▼
┌────────────────────────┐ ┌───────────────────────┐
│ EVaDE Layers │ │ Next-State Prediction │
│ • Event Translation │ └───────────────────────┘
│ • Event Weighting │
│ • Event Interaction │
└───────────┬────────────┘
│ (Perturbed Reward Sample)
▼
┌────────────────────────┐
│ Reward Output │
└────────────────────────┘
Datasets
The authors evaluated their method on the Atari 100K benchmark, which restricts training to only 100,000 environment interaction steps (roughly two hours of real-time gameplay) across 26 Atari games. An ablation study was conducted on a randomly selected 12-game subset.
Key Insights and Ideas
- Variational approximations need not be exact to be useful: For exploration, a variational posterior does not need to precisely fit the true Bayesian posterior. It simply needs to perturb the policy out of local optima into promising state regions; subsequent environment interactions will correct any modeling errors.
- Architecture as an inductive bias for exploration: Instead of adding generic parameter noise, EVaDE structures the perturbations around the semantics of vision-based environments: what interacted (interactions), how much it matters (weighting), and where it occurred (translation).
- Inductive bias beats parameter count: The gains are not due to added capacity. The noisy event translation layer adds only 4K parameters to a 10M-parameter model, yet significantly outperforms the baseline. The entire EVaDE architecture adds just 0.43M parameters.
- Qualitative policy shifts: In RoadRunner, agents trained with only translation layers learned cautious, collision-avoidant behaviors, while agents with interaction layers learned to lure the coyote into obstacles. The full EVaDE agent combined both strategies: tricking the coyote while keeping a safe distance.
Key Results
- Higher Benchmark Scores: EVaDE-SimPLe achieved a mean Human-Normalized Score (HNS) of 0.682, outperforming vanilla SimPLe (0.443), an iteration-matched SimPLe(30) baseline (0.525), and representation-learning baselines like CURL (0.381) and OverTrained Rainbow (0.264).
- Robust Statistics: EVaDE-SimPLe yielded an Inter-Quartile Mean (IQM) of 0.339 (a 68% improvement over SimPLe(30)'s 0.202) and outperformed SimPLe(30) in 23 of the 26 games ($p = 3 \times 10^{-3}$, paired t-test).
- Superhuman Performance: EVaDE-SimPLe exceeded human-level performance in 5 games: Asterix, Boxing, CrazyClimber, Freeway, and Krull.
- Ablations: In the 12-game ablation study, each of the three layers provided individual performance gains over the baseline, with the full combination achieving the highest overall HNS (0.77 vs. 0.52).
Limitations and Future Work
- Domain Dependency: EVaDE is explicitly designed for object-centric domains where visual feature maps correspond to distinct entities and events; it may not generalize straightforwardly to continuous-control tasks or domains lacking clear spatial object interactions.
- Exploration Budget Trade-offs: Learning curves show that EVaDE sometimes experiences delayed performance early in training. The authors suggest the agent occasionally spends its limited interaction budget testing ineffective exploration directions suggested by one of the layers before finding high-reward trajectories.
- Restricted to Reward Perturbations: EVaDE was only applied to the reward model. Applying structured variational distributions to the transition dynamics model remains unexplored.
- Compatibility with Tree-Search: The authors note that concurrent state-of-the-art algorithms (like EfficientZero) leverage Monte Carlo Tree Search; combining EVaDE's reward exploration with tree-search policy planning is left as promising future work.
Summary updated