Diverse Projection Ensembles for Distributional Reinforcement Learning

Moritz Akiya Zanger, Wendelin Boehmer, Matthijs T. J. Spaan

TL;DR: Distributional Reinforcement Learning (RL) maps complex return distributions onto simpler forms using "projections." This paper demonstrates that an ensemble of models using different projection methods (e.g., categorical vs. quantile) creates diverse generalization behaviors. This diversity leads to significantly better epistemic uncertainty estimation and more effective exploration in challenging environments.

The Challenge: Epistemic Uncertainty in Distributional RL

Reinforcement learning agents face a fundamental trade-off: exploiting known rewards versus exploring the environment to discover better ones. Effective exploration requires agents to quantify epistemic uncertainty—an awareness of what they do not yet know. While deep ensembles are a popular tool for estimating this uncertainty, standard ensembles often lack sufficient diversity in how they generalize to unseen states, frequently leading to overconfidence.

In Distributional RL, agents learn the entire distribution of future returns rather than just the expected average. This captures aleatoric uncertainty (inherent environmental randomness). To achieve this, algorithms must "project" the true, complex return distribution onto a simpler parametric set, such as a histogram or a set of quantiles. The authors argue that these projection steps impose a strong inductive bias. If an ensemble uses the identical projection method for every member, the models are likely to generalize in the exact same way in novel states, rendering them blind to their own epistemic uncertainty.

Methodology: Diverse Projection Ensembles

To overcome this lack of diversity, the authors introduce Projection Ensembles, a framework that combines different distributional representations and projections within a single ensemble.

  1. Projection Mixture Operator ($\Omega_M$): The core theoretical contribution is a formal operator that applies multiple different projections (like Categorical and Quantile) to the distributional Bellman target. The authors prove that this combined operator remains a contraction, guaranteeing that the learning process converges.
  2. Uncertainty Propagation: The framework includes a scheme to propagate "distributional TD errors" over time. This allows the agent to estimate the total uncertainty of a state-action pair by measuring the disagreement between ensemble members.
  3. PE-DQN (Projection Ensemble DQN): The practical deep RL implementation utilizes an ensemble consisting of two popular distributional models: C51 (categorical) and QR-DQN (quantile regression).
  4. Wasserstein Bonus: Instead of using simple variance, the agent uses the average 1-Wasserstein distance between ensemble members as an exploration bonus. This metric is naturally suited for comparing probability distributions.
Illustration of the projection mixture operator: a distribution is updated via the Bellman operator, projected onto different representations, and recombined.

Key Insight: Harnessing Inductive Bias for Diversity

The core breakthrough of this paper is realizing that the "limitations" or biases of different projections can be leveraged as a feature.

Different projections cause models to generalize very differently in regions where they lack training data. By combining these distinct biases in an ensemble, the disagreement between them becomes a highly reliable signal for novel, unexplored states.

Toy regression demonstrating how Categorical (blue) and Quantile (orange) projections lead to visibly different predictive distributions in areas with sparse training data.

Furthermore, the choice of projection fundamentally reshapes how the model predicts higher-order moments like variance, skewness, and kurtosis in novel states. This variance in higher-order statistics is crucial for risk-sensitive exploration. Surprisingly, the authors found that an ensemble of just 4 diverse distributional models (2 Categorical, 2 Quantile) often outperformed standard ensembles of 20 identical models, proving that diversity of representation is more powerful than sheer scale.

Empirical Performance

The authors evaluated PE-DQN across several rigorous benchmarks:

  • bsuite (Behavior Suite): PE-DQN outperformed strong baselines (like Bootstrapped DQN and IDS) across almost all 468 environments, with the most significant gains in the "Exploration" and "Generalization" categories.
  • Deep Sea: In this notoriously difficult "chain" environment—scaled up to $100 \times 100$ to test extreme exploration—PE-DQN reduced median regret by roughly half compared to the next best baseline.
  • VizDoom (MyWayHome): In a high-dimensional 3D visual navigation task, PE-DQN was the only algorithm that successfully found the goal across all 10 test seeds in the "Very Sparse" reward setting, where baselines completely failed.

Limitations and Future Directions

While highly effective, the approach has a few limitations. The current work uses a simple uniform mixture of ensemble members; future iterations could employ a learned or weighted collection of models to exploit the strengths of specific projections in different regions of the state space. Additionally, because the method relies on finite-dimensional representations (a fixed number of atoms or quantiles), there is an irreducible approximation error, meaning ensemble disagreement never truly reaches zero. Finally, distributional models carry a higher computational overhead than standard Q-learning, though the authors note that wall-clock times remain competitive with other ensemble methods.

Summary updated

Abstract

In contrast to classical reinforcement learning, distributional RL algorithms aim to learn the distribution of returns rather than their expected value. Since the nature of the return distribution is generally unknown a priori or arbitrarily complex, a common approach finds approximations within a set of representable, parametric distributions. Typically, this involves a projection of the unconstrained distribution onto the set of simplified distributions. We argue that this projection step entails a strong inductive bias when coupled with neural networks and gradient descent, thereby profoundly impacting the generalization behavior of learned models. In order to facilitate reliable uncertainty estimation through diversity, this work studies the combination of several different projections and representations in a distributional ensemble. We establish theoretical properties of such projection ensembles and derive an algorithm that uses ensemble disagreement, measured by the average $1$-Wasserstein distance, as a bonus for deep exploration. We evaluate our algorithm on the behavior suite benchmark and find that diverse projection ensembles lead to significant performance improvements over existing methods on a wide variety of tasks with the most pronounced gains in directed exploration problems.
BibTeX
@inproceedings{zanger2024diverse,
  author = {Moritz Akiya Zanger and Wendelin Boehmer and Matthijs T. J. Spaan},
  title = {{Diverse Projection Ensembles for Distributional Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=qe49ybvvPs},
}
Figure