Diverse Projection Ensembles for Distributional Reinforcement Learning

Moritz A. Zanger, Wendelin Böhmer, Matthijs T. J. Spaan

TL;DR

This paper introduces "Diverse Projection Ensembles" (PE-DQN) for distributional reinforcement learning (RL). By combining different ways of approximating return distributions within an ensemble, PE-DQN achieves more reliable uncertainty estimation and significantly improves exploration performance on challenging tasks, outperforming existing methods.

What Problem Does This Paper Address?

Reinforcement learning agents aim to maximize cumulative rewards by balancing exploration (trying new actions) and exploitation (using known good actions). Distributional RL goes beyond predicting the average reward to learning the entire distribution of possible future rewards. This is useful for risk-sensitive decision-making and better uncertainty estimation. However, representing these complex distributions often involves "projecting" them onto simpler, predefined forms (like categorical or quantile distributions). The authors argue that this projection step, especially when combined with neural networks and gradient descent, introduces a strong inductive bias that can hinder generalization and accurate uncertainty estimation. They aim to address this by creating diverse ensembles of distributional learners, where each member uses a different projection method, to better capture uncertainty and drive exploration.

Methods

The core contribution is the concept of Distributional Projection Ensembles. Instead of relying on a single type of distributional projection (e.g., only categorical or only quantile), PE-DQN uses an ensemble where each member learns a different projection of the return distribution.

Key techniques include:

  • Projection Mixture Operator: A theoretical framework for combining multiple distributional projections.
  • Optimism-Based Exploration: Leveraging ensemble disagreement (measured by the 1-Wasserstein distance) as an exploration bonus. This bonus encourages the agent to visit states where the ensemble members disagree, indicating uncertainty.
  • Uncertainty Propagation Scheme: A method to estimate epistemic uncertainty (reducible uncertainty due to lack of knowledge) by relating one-step distributional TD errors to the overall uncertainty.
  • PE-DQN Algorithm: A deep RL implementation that combines categorical and quantile projections within a neural network ensemble. It uses a novel loss function that incorporates both projection types and learns an exploration bonus based on ensemble disagreement.

Datasets

  • Behavior Suite (bsuite): A collection of 23 RL tasks with variations, used for evaluating general performance and exploration capabilities.
  • VizDoom (MyWayHome tasks): A high-dimensional visual domain used to test the algorithm's performance in complex environments with varying reward sparsity.

Key Insights and Ideas

  • Projection Inductive Bias: The choice of distributional projection method significantly impacts how an agent generalizes to unseen states and actions. Different projections lead to different "generalization behaviors."
  • Diversity is Key for Uncertainty: Combining diverse projection methods within an ensemble allows for more robust estimation of epistemic uncertainty. When different projection types disagree, it signals a region of high uncertainty.
  • Ensemble Disagreement as an Exploration Bonus: The authors show that ensemble disagreement, quantified by the Wasserstein distance, can be effectively used as an intrinsic reward to guide exploration towards uncertain regions.
  • Theoretical Guarantees: The paper provides theoretical properties for projection ensembles, including convergence conditions and bounds on approximation errors, which are crucial for understanding their behavior.

Key Results

  • Improved Exploration: PE-DQN significantly outperforms baseline methods on exploration-heavy tasks, particularly in the Behavior Suite and VizDoom environments.
  • Robust Performance: The algorithm shows strong performance across a wide range of tasks, even outperforming baselines with significantly larger ensemble sizes.
  • Scalability: PE-DQN scales well to larger and more complex environments, demonstrating reduced regret in deep-sea environments as grid sizes increase.
  • Reliable Uncertainty Estimation: The diverse projection ensembles lead to more reliable uncertainty estimates, which in turn drive more effective exploration.

Limitations and Future Work

  • Uniform Mixture Assumption: The current work assumes a uniform mixture over distributional ensembles. Future work could explore using a diverse collection of models more conservatively, exploiting the strengths of particular models in specific state-action regions.
  • Computational Cost: While PE-DQN shows improved performance, using ensembles inherently increases computational cost. Further research could focus on optimizing ensemble efficiency.
  • Hyperparameter Sensitivity: Like many RL algorithms, PE-DQN's performance can be sensitive to hyperparameters, particularly the exploration bonus parameter $\beta$.

The authors suggest that their work opens up promising avenues for future research in leveraging diverse distributional representations for improved RL performance.

Summary updated

Abstract

In contrast to classical reinforcement learning, distributional reinforcement learning algorithms aim to learn the distribution of returns rather than their expected value. Since the nature of the return distribution is generally unknown a priori or arbitrarily complex, a common approach finds approximations within a set of representable, parametric distributions. Typically, this involves a projection of the unconstrained distribution onto the set of simplified distributions. We argue that this projection step entails a strong inductive bias when coupled with neural networks and gradient descent, thereby profoundly impacting the generalization behavior of learned models. In order to facilitate reliable uncertainty estimation through diversity, this work studies the combination of several different projections and representations in a distributional ensemble. We establish theoretical properties of such projection ensembles and derive an algorithm that uses ensemble disagreement, measured by the average $1$-Wasserstein distance, as a bonus for deep exploration. We evaluate our algorithm on the behavior suite benchmark and find that diverse projection ensembles lead to significant performance improvements over existing methods on a wide variety of tasks with the most pronounced gains in directed exploration problems.
BibTeX
@inproceedings{zanger2024diverse,
  author = {Moritz A. Zanger and Wendelin Böhmer and Matthijs T. J. Spaan},
  title = {{Diverse Projection Ensembles for Distributional Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=qe49ybvvPs},
}
Figure