A Unified Uncertainty-Aware Exploration: Combining Epistemic and Aleatory Uncertainty

Parvin Malekzadeh, Ming Hou, Konstantinos N. Plataniotis

TL;DR

This paper introduces a novel algorithm, UUaE, that unifies the estimation and utilization of both epistemic (model uncertainty) and aleatory (inherent randomness) uncertainties in reinforcement learning (RL). By developing a belief-based distributional RL framework, UUaE enables more robust and sample-efficient exploration in complex environments, outperforming existing methods that handle these uncertainties separately or additively.

What Problem Does This Paper Address?

Reinforcement learning agents often struggle with exploration, especially in real-world scenarios where collecting data is expensive or risky. Uncertainty-aware exploration aims to tackle this by using estimates of uncertainty to guide the agent's learning. However, a key challenge lies in effectively combining two types of uncertainty: epistemic uncertainty, which reflects the agent's lack of knowledge about the environment and decreases with more data, and aleatory uncertainty, which stems from the environment's inherent randomness and cannot be reduced.

Existing methods often estimate these uncertainties separately and combine them additively. This can lead to problems: focusing solely on epistemic uncertainty might make an agent overly optimistic and take excessive risks in areas with high aleatory uncertainty, leading to instability. Conversely, simply adding them might not accurately capture their combined impact. This paper addresses the difficulty of jointly quantifying and leveraging both epistemic and aleatory uncertainties for effective and stable exploration.

Methods

The core of the proposed solution is the Unified Uncertainty-aware Exploration (UUaE) algorithm, built upon a novel belief-based distributional RL framework. Here's a breakdown:

  • Belief-based Distributional RL: Instead of directly estimating the return distribution, UUaE models the uncertainty in the parameters of this distribution. It maintains a "belief distribution" over these parameters, effectively capturing epistemic uncertainty. This belief is then used to derive a belief-based distributional Bellman equation, which formally connects aleatory and epistemic uncertainties.
  • Unified Uncertainty Estimation: To make the belief distribution computationally tractable, it's modeled as a mixture of Dirac deltas. From this, Moment-Generating Function (MGF) features are derived. These features provide a rich statistical summary of the belief and allow the algorithm to estimate the composite uncertainty.
  • Composite Uncertainty-Aware Exploration: The exploration strategy then uses this unified uncertainty estimate. It aims to maximize expected rewards while simultaneously minimizing the composite uncertainty (which is interpreted as the variance of the return distribution, influenced by both epistemic and aleatory components). This is achieved by selecting actions that balance expected return and uncertainty, formulated as E[Return] - Var(Return).

Datasets

The paper evaluates UUaE on:

  • Two Atari games: Atari-Asterix and Atari-Seaquest. These tasks are noted for their sparse reward functions, making exploration particularly challenging.
  • An autonomous vehicle driving simulator: A highway domain where rewards are designed to penalize unsafe driving. This task introduces risk-sensitivity challenges.

The authors do not introduce new benchmarks but use these existing environments to demonstrate the effectiveness of their approach in scenarios that highlight exploration and risk challenges.

Key Insights and Ideas

  • Theoretical Connection: The paper establishes a formal theoretical link between epistemic and aleatory uncertainties, moving beyond simple additive combinations. This is crucial for a more nuanced understanding of risk in RL.
  • Belief as a Unified Representation: Using a belief distribution over return distribution parameters elegantly unifies the estimation of both uncertainty types.
  • MGFs for Tractability: Employing MGF features derived from Dirac delta mixtures is a clever way to summarize complex belief distributions into a format usable by neural networks, enabling efficient learning.
  • Risk-Sensitive Exploration: The proposed exploration strategy directly incorporates the composite uncertainty, leading to more stable and less risk-seeking behavior in high-uncertainty regions, especially those with high aleatory uncertainty.

Key Results

UUaE consistently outperforms three baseline methods (SUNRISE, DLTV, and IV-DQN) across all tested environments. These baselines represent approaches that focus on epistemic uncertainty, aleatory uncertainty, or an additive combination, respectively.

  • The method demonstrates substantial improvements in stability and sample efficiency.
  • In the autonomous driving task, where SUNRISE (which doesn't consider aleatory uncertainty) deteriorates, UUaE shows good risk-sensitive performance, highlighting its advantage in safety-critical applications.
  • The learning curves (Figure 2) show that UUaE achieves higher cumulative rewards faster and with lower variance compared to the baselines.

Limitations and Future Work

  • Hyperparameter Tuning: The performance of UUaE is sensitive to hyperparameters like L (GMM components), K (Dirac delta mixtures), and M (MGF features). While the paper suggests a trade-off between computation time and performance, finding optimal values might require careful tuning.
  • Computational Cost: While the MGF approach makes the belief tractable, it still adds computational overhead compared to methods that ignore one type of uncertainty.

The authors suggest that their work paves the way for future research on designing exploratory policies that more effectively consider the combined effects of epistemic and aleatory uncertainties. The unified framework could be extended to other RL algorithms and more complex environments.

Summary updated

Abstract

Exploration is a significant challenge in practical reinforcement learning (RL), and uncertainty-aware exploration that incorporates the quantification of epistemic and aleatory uncertainty has been recognized as an effective exploration strategy. However, capturing the combined effect of aleatory and epistemic uncertainty for decision-making is difficult. Existing works estimate aleatory and epistemic uncertainty separately and consider the composite uncertainty as an additive combination of the two. Nevertheless, the additive formulation leads to excessive risk-taking behavior, causing instability. In this paper, we propose an algorithm that clarifies the theoretical connection between aleatory and epistemic uncertainty, unifies aleatory and epistemic uncertainty estimation, and quantifies the combined effect of both uncertainties for a risk-sensitive exploration. Our method builds on a novel extension of distributional RL that estimates a parameterized return distribution whose parameters are random variables encoding epistemic uncertainty. Experimental results on tasks with exploration and risk challenges show that our method outperforms alternative approaches.
BibTeX
@inproceedings{malekzadeh2023unified,
  author = {Parvin Malekzadeh and Ming Hou and Konstantinos N. Plataniotis},
  title = {{A Unified Uncertainty-Aware Exploration: Combining Epistemic and Aleatory Uncertainty}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=pf4iIvXWl9},
}
Figure