How to Enable Uncertainty Estimation in Proximal Policy Optimization

Eugene Bykovets, Yannick Metz, Mennatallah El-Assady, Daniel A. Keim, Joachim M. Buhmann

Abstract

While deep reinforcement learning (RL) agents have showcased strong results across many domains, a major concern is their inherent opaqueness and the safety of such systems in real-world use cases. To overcome these issues, we need agents that can quantify their uncertainty and detect out-of-distribution (OOD) states. Existing uncertainty estimation techniques, like Monte-Carlo Dropout or Deep Ensembles, have not seen widespread adoption in on-policy deep RL. We posit that this is due to two reasons: concepts like uncertainty and OOD states are not well defined compared to supervised learning, especially for on-policy RL methods. Secondly, available implementations and comparative studies for uncertainty estimation methods in RL have been limited. To overcome the first gap, we propose definitions of uncertainty and OOD for Actor-Critic RL algorithms, namely, proximal policy optimization (PPO), and present possible applicable measures. In particular, we discuss the concepts of value and policy uncertainty. The second point is addressed by implementing different uncertainty estimation methods and comparing them across a number of environments. The OOD detection performance is evaluated via a custom evaluation benchmark of in-distribution (ID) and OOD states for various RL environments. We identify a trade-off between reward and OOD detection performance. To overcome this, we formulate a Pareto optimization problem in which we simultaneously optimize for reward and OOD detection performance. We show experimentally that the recently proposed method of Masksembles strikes a favourable balance among the survey methods, enabling high-quality uncertainty estimation and OOD detection while matching the performance of original RL agents.

Introduction: Safety and Uncertainty in Deep RL

Deep Reinforcement Learning (RL) has achieved remarkable success in complex domains, yet its "black box" nature poses significant risks for real-world deployment. A critical requirement for safe autonomous systems is the ability to quantify uncertainty and detect out-of-distribution (OOD) states—situations the agent did not encounter during training. Without this "sense of doubt," an agent might confidently execute catastrophic actions when faced with novel environments.

The Challenge of On-Policy Learning

In supervised learning, OOD detection is a mature field. However, RL introduces unique complexities. As shown in Figure 1, RL paradigms differ significantly in how they collect and utilize data. On-policy algorithms like Proximal Policy Optimization (PPO) are particularly challenging for uncertainty estimation.

Comparison of RL paradigms: (a) Offline/Batch-RL, (b) Off-policy RL, and (c) On-Line RL.

PPO relies on a "closeness" constraint to maintain training stability, requiring the agent to learn from data generated by its most recent policy. Traditional uncertainty methods like Deep Ensembles often fail in this setting because independent sub-models tend to diverge, making the collected on-policy data irrelevant for some ensemble members. Conversely, methods like Monte-Carlo Dropout can introduce excessive noise, degrading the agent's ability to maximize rewards. The authors identify a fundamental trade-off: model diversity is needed to detect OOD states, but too much diversity breaks the on-policy training loop.

Methodology: Masksembles and Uncertainty Metrics

To bridge this gap, the authors propose a framework for uncertainty estimation tailored to Actor-Critic RL. They define two primary metrics:

  1. Value Uncertainty: The standard deviation among multiple "critic" sub-models regarding expected future rewards.
  2. Policy Uncertainty: The disagreement among multiple "actor" sub-models regarding action selection.

The core architectural contribution is the adaptation of Masksembles to the PPO framework. As illustrated in Figure 2, Masksembles provide a middle ground between the high computational cost of full ensembles and the instability of dropout.

Comparison of uncertainty estimation architectures: (a) Baseline, (b) MC Dropout, (c) MC Dropconnect, (d) Ensembles, and (e) Masksembles.

Masksembles utilize a fixed set of binary masks to create partially overlapping sub-models within a single network. This structure allows for "smooth interpolation" between highly correlated sub-models (promoting training stability) and diverse sub-models (enabling effective OOD detection). To find the optimal balance between reward and safety, the authors employ Pareto optimization using the Optuna framework.

Experimental Results

The proposed methods were evaluated on a custom benchmark involving MuJoCo (continuous control) and Atari (discrete actions) environments. OOD states were generated by perturbing physics parameters in MuJoCo and using unseen high-level frames from the Atari-HEAD dataset.

Key findings include:

  • Superior OOD Detection: Masksembles consistently outperformed MC Dropout and standard Ensembles, achieving ROC-AUC scores between 0.76 and 0.88 in MuJoCo tasks.
  • Performance Preservation: Unlike MC Dropout, which often significantly reduced agent rewards, Masksembles maintained performance levels comparable to vanilla PPO baselines.
  • Value Uncertainty Utility: Disagreement between critics (Value Uncertainty) was found to be a more reliable indicator of environmental changes than actor disagreement, serving as a better "canary in the coal mine."
  • Efficiency: Masksembles generate uncertainty estimates in a single forward pass, making them suitable for real-time inference.

Conclusion

By integrating Masksembles into PPO, this work demonstrates that on-policy RL agents can be equipped with robust uncertainty estimation capabilities. This approach strikes a favorable balance between reward maximization and safety-critical OOD detection, paving the way for more reliable real-world RL applications. Future work may explore disentangling aleatoric and epistemic uncertainty or using these measures to drive active exploration during training.

Summary updated

Abstract

While deep reinforcement learning (RL) agents have showcased strong results across many domains, a major concern is their inherent opaqueness and the safety of such systems in real-world use cases. To overcome these issues, we need agents that can quantify their uncertainty and detect out-of-distribution (OOD) states. Existing uncertainty estimation techniques, like Monte-Carlo Dropout or Deep Ensembles, have not seen widespread adoption in on-policy deep RL. We posit that this is due to two reasons: concepts like uncertainty and OOD states are not well defined compared to supervised learning, especially for on-policy RL methods. Secondly, available implementations and comparative studies for uncertainty estimation methods in RL have been limited. To overcome the first gap, we propose definitions of uncertainty and OOD for Actor-Critic RL algorithms, namely, proximal policy optimization (PPO), and present possible applicable measures. In particular, we discuss the concepts of value and policy uncertainty. The second point is addressed by implementing different uncertainty estimation methods and comparing them across a number of environments. The OOD detection performance is evaluated via a custom evaluation benchmark of in-distribution (ID) and OOD states for various RL environments. We identify a trade-off between reward and OOD detection performance. To overcome this, we formulate a Pareto optimization problem in which we simultaneously optimize for reward and OOD detection performance. We show experimentally that the recently proposed method of Masksembles strikes a favourable balance among the survey methods, enabling high-quality uncertainty estimation and OOD detection while matching the performance of original RL agents.
BibTeX
@inproceedings{bykovets2023how,
  author = {Eugene Bykovets and Yannick Metz and Mennatallah El-Assady and Daniel A. Keim and Joachim M. Buhmann},
  title = {{How to Enable Uncertainty Estimation in Proximal Policy Optimization}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=XKq49kJ5mZX},
}
Figure