One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning

Marc Rigter, Bruno Lacerda, Nick Hawes

Abstract

Offline reinforcement learning (RL) must navigate the dual challenges of distributional shift—where agents exploit inaccuracies in learned models—and environment stochasticity, which can lead to catastrophic failures in safety-critical domains. We introduce 1R2R (One Risk to Rule Them All), a model-based offline RL algorithm that unifies these challenges under a single risk-sensitive framework. By treating both epistemic uncertainty (lack of data) and aleatoric uncertainty (environment noise) as forms of risk, 1R2R employs an ensemble of models and risk-sensitive re-weighting to learn robust policies. Our results across diverse benchmarks, including medical and financial simulations, demonstrate that 1R2R achieves state-of-the-art performance by naturally avoiding both "hallucinated" rewards and inherently dangerous actions.

Introduction

Reinforcement learning typically relies on active exploration, but in domains like autonomous driving or healthcare, learning must occur from fixed, "offline" datasets. This setting introduces two major hurdles. First, distributional shift occurs when an agent selects actions poorly represented in the training data, leading the learned model to "hallucinate" high rewards. Second, environment stochasticity requires agents to be cautious of rare but catastrophic outcomes.

Traditionally, these issues are addressed separately: constraints or penalties keep agents near the data, while risk-sensitive objectives manage noise. 1R2R proposes a unified perspective: what if both are treated as risk? By being risk-averse to the uncertainty in a model ensemble, an agent can simultaneously avoid out-of-distribution states and high-variance environment transitions.

Methods: The 1R2R Framework

1R2R is a model-based approach that learns a "world model" from offline data and subsequently trains a policy within this simulated environment.

The 1R2R workflow: from ensemble training to risk-sensitive RL training.

The core of 1R2R lies in three components:

  1. Ensemble Modeling: Instead of a single model, an ensemble of neural networks is trained. Disagreement among these models serves as a proxy for epistemic uncertainty.
  2. Adversarial Transitions: During policy training, the agent does not experience the mean transition. Instead, 1R2R uses a risk measure, such as Conditional Value at Risk (CVaR), to "tilt" the transition distribution. This forces the agent to plan against worst-case outcomes within the ensemble's predictions.
  3. Unified Risk Management: Because the ensemble captures both model uncertainty and environment noise, a risk-averse agent naturally avoids data-poor regions (where models disagree) and inherently risky actions (where the environment is volatile).

Key Insights

A central contribution of this work is the realization that epistemic uncertainty is risk. Framing distributional shift as a risk-management problem removes the need for complex, explicit constraints on the policy. By changing the "physics" of the simulation to be pessimistic through adversarial sampling, 1R2R simplifies the offline RL pipeline while maintaining robustness.

Visualization of the risk-sensitive value function Q(s,a) showing how it penalizes regions with high uncertainty.

Experimental Evaluation

The authors evaluated 1R2R on several benchmarks, including standard D4RL MuJoCo tasks, a stochastic version of MuJoCo with random "wind" forces, and high-stakes simulations for Currency Exchange and HIV Treatment.

Results

  • Robustness Across Domains: 1R2R matched or exceeded the performance of top-tier risk-neutral algorithms (e.g., IQL, CQL) on deterministic tasks, proving that its conservative nature does not sacrifice efficiency.
  • Superiority in Stochastic Settings: In the HIV and Currency Exchange domains, 1R2R significantly outperformed existing risk-sensitive methods like ORAAC and CODAC, particularly in optimizing for worst-case outcomes.
  • Ablation Studies: Removing the risk-aversion component led to performance collapse, confirming that risk-sensitive sampling is the primary mechanism preventing model exploitation.

Conclusion and Future Work

1R2R demonstrates that a unified risk-sensitive approach can effectively solve the core challenges of offline RL. While the current implementation assumes Gaussian noise and carries the computational overhead of ensemble modeling, it provides a conceptually simple and high-performing alternative to hybrid constraint-based methods. Future work will explore non-Gaussian noise distributions and the theoretical bridge between dynamic and static risk measures.

Summary updated

BibTeX
@inproceedings{rigter2023one,
  author = {Marc Rigter and Bruno Lacerda and Nick Hawes},
  title = {{One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=qZjl2TKvUY},
}
Figure