Abstract
In safety-critical domains where online data collection is infeasible, offline reinforcement learning (RL) offers an attractive alternative but only if policies deliver high returns without incurring catastrophic lower-tail risk. Prior work on risk-averse offline RL achieves safety at the cost of value or model-based pessimism, and restricted policy classes that limit policy expressiveness, whereas diffusion/flow-based expressive generative policies trained with a behavioral-cloning (BC) objective have been used only in risk-neutral settings. Here, we address this gap by introducing the Risk-Aware Multimodal Actor-Critic (RAMAC), which couples an expressive generative actor with a distributional critic and, to our knowledge, is the first model-free approach that learns risk-aware expressive generative policies. RAMAC differentiates a composite objective that adds a Conditional Value-at-Risk (CVaR) term to a BC loss, achieving risk-sensitive learning in complex multimodal scenarios. Since out-of-distribution (OOD) actions are a major driver of catastrophic failures in offline RL, we further analyze OOD behavior under prior-anchored perturbation schemes from recent BC-regularized risk-averse offline RL. This clarifies why a behavior-regularized objective that directly constrains the expressive generative policy to the dataset support provides an effective, risk-agnostic mechanism for suppressing OOD actions in modern expressive policies. We instantiate RAMAC with a diffusion-based actor, using it both to illustrate the analysis in a 2-D risky bandit and to deploy OOD-action detectors on Stochastic-D4RL benchmarks, empirically validating our insights. Across these tasks, we observe consistent gains in $\mathrm{CVaR}_{0.1}$ while maintaining strong returns.
Introduction
In safety-critical applications such as autonomous driving or robotic surgery, "trial and error" in the real world is prohibitively dangerous. Offline Reinforcement Learning (RL) addresses this by learning from pre-collected datasets. However, offline RL must navigate two primary challenges: avoiding "out-of-distribution" (OOD) actions—actions not represented in the training data—and managing "tail risk," the rare but catastrophic failures that standard average-reward agents often overlook.
Current state-of-the-art methods often face a trade-off between expressiveness and safety. While modern generative policies (e.g., Diffusion-QL) excel at capturing complex, multimodal data distributions, they are typically risk-neutral. Conversely, existing risk-averse methods often rely on restricted policy classes (like single Gaussians) or overly pessimistic value estimates that can lead to overly conservative behavior.
The RAMAC Framework
The Risk-Aware Multimodal Actor-Critic (RAMAC) framework bridges this gap by combining highly expressive generative models with risk-aware distributional critics.
RAMAC consists of two core components:
- The Actor: An expressive generative model (implemented as either a Diffusion model in RADAC or Flow-matching in RAFMAC). This actor is fully differentiable, allowing the critic's signal to steer the generation process toward safer regions.
- The Critic: An Implicit Quantile Network (IQN) that predicts the entire distribution of possible returns rather than a single scalar value.
The actor is trained using a composite objective: $$L = L_{BC} + \eta L_{Risk}$$ where $L_{BC}$ is a Behavioral Cloning loss that anchors the policy to the training data manifold, and $L_{Risk}$ is the Conditional Value-at-Risk (CVaR), focusing on the bottom $\alpha$-fraction of outcomes. By backpropagating the CVaR signal through the generative process, RAMAC reallocates probability mass away from risky modes and toward safe, high-return ones.
Theoretical Insights: Behavior Regularization as Safety
A key contribution of the RAMAC paper is the analysis of OOD behavior. Previous risk-averse methods often used "prior-anchored perturbation," which adds a small nudge to a fixed behavior policy. The authors demonstrate geometrically that this can inadvertently push the agent into dangerous OOD territory when the data support is thin or non-convex.
In contrast, RAMAC uses direct Behavioral Cloning (BC) on expressive models. The authors prove that this provides a theoretical upper bound on the probability of taking OOD actions, making BC a robust, risk-agnostic safety mechanism. This allows the agent to maintain high expressiveness—representing multiple safe paths—while precisely avoiding dangerous OOD regions.
Experimental Evaluation
The researchers evaluated RAMAC on a 2D synthetic "Risky Bandit" task and the Stochastic-D4RL benchmarks. Stochastic-D4RL modifies standard locomotion tasks (Hopper, Walker2d, HalfCheetah) by adding rare, heavy-tailed penalties for hazardous states (e.g., excessive speed or dangerous torso angles).
Key results include:
- Superior Risk Control: RADAC (the diffusion variant) consistently outperformed baselines in $CVaR_{0.1}$ while maintaining competitive mean returns.
- OOD Suppression: In the Walker2d-Medium-Expert task, RADAC reduced the OOD action rate to 0.75%, compared to 10.84% for previous state-of-the-art methods like ORAAC.
- Multimodal Safety: In the Toy Bandit task, RADAC successfully identified the "safe center" while ignoring the "risky ring," without collapsing its multimodal representation.
Conclusion
RAMAC demonstrates that policy expressiveness is not an obstacle to safety but a tool for it. By coupling generative models with distributional critics and behavior regularization, RAMAC achieves a new state-of-the-art in risk-aware offline RL. Future work will explore reducing the computational overhead of backpropagating through diffusion steps and extending the framework to model-based settings for even greater sample efficiency.
Summary updated