Risk-sensitive Reinforcement Learning Based on Convex Scoring Functions

Shanyu Han, Yang Liu, Xiang YuarXiv 2025 · DOI · arXiv

This paper, titled "Risk-sensitive Reinforcement Learning Based on Convex Scoring Functions," tackles one of the biggest hurdles in deploying AI to high-stakes environments: how to make agents that aren't just "smart," but also "cautious."

TL;DR

Standard Reinforcement Learning (RL) is risk-neutral, focusing only on average rewards, which is dangerous in fields like finance or medicine. This paper introduces a unified framework using convex scoring functions to handle a wide variety of risk measures (like variance and Expected Shortfall) while solving the "time-inconsistency" problem through state augmentation and a novel Actor-Critic algorithm.

What Problem Does This Paper Address?

Most RL algorithms aim to maximize the expected cumulative reward. While this works for games, it fails in the real world where a single "tail event" (like a stock market crash or a car accident) can be catastrophic.

The challenge in building risk-averse RL is twofold:

  1. Diversity of Risk: Different industries care about different risks. Some want to minimize variance; others care only about the "worst-case" 5% of outcomes (Expected Shortfall).
  2. Time-Inconsistency: Many popular risk measures are "time-inconsistent." This means a decision that looks optimal at the start of an episode might no longer look optimal halfway through. This breaks the Bellman Equation, the fundamental mathematical tool used to solve RL problems.

Methods

The authors propose a three-part solution to bring mathematical rigor to risk-sensitive RL:

  • Convex Scoring Functions: Instead of building a custom algorithm for every specific risk measure, they use "convex scoring functions"—a general tool from statistics. This allows their framework to support Variance, Expected Shortfall (ES), entropic Value-at-Risk (EVaR), and mean-risk utility under one umbrella.
  • State Augmentation: To fix the time-inconsistency issue, they "augment" the state space. They track not just the current environment state ($S_t$), but also the accumulated cost ($Y_t$) and an auxiliary variable ($v$). This transformation turns a non-Markovian risk problem into a standard Markov Decision Process (MDP) that can be solved with dynamic programming.
  • Two-Stage Actor-Critic: They develop a customized Actor-Critic (AC) architecture.
  • The Critic learns to approximate the value function across the augmented state space.
  • The Actor updates the policy using gradients.
  • An Outer Optimization step uses Stochastic Gradient Descent (SGD) or alternating minimization to find the optimal auxiliary variable ($v$), which essentially "tunes" the agent's risk threshold.

Datasets

The paper evaluates the algorithm using a simulated financial environment focused on statistical arbitrage trading.

  • Asset Pricing: Modeled using an Ornstein-Uhlenbeck (OU) process, a standard stochastic model for mean-reverting prices.
  • Trading Constraints: The simulation includes transaction costs and terminal penalties for holding inventory, forcing the agent to balance profit against the risk of being stuck with a losing position.

Key Insights and Ideas

  • The "One Framework to Rule Them All": By using convex scoring functions, the authors prove that many seemingly different risk-averse strategies are actually mathematically related. This simplifies the "zoo" of risk-sensitive RL.
  • No Continuity Required: Most RL theory assumes the environment changes smoothly (continuity). A major theoretical contribution here is proving that their approximation guarantees hold even if the MDP is discontinuous, making it more applicable to "jumpy" real-world data like stock prices.
  • Efficient Sampling: Instead of the computationally expensive "simulation-upon-simulation" approach used in prior work, they propose an alternating minimization method to sample the auxiliary variable $v$, significantly speeding up training.

Key Results

The authors compared their "RL-ES" (Expected Shortfall) and "RL-Var" (Variance) agents against a standard "RL-mean" (risk-neutral) agent:

  • Tail Risk Control: The RL-ES agent achieved the lowest Expected Shortfall in out-of-sample tests. While the risk-neutral agent had a better average return, it suffered from much larger "tail losses."
  • Thinner Tails: Visualizing the distribution of total costs showed that the proposed risk-sensitive agents had significantly "thinner" tails on the loss side compared to the baseline.
  • Dynamic Conservatism: The learned policies showed that as the "accumulated cost" ($y$) increased, the agent became more conservative, actively trying to close positions to avoid further loss—a behavior not seen in standard RL.

Limitations and Future Work

  • State Space Complexity: Augmenting the state with $Y_t$ and $v$ increases the dimensionality of the input. In very complex environments (like high-dimensional robotics), this could lead to the "curse of dimensionality."
  • Hyperparameter Sensitivity: The efficiency of the auxiliary variable sampling depends on decreasing the exploration variance ($\sigma_n^2$) over time, which may require careful tuning for different applications.
  • Real-World Markets: While the OU process is a standard model, real financial markets are more chaotic. Testing the framework on historical "limit order book" data would be a logical next step to prove its robustness.

Summary updated

Abstract

We propose a reinforcement learning (RL) framework under a broad class of risk objectives, characterized by convex scoring functions. This class covers many common risk measures, such as variance, Expected Shortfall, entropic Value-at-Risk, and mean-risk utility. To resolve the time-inconsistency issue, we consider an augmented state space and an auxiliary variable and recast the problem as a two-state optimization problem. We propose a customized Actor-Critic algorithm and establish some theoretical approximation guarantees. A key theoretical contribution is that our results do not require the Markov decision process to be continuous. Additionally, we propose an auxiliary variable sampling method inspired by the alternating minimization algorithm, which is convergent under certain conditions. We validate our approach in simulation experiments with a financial application in statistical arbitrage trading, demonstrating the effectiveness of the algorithm.
BibTeX
@misc{han2025risk,
  author = {Shanyu Han and Yang Liu and Xiang Yu},
  title = {{Risk-sensitive Reinforcement Learning Based on Convex Scoring Functions}},
  year = {2025},
  eprint = {2505.04553},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.04553},
  howpublished = {arXiv},
}
Figure