Risk-Sensitive Reinforcement Learning with Function Approximation: A Debiasing Approach

Yingjie Fei, Zhuoran Yang, Zhaoran WangICML 2021 · OpenReview

Introduction

This paper introduces the first risk-sensitive reinforcement learning (RL) algorithms that work with function approximation while providing provable sub-linear regret guarantees. By employing a clever "debiasing" strategy, the authors overcome the mathematical hurdles of non-linear risk objectives, allowing RL to scale to complex, large-scale environments without losing theoretical rigor.

The Challenge of Risk-Sensitive RL

Standard RL is typically "risk-neutral," meaning it aims to maximize the expected total reward. However, in high-stakes fields like finance, healthcare, or autonomous driving, we often care deeply about the variance of outcomes. Depending on the application, we might be risk-averse (avoiding rare but catastrophic failures) or risk-seeking (chasing high-upside "lottery ticket" outcomes).

The gold standard for modeling this variance is the entropic risk measure, which uses a risk parameter $\beta$ to weight outcomes. The primary challenge is that this objective is non-linear (involving a log-sum-exp calculation). In standard RL, "function approximation" (like neural networks or linear models) is used to handle massive state spaces, usually via least-squares regression. Because the risk-sensitive objective is non-linear, standard regression techniques break down, suffering from massive biases and improper regularization.

Prior to this work, risk-sensitive RL was mostly limited to "tabular" settings (small, discrete grids) or lacked "regret bounds"—the mathematical proof that an algorithm actually learns efficiently over time.

Proposed Framework: MetaRSVI

The authors propose a high-level framework called MetaRSVI (Meta Risk-Sensitive Value Iteration), which they then specialize into two concrete algorithms.

Algorithm 1: The MetaRSVI framework provides a high-level structure for both the linear and general algorithms.

1. RSVI-L (Linear Function Approximation)

Designed for environments where transitions can be modeled linearly, RSVI-L introduces three key innovations to handle the non-linear risk objective:

  • Debiased Regression: Instead of regressing on raw values, it transforms features and targets into a "shifted exponential" space ($e^{\beta V} - 1$).
  • Risk-Adaptive Regularization: It uses a regularization parameter $\lambda$ that specifically scales with the risk sensitivity $\beta$ to maintain mathematical stability.
  • Debiased Backward Induction: It adds a correction term (+1) during the value update to align the transformed regression results back with the standard Bellman equation.

2. RSVI-G (General Function Approximation)

RSVI-G is a more flexible version that works with any general class of functions. It uses "confidence balls" and optimization problems to maintain optimism, performing what the authors term "implicit debiasing."

Key Theoretical Insights

  • The Synergy of Debiasing and Regularization: A major insight in the paper is that simply fixing the regression targets is insufficient; the regularization must also change. The authors demonstrate that the regularization $\lambda$ must be carefully tuned to the scale of the debiased features to prevent the algorithm from becoming unstable or "blind."
  • Recovering Structure from Non-linearity: While the entropic risk objective is non-linear, the authors realized that by working with the exponential of the value function, they could recover a structure that behaves enough like a linear system to allow for efficient estimation.
  • A Unified Theory: The authors developed a single "Optimism Condition." If an algorithm satisfies this condition, it is guaranteed to have good regret. This unified framework allowed them to prove the effectiveness of both the simple linear algorithm and the complex general one using the same theoretical toolkit.

Main Results

This is a theoretical work focused on algorithm design and proving mathematical convergence rates (regret bounds) rather than empirical benchmarking on specific datasets. The key theoretical results include:

  • First Sub-linear Regret: The paper proves that both algorithms achieve a regret of $\tilde{O}(\sqrt{K})$, where $K$ is the number of episodes. This guarantees that the more the agent plays, the closer it gets to the optimal risk-sensitive policy at a predictable, efficient rate.
  • Near Optimality: For small risk levels, their bounds match the best-known theoretical limits for risk-sensitive RL.
  • Seamless Transition: The authors show that as the risk parameter $\beta$ approaches zero, their algorithms naturally "collapse" back into standard, state-of-the-art risk-neutral RL algorithms.

Limitations and Future Work

  • Computational Cost of RSVI-G: While RSVI-G is theoretically powerful, it requires solving complex optimization problems (quadratic and linear programs) at every step. This might be too slow for real-time applications without further algorithmic optimization.
  • Known Rewards: For simplicity, the paper assumes the reward function is already known. Extending this framework to handle unknown, noisy rewards in a risk-sensitive context is a logical next step.
  • Model-Free Variants: These algorithms are "model-based" (they attempt to estimate the environment's transitions). Developing "model-free" versions (like risk-sensitive Q-learning) with function approximation remains an open challenge for future research.

Summary updated

Abstract

We study function approximation for episodic reinforcement learning with entropic risk measure. We first propose an algorithm with linear function approximation. Compared to existing algorithms, which suffer from improper regularization and regression biases, this algorithm features debiasing transformations in backward induction and regression procedures. We further propose an algorithm with general function approximation, which features implicit debiasing transformations. We prove that both algorithms achieve a sublinear regret and demonstrate a trade-off between generality and efficiency. Our analysis provides a unified framework for function approximation in risk-sensitive reinforcement learning, which leads to the first sublinear regret bounds in the setting.
BibTeX
@inproceedings{fei2021risk,
  author = {Yingjie Fei and Zhuoran Yang and Zhaoran Wang},
  title = {{Risk-Sensitive Reinforcement Learning with Function Approximation: A Debiasing Approach}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=TxwUFsANUm},
}
Figure