TOPS: Transition-based VOlatility-controlled Policy Search and its Global Convergence

Liangliang Xu, Aiwen Jiang, Daoming Lyu, Bo LiuarXiv 2022 · arXiv

Introduction

Standard Reinforcement Learning (RL) is typically "risk-neutral," focusing on maximizing expected cumulative rewards. However, in high-stakes domains like finance, healthcare, or robotics, the variance (volatility) of outcomes is as critical as the average. Risk-averse RL seeks to maximize rewards while minimizing this volatility. The mathematical challenge lies in the fact that variance is not "time-additive," making standard Bellman equations difficult to apply. The STOPS (Short-term Volatility-controlled Policy Search) framework addresses this by providing the first global convergence guarantees for risk-averse RL using deep neural networks.

Methodology: The STOPS Framework

The authors introduce a transformation that tackles the mean-variance trade-off by using an auxiliary variable $y$ to represent the expected return. This turns a complex variance-penalized objective into a standard-looking RL problem on an "augmented" MDP.

Key technical components include:

  • Two-Layer Over-parameterized Networks: Policy and value functions are represented by wide, two-layer ReLU neural networks, allowing the use of the "Neural Tangent Kernel" (NTK) regime where the optimization landscape is more tractable.
  • Iterative Update Scheme: The algorithm alternates between updating the auxiliary variable $y$ and updating policy parameters $\theta$ using either Neural Natural Policy Gradient (NPG) or Neural Proximal Policy Optimization (PPO).

Convergence Analysis

A major contribution of this work is the rigorous mathematical framework showing that errors from policy evaluation, $y$-variable estimation, and policy updates all decay over time. This leads to global optimality. The unified theoretical structure for these proofs is summarized in the following flowchart:

Unified theoretical structure for global convergence proofs.

A critical step in this process is establishing a bound on the policy evaluation error (Lemma 2). This lemma shows that the error scales inversely with the network width and the number of TD iterations, ensuring that the "noise" from neural network approximations does not cause divergence.

Logical flow for the Lemma 2 policy evaluation bound.

Key Results

  • Convergence Rate: The authors prove that STOPS converges to the global optimum at a rate of $O(1/\sqrt{K})$, matching the known rates for standard risk-neutral neural policy gradient methods.
  • Optimality Gap: Theorems 1 and 2 demonstrate that the difference between the achieved risk-averse objective and the true global maximum vanishes as the number of iterations and the network width increase.
  • Unified Theory: The framework provides a shared theoretical structure that handles both NPG and PPO, showing both are capable of global convergence in risk-averse settings.

Conclusion and Future Work

While STOPS provides a theoretical "safety net" for deep RL, it relies on the assumption of extreme over-parameterization. Future work may extend this framework to deeper architectures like ResNets or Transformers and explore other risk measures such as Conditional Value-at-Risk (CVaR) to provide even more robust safety guarantees.

Summary updated

BibTeX
@misc{xu2022tops,
  author = {Liangliang Xu and Aiwen Jiang and Daoming Lyu and Bo Liu},
  title = {{TOPS: Transition-based VOlatility-controlled Policy Search and its Global Convergence}},
  year = {2022},
  eprint = {2201.09857},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.09857},
}
Figure