Variance Penalized On-Policy and Off-Policy Actor-Critic

Arushi Jain, Gandharv Patil, Ayush Jain, Khimya Khetarpal, Doina Precup

Abstract

Reinforcement learning algorithms are typically geared towards optimizing the expected return of an agent. However, in many practical applications, low variance in the return is desired to ensure the reliability of an algorithm. In this paper, we propose on-policy and off-policy actor-critic algorithms that optimize a performance criterion involving both mean and variance in the return. Previous work uses the second moment of return to estimate the variance indirectly. Instead, we use a much simpler recently proposed direct variance estimator which updates the estimates incrementally using temporal difference methods. Using the variance-penalized criterion, we guarantee the convergence of our algorithm to locally optimal policies for finite state action Markov decision processes. We demonstrate the utility of our algorithm in tabular and continuous MuJoCo domains. Our approach not only performs on par with actor-critic and prior variance-penalization baselines in terms of expected return, but also generates trajectories which have lower variance in the return.

Introduction

Standard Reinforcement Learning (RL) agents are typically "risk-neutral," focusing solely on maximizing the expected return. However, in high-stakes domains such as medicine, finance, or robotics, the variability of outcomes is as critical as the average performance. A robot that succeeds most of the time but fails catastrophically in a small percentage of cases is often too risky for real-world deployment.

The primary challenge in risk-sensitive RL is effectively measuring and penalizing variance during the learning process. Traditional "mean-variance" methods often rely on an indirect approach, estimating the first and second moments of rewards separately ($Var(G) = E[G^2] - E[G]^2$). This approach is notoriously unstable because it involves subtracting two large, noisy estimates. This paper introduces VPAC (Variance Penalized Actor-Critic), which utilizes a stable, "direct" variance estimator for both on-policy and off-policy learning.

Methodology

VPAC modifies the standard policy gradient objective to include a variance penalty term: $$J(\theta) = E[Q_\pi(s, a) - \psi \sigma_\pi(s, a)]$$ where $\psi$ represents a user-defined trade-off parameter between reward and risk.

Technical Innovations

  • Direct Variance Estimation: Rather than tracking the second moment, VPAC uses a Bellman-like equation to estimate variance incrementally. This direct approach is numerically more stable and handles noisy or off-policy data better than indirect methods.
  • Multi-Timescale Learning: To ensure convergence, the algorithm employs three distinct learning rates. The $Q$-value (mean) converges fastest, followed by the variance estimate ($\sigma$), and finally the policy ($\pi$). This hierarchy ensures the actor makes decisions based on stable critic statistics.
  • Off-Policy Derivation: The authors derived an off-policy version of the variance-penalized policy gradient, using importance sampling to learn risk-averse policies from data generated by risk-neutral behavior policies.

Experimental Evaluation

The approach was evaluated across several benchmarks, most notably the Tabular Four Rooms grid-world. This environment was modified to include "frozen states" that yield high-variance rewards (mean 0, but high standard deviation), forcing the agent to choose between a short, risky path and a longer, safer one.

The Four Rooms environment setup. The red 'frozen' states represent high-variance regions between the start (S) and goal (G).

Additional testing included continuous control tasks in MuJoCo (Hopper, Walker2d, HalfCheetah) and Puddle World, where VPAC was integrated into the Proximal Policy Optimization (PPO) algorithm.

Key Results and Insights

VPAC demonstrates that treating variance as a quantity satisfying its own Bellman equation significantly stabilizes the learning process. In the Four Rooms experiment, while a risk-neutral agent takes the shortest path through the high-variance region, VPAC learns to navigate the safer, more predictable hallway.

Comparison of state visitation frequency and variance. Standard Actor-Critic (top) visits high-variance regions, while VPAC (bottom) learns a lower-variance path.

Key performance metrics include:

  • Significant Risk Reduction: In the HalfCheetah environment, VPAC achieved a 93% reduction in variance compared to standard PPO with only a minor decrease in average reward.
  • Superior Stability: VPAC showed tighter performance distributions across multiple runs compared to the previous state-of-the-art (VAAC), indicating lower sensitivity to random initialization.
  • Pareto Frontier: The results highlight a clear trade-off: increasing the variance penalty ($\psi$) successfully reduces risk but eventually impacts the mean return. VPAC finds a more efficient balance on this frontier than prior methods.

Conclusion

VPAC provides a robust framework for risk-sensitive reinforcement learning by leveraging direct variance estimation. While the authors provide formal convergence proofs for the on-policy tabular case, future work remains to extend these proofs to off-policy and continuous settings. Additionally, implementing a dynamic scheduler for the penalty parameter $\psi$ could allow agents to explore freely early in training before "locking in" a reliable, low-variance policy.

Summary updated

Abstract

Reinforcement learning algorithms are typically geared towards optimizing the expected return of an agent. However, in many practical applications, low variance in the return is desired to ensure the reliability of an algorithm. In this paper, we propose on-policy and off-policy actor-critic algorithms that optimize a performance criterion involving both mean and variance in the return. Previous work uses the second moment of return to estimate the variance indirectly. Instead, we use a much simpler recently proposed direct variance estimator which updates the estimates incrementally using temporal difference methods. Using the variance-penalized criterion, we guarantee the convergence of our algorithm to locally optimal policies for finite state action Markov decision processes. We demonstrate the utility of our algorithm in tabular and continuous MuJoCo domains. Our approach not only performs on par with actor-critic and prior variance-penalization baselines in terms of expected return, but also generates trajectories which have lower variance in the return.
BibTeX
@inproceedings{jain2021variance,
  author = {Arushi Jain and Gandharv Patil and Ayush Jain and Khimya Khetarpal and Doina Precup},
  title = {{Variance Penalized On-Policy and Off-Policy Actor-Critic}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=B9WdKjOozq},
}
Figure