Variance Penalized On-Policy and Off-Policy Actor-Critic

Arushi Jain, Gandharv Patil, Ayush Jain, Khimya Khetarpal, Doina Precup

Abstract

Reinforcement learning algorithms are typically geared towards optimizing the expected return of an agent. However, in many practical applications, low variance in the return is desired to ensure the reliability of an algorithm. In this paper, we propose on-policy and off-policy actor-critic algorithms that optimize a performance criterion involving both mean and variance in the return. Previous work uses the second moment of return to estimate the variance indirectly. Instead, we use a much simpler recently proposed direct variance estimator which updates the estimates incrementally using temporal difference methods. Using the variance-penalized criterion, we guarantee the convergence of our algorithm to locally optimal policies for finite state action Markov decision processes. We demonstrate the utility of our algorithm in tabular and continuous MuJoCo domains. Our approach not only performs on par with actor-critic and prior variance-penalization baselines in terms of expected return, but also generates trajectories which have lower variance in the return.

Introduction

Standard Reinforcement Learning (RL) focuses on maximizing average rewards, often ignoring the "risk" of highly unpredictable outcomes. In high-stakes fields like robotics, finance, or healthcare, a policy that succeeds brilliantly 50% of the time but fails catastrophically the other 50% is often less desirable than a "boring" policy that succeeds consistently. This paper addresses the challenge of mean-variance optimization by introducing VPAC (Variance Penalized Actor-Critic), a framework that directly estimates and penalizes return variance in both on-policy and off-policy settings.

The VPAC Framework

The core innovation of VPAC is the use of a direct variance estimator. Traditional "indirect" methods calculate the second moment of the return ($E[G^2]$) to derive variance, which is often noisy and unstable. VPAC instead utilizes a Bellman-style equation that allows the variance itself to be updated incrementally using Temporal Difference (TD) methods.

Technical Contributions

  • Modified Objective: The policy objective is redefined as $J(\theta) = E[Q_\pi(s,a) - \psi \sigma_\pi(s,a)]$, where $\psi$ is a user-defined penalty parameter controlling the trade-off between high returns and low variability.
  • Policy Gradient Theorems: The authors derive specific gradients for this variance-penalized objective for both on-policy and off-policy scenarios.
  • Multi-Timescale Learning: To ensure stability, the algorithm uses three different learning rates: the value function (critic) converges fastest, followed by the variance estimator, and finally the policy (actor).
  • Off-Policy Correction: The off-policy version incorporates importance sampling ratios directly into the variance Bellman equation to account for discrepancies between behavior and target policies.

Experimental Evaluation

The researchers evaluated VPAC across several simulated environments, including a tabular Four Rooms grid-world and continuous control tasks in MuJoCo (Hopper, Walker2d, HalfCheetah).

Figure 4: In the Four Rooms environment, VPAC learns a path (orange arrows) that avoids the high-variance "frozen" states (red), even if it means taking a longer route to the goal.

In the Four Rooms experiment, the agent could take a path through a "frozen" region with stochastic rewards. While a standard agent is indifferent to this noise, VPAC learns to take the "long way around" to ensure a predictable outcome.

Key Results and Insights

VPAC achieved significant variance reduction across all benchmarks. In MuJoCo tasks, it achieved up to a 93% reduction in return variance for HalfCheetah and 39.4% for Hopper compared to standard PPO. Crucially, VPAC maintained mean returns comparable to standard Actor-Critic and other risk-sensitive baselines.

Figure 2: Heatmap showing the significantly reduced variance achieved by the VPAC algorithm in the grid-world environment.

Direct vs. Indirect Estimation

The results reinforce that direct variance estimation is "better behaved" than indirect methods. It is less sensitive to noise in value estimates and handles eligibility traces more gracefully. The parameter $\psi$ acts as a "risk-aversion" dial; while increasing it reduces variance, it can eventually curb exploration, suggesting that a scheduled $\psi$ might be beneficial in practice.

Conclusion and Future Work

VPAC provides a robust method for training reliable RL agents. While the authors provide convergence proofs for the tabular case, extending these to non-linear function approximation (like deep neural networks) remains an open challenge. Future work may also explore automating the selection of the trade-off parameter $\psi$ based on specific safety constraints.

Summary updated

Abstract

Reinforcement learning algorithms are typically geared towards optimizing the expected return of an agent. However, in many practical applications, low variance in the return is desired to ensure the reliability of an algorithm. In this paper, we propose on-policy and off-policy actor-critic algorithms that optimize a performance criterion involving both mean and variance in the return. Previous work uses the second moment of return to estimate the variance indirectly. Instead, we use a much simpler recently proposed direct variance estimator which updates the estimates incrementally using temporal difference methods. Using the variance-penalized criterion, we guarantee the convergence of our algorithm to locally optimal policies for finite state action Markov decision processes. We demonstrate the utility of our algorithm in tabular and continuous MuJoCo domains. Our approach not only performs on par with actor-critic and prior variance-penalization baselines in terms of expected return, but also generates trajectories which have lower variance in the return.
BibTeX
@inproceedings{jain2021variance,
  author = {Arushi Jain and Gandharv Patil and Ayush Jain and Khimya Khetarpal and Doina Precup},
  title = {{Variance Penalized On-Policy and Off-Policy Actor-Critic}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=B9WdKjOozq},
}
Figure