Abstract
Standard reinforcement learning (RL) objectives often ignore risk, which is critical for high-stakes applications like finance and robotics. This paper provides a rigorous finite-sample analysis of the Mean-Volatility Actor-Critic (MVAC) algorithm, which optimizes a risk-averse objective by penalizing reward variance. We demonstrate that incorporating this risk penalty does not worsen the fundamental sample complexity compared to risk-neutral RL, achieving an $\epsilon$-accurate stationary point in $\mathcal{O}(\epsilon^{-2} \log(\epsilon^{-1}))$ samples. Empirical evaluations on a Point-Reacher environment validate the theoretical findings and illustrate the trade-offs between return and volatility.
Introduction
In standard RL, the objective is typically to maximize the expected cumulative return. However, this risk-neutral approach can lead to policies that exhibit high variance or catastrophic failures in stochastic environments. Risk-averse RL addresses this by penalizing uncertainty. A popular objective is Mean-Volatility, which penalizes the variance of per-step rewards.
A significant challenge in optimizing the mean-volatility objective is that the risk penalty depends on the policy's expected return, which is unknown and must be estimated from data. This paper addresses the core question: how does the error from estimating the expected return affect the convergence of actor-critic algorithms? Prior to this work, the finite-sample complexity for this specific risk-averse setup was unknown.
Methods
The authors analyze the Mini-batch Mean-Volatility Actor-Critic (MVAC) algorithm. To handle the policy-dependent reward transformation, two critic methods are proposed:
- Direct Method: Estimates expected return via Monte Carlo rollouts, transforms rewards by subtracting a penalty based on this estimate, and applies standard Temporal Difference (TD) learning.
- Factored Method: Learns two separate value functions in parallel—one for the standard reward and one for the squared reward—combining them to evaluate the risk-averse objective. This method avoids biases introduced by directly squaring estimated returns.
The analysis bounds the expected gradient norm to determine the number of samples required to reach a local optimum (stationary point).
Experimental Setup
The algorithm is evaluated on Point-Reacher, a 1D continuous control environment. An agent must navigate to the origin; larger steps increase expected reward but also introduce significant variance (risk), creating a clear trade-off for testing risk-averse policies.
Results and Discussion
The theoretical analysis proves that the critic requires $\mathcal{O}(\epsilon^{-1} \log(\epsilon^{-1}))$ samples, while the overall actor-critic loop converges in $\mathcal{O}(\epsilon^{-2} \log(\epsilon^{-1}))$ samples. This matches the state-of-the-art bounds for risk-neutral actor-critic algorithms, suggesting that risk-aversion does not fundamentally slow down learning in an asymptotic sense.
Empirically, MVAC successfully maps a Pareto frontier by varying the risk parameter $\lambda$ (Figure 1f, 1i). As $\lambda$ increases, the agent achieves lower volatility at the cost of lower expected returns. However, comparison with Trust Region Volatility Optimization (TRVO) shows that while MVAC is theoretically sound, TRVO often exhibits smoother and faster convergence in practice (Figure 1d, 1e, 1g, 1h).
Conclusion and Future Work
This work provides the first finite-sample complexity analysis for mean-volatility actor-critic algorithms, proving that risk-aversion is not inherently more complex to learn than risk-neutral objectives. However, the theoretical analysis currently assumes independent sample batches for return estimation, which is inefficient. Future work includes extending this analysis to more complex algorithms like TRVO and relaxing the requirement for independent sample batches to allow for data reuse.
Summary updated