An Analysis of Measure-Valued Derivatives for Policy Gradients

João Carvalho, Jan PetersarXiv 2022 · OpenReview

TL;DR

This paper explores the Measure-Valued Derivative (MVD) as a novel, low-variance gradient estimator for reinforcement learning (RL) policy gradients. They demonstrate that MVDs perform comparably to established methods like the likelihood-ratio (SF) and reparametrization (Rep-trick) estimators, even in complex, high-dimensional scenarios, and importantly, MVDs don't require differentiable value function approximators.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents learn by optimizing a policy to maximize rewards. Policy gradient methods are a cornerstone of RL, aiming to directly optimize the policy's parameters. A key challenge in this process is obtaining accurate and efficient gradient estimates. Traditional methods like the likelihood-ratio trick (SF) are unbiased but suffer from high variance, making learning unstable. The reparametrization trick (Rep-trick) offers lower variance but has a critical limitation: it requires the policy's value function approximator to be differentiable, which excludes many powerful function classes like decision trees. This paper addresses the need for a policy gradient estimator that is both low-variance and broadly applicable, even when dealing with non-differentiable components.

Methods

The paper introduces and analyzes the Measure-Valued Derivative (MVD) as a third option for computing policy gradients. Unlike SF and Rep-trick, MVD represents the gradient of a probability distribution as a difference between two related distributions. This decomposition allows for unbiased gradient estimation with low variance. The authors then integrate MVD into actor-critic RL algorithms. They propose using MVD to estimate the policy gradient in both simulated (Linear Quadratic Regulator - LQR) and deep RL settings (Soft Actor-Critic - SAC). A key aspect of their approach is demonstrating MVD's utility even when the critic (value function approximator) is not perfectly accurate or is non-differentiable, a scenario where Rep-trick would fail.

Datasets

Not discussed in the paper. The experiments utilize simulated environments (LQR) and standard deep RL control tasks from the PyBullet simulator.

Key Insights and Ideas

  • MVD as a Viable Alternative: The core insight is that MVD is not just a theoretical curiosity but a practical and effective alternative to SF and Rep-trick, offering a compelling trade-off between variance, bias, and applicability.
  • Robustness to Non-Differentiable Critics: A major advantage highlighted is MVD's ability to work with non-differentiable value function approximators. This opens doors to using a wider range of powerful function classes in RL, which is a significant limitation for Rep-trick.
  • Performance on Par with State-of-the-Art: The empirical results show that MVD can achieve comparable performance to Rep-trick and SF in both low and high-dimensional action spaces, suggesting it's not a compromise in terms of learning effectiveness.
  • Less Sensitive to Approximation Errors: In scenarios where the critic's value function is approximated with errors, MVD (and SF) proved more robust than Rep-trick, which was significantly affected by action-correlated errors.

Key Results

  • In the LQR setting with an oracle critic, MVD performed better than SF and slightly worse than Rep-trick.
  • When the critic's function approximation had errors (especially action-correlated ones), MVD and SF were unaffected by error frequency, while Rep-trick's performance degraded.
  • In high-dimensional deep RL tasks (using a modified SAC algorithm), SAC-MVD achieved performance comparable to SAC with Rep-trick, even when using only one gradient estimate per update.
  • Increasing the number of samples did not significantly improve SAC-MVD's performance, suggesting its efficiency.

Limitations and Future Work

  • Computational Complexity: The paper acknowledges that MVD can have higher computational complexity than SF and Rep-trick, requiring more queries to the function being differentiated. Future work aims to reduce this complexity by computing derivatives along important dimensions and using convex combinations of estimators.
  • General RL Framework Applicability: While the paper demonstrates MVD's applicability in general RL settings (avoiding environment resets), further exploration of its use in diverse and complex environments is suggested.
  • Exploration of Other Function Classes: The authors suggest investigating how MVDs can be used with other function approximators beyond those tested, particularly those that are non-differentiable.

Summary updated

Abstract

Reinforcement learning methods for robotics are increasingly successful due to the constant development of better policy gradient techniques. A precise (low variance) and accurate (low bias) gradient estimator is crucial to face increasingly complex tasks. Traditional policy gradient algorithms use the likelihood-ratio trick, which is known to produce unbiased but high variance estimates. More modern approaches exploit the reparametrization trick, which gives lower variance gradient estimates but requires differentiable value function approximators. In this work, we study a different type of stochastic gradient estimator - the Measure-Valued Derivative. This estimator is unbiased, has low variance, and can be used with differentiable and non-differentiable function approximators. We empirically evaluate this estimator in the actor-critic policy gradient setting and show that it can reach comparable performance with methods based on the likelihood-ratio or reparametrization tricks, both in low and high-dimensional action spaces. With this work, we want to show that the Measure-Valued Derivative estimator can be a useful alternative to other policy gradient estimators.
BibTeX
@misc{carvalho2022analysis,
  author = {João Carvalho and Jan Peters},
  title = {{An Analysis of Measure-Valued Derivatives for Policy Gradients}},
  year = {2022},
  eprint = {2203.03917},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.03917},
}
Figure