Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation

Yunhao Tang, Tadashi Kozuno, Mark Rowland, Rémi Munos, Michal Valko

Introduction

Reinforcement Learning (RL) is notoriously sample-inefficient, often requiring millions of trials to learn a single task. Meta-RL attempts to address this by training agents to "learn how to learn," allowing them to adapt to new environments with just a few samples. A popular approach is gradient-based meta-learning (such as MAML), which involves an "inner loop" for task-specific adaptation and an "outer loop" to optimize the agent's starting parameters.

Optimizing the outer loop requires differentiating through the inner loop's gradient update. Mathematically, this necessitates calculating the Hessian matrix (second-order derivatives) of the value function. In standard supervised learning, auto-differentiation handles this seamlessly. However, in RL, value functions depend on the probability distribution of the policy itself. Simply "plugging in" standard gradient estimates and differentiating them often leads to high bias or extreme variance, destabilizing training. Prior works have attempted to solve this using complex "stochastic computation graphs," but these methods often feel disconnected and are difficult to implement or compare.

A Unified Framework via Off-Policy Evaluation

To resolve these challenges, this paper introduces a unified framework based on a key insight: estimating higher-order derivatives is mathematically equivalent to differentiating through an Off-Policy Evaluation (OPE) objective.

This connection demystifies the "black magic" of prior Hessian estimators, revealing that they are actually standard off-policy evaluation methods in disguise. The authors demonstrate that several existing, seemingly unrelated algorithms are simply special cases of differentiating specific OPE estimators.

Mapping prior meta-RL gradient estimators to their corresponding Off-Policy Evaluation (OPE) equivalents.

As shown in the table above, the framework maps prior works directly to OPE strategies:

  • DiCE corresponds to differentiating Step-wise Importance Sampling.
  • Loaded DiCE corresponds to the Doubly Robust estimator.
  • LVC (Low Variance Curvature) is equivalent to a first-order Taylor expansion (TayPO-1).

Building on this unification, the authors introduce a new family of estimators based on Taylor expansions of value functions, dubbed TayPO-K. Specifically, they highlight TayPO-2, a second-order estimator that captures more curvature information than previous methods while maintaining better stability than strictly unbiased estimators. Because the framework relies on OPE objectives, these estimators can be implemented easily in modern libraries like PyTorch or JAX—developers simply define the OPE scalar objective, and the auto-diff engine handles the complex Hessian-vector products.

Experimental Insights and Results

The authors evaluated their framework across three tiers of complexity: tabular MDPs (for precise bias/variance measurement), 2D Navigation, and high-dimensional MuJoCo locomotion tasks (e.g., Ant, HalfCheetah, Walker2D).

The Bias-Variance Trade-off

A central finding of the paper is the clear trade-off between bias and variance in these estimators. Higher-order Taylor expansions (like TayPO-2) reduce bias by more accurately approximating the true Hessian, but they increase variance because they involve more products of Importance Sampling (IS) ratios. The authors found that while unbiased estimators (like DiCE) are theoretically appealing, their variance often explodes in practice. Biased but low-variance estimators like TayPO-2 lead to much more stable training.

Improved Stability and Performance

In tabular experiments, the second-order TayPO-2 estimator achieved a significantly better balance of bias and variance than either the first-order LVC or the high-variance DiCE. This stability translated directly to more complex environments.

Training curves in a 2D navigation task demonstrating the superior stability of PROMP-TayPO-2 compared to baseline methods.

In 2D navigation tasks, the proposed PROMP-TayPO-2 algorithm proved remarkably stable. As seen in the training curves, baseline methods like MAML or EMAML frequently became unstable and failed to converge during initial training phases, whereas TayPO-2 maintained steady improvement. On complex MuJoCo tasks, TayPO-2 provided consistent gains over PROMP (a strong first-order baseline), achieving faster learning speeds and higher final performance in environments like Walker2D.

Limitations and Future Directions

Despite these improvements, the variance introduced by Importance Sampling ratios remains a bottleneck—a phenomenon the authors call the "variance wall." In very high-dimensional tasks, the noise from these ratios can sometimes drown out the benefits of having a more accurate Hessian estimate. Consequently, while TayPO-2 outperformed first-order methods, the performance gap in some MuJoCo environments was marginal, suggesting that first-order approximations are sometimes "good enough" without further variance reduction techniques.

Ultimately, this unifying OPE framework not only clarifies the landscape of meta-RL gradient estimation but also opens the door for future applications. The authors suggest this approach could be extended to other areas of RL that rely on higher-order derivatives, such as hyper-parameter optimization and the discovery of reward functions.

Summary updated

Abstract

Model-agnostic meta-reinforcement learning requires estimating the Hessian matrix of value functions. This is challenging from an implementation perspective, as repeatedly differentiating policy gradient estimates may lead to biased Hessian estimates. In this work, we provide a unifying framework for estimating higher-order derivatives of value functions, based on off-policy evaluation. Our framework interprets a number of prior approaches as special cases and elucidates the bias and variance trade-off of Hessian estimates. This framework also opens the door to a new family of estimates, which can be easily implemented with auto-differentiation libraries, and lead to performance gains in practice.
BibTeX
@inproceedings{tang2021unifying,
  author = {Yunhao Tang and Tadashi Kozuno and Mark Rowland and Rémi Munos and Michal Valko},
  title = {{Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=f2Zz3RzTabY},
}
Figure