Taylor Expansion Policy Optimization

Yunhao Tang, Michal Valko, Rémi Munos

Taylor Expansion Policy Optimization (TayPO) provides a unified mathematical framework that links two foundational pillars of reinforcement learning: trust-region methods (like TRPO and PPO) and off-policy corrections (like V-trace). By applying higher-order Taylor expansions to the RL objective, TayPO yields a new class of algorithms that are more robust to delays and perform better in large-scale distributed settings.

The Challenge: Off-Policy Discrepancies

In reinforcement learning (RL), practitioners frequently encounter a discrepancy between the behavior policy (the policy used to collect data) and the target policy (the policy currently being optimized). This discrepancy typically arises for two reasons:

  1. Trust Regions: To ensure stable learning, algorithms intentionally constrain the new policy to remain close to the old one.
  2. Distributed Lag: In large-scale systems, "actors" collect data using an older version of the policy while a central "learner" updates the latest version. This inherent delay creates "off-policy" data.

Historically, researchers handled these issues with two distinct toolsets: trust-region constraints to limit policy changes, and importance sampling (IS) to correct for off-policy data. This paper addresses the lack of a unified theory connecting these approaches, demonstrating that they are actually two sides of the same Taylor-expanded coin.

The TayPO Framework

The authors propose viewing the RL objective function through the lens of Taylor expansions.

  • The Discovery: They prove that the standard "surrogate objective" used in popular algorithms like TRPO and PPO is actually just a first-order Taylor expansion of the true RL objective.
  • TayPO-k: They generalize this objective to the $k$-th order. While first-order methods only look at the immediate ratio between policies, higher-order expansions (like TayPO-2) look at the correlations between policy ratios across multiple time steps.
  • The Connection: The paper demonstrates a formal link between Taylor expansions and the $Q(\lambda)$ operator. Specifically, a $K$-th order Taylor expansion is equivalent to applying a return-based off-policy evaluation operator $K$ times.

Based on this, they introduce TayPO-2, which adds a second-order correction term to the policy gradient. This term uses importance sampling ratios from two different time steps to provide a tighter approximation of the true objective without the extreme variance typically associated with full importance sampling.

Key Insights

  • TRPO is a Special Case: The realization that TRPO/PPO are simply first-order Taylor approximations suggests that these famous algorithms are just the beginning of a hierarchy of possible objectives.
  • The "Trust Region" is Built-In: Taylor expansions only converge within a certain "radius." This radius of convergence mathematically justifies why we need trust regions: if the policy changes too much, the Taylor approximation (and thus our gradient) becomes highly inaccurate.
  • Bias-Variance Trade-off: While a full infinite-order expansion would be perfectly accurate, it would have infinite variance. TayPO-2 hits a "sweet spot"—it is more accurate than PPO but more stable than full importance sampling.

Empirical Results

The authors evaluated their approach on the Atari-57 benchmark, comparing TayPO against state-of-the-art baselines including PPO, IMPALA (using V-trace), and R2D2.

Better Asymptotics

In "near on-policy" settings with low delay, TayPO-2 achieved higher mean human-normalized scores on Atari-57 compared to PPO and zero-order baselines.

Performance comparison on Atari-57 showing second-order TayPO outperforming first-order PPO.

Graceful Degradation Under Lag

In distributed systems, there is always a delay between actors and learners. In experiments where the authors artificially injected "lag" (delaying the policy update to actors), TayPO-2's performance decayed much more slowly than V-trace or first-order methods. It remained effective even when data was significantly off-policy.

Performance degradation under artificial lag, demonstrating TayPO's robustness compared to V-trace.

Value-Based Gains

The benefits of TayPO are not limited to policy gradients. When integrated into the R2D2 architecture, second-order corrections outperformed the standard "Retrace" algorithm, showing that these insights apply to Q-learning as well.

Limitations and Future Work

While TayPO offers significant improvements, it comes with trade-offs. Computing second-order terms is computationally more expensive than first-order terms. Additionally, while the authors primarily focused on $K=2$, they note that as $K$ increases, the variance of the estimator grows. Finding the optimal $K$ for a specific environment, as well as tuning the weighting parameter ($\eta$) for the second-order term, remain open questions for future research.

Summary updated

Abstract

In this work, we investigate the application of Taylor expansions in reinforcement learning. In particular, we propose Taylor Expansion Policy Optimization, a policy optimization formalism that generalizes prior work as a first-order special case. We also show that Taylor expansions intimately relate to off-policy evaluation. Finally, we show that this new formulation entails modifications which improve the performance of several state-of-the-art distributed algorithms.
BibTeX
@inproceedings{tang2020taylor,
  author = {Yunhao Tang and Michal Valko and Rémi Munos},
  title = {{Taylor Expansion Policy Optimization}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=UCvNlNakHW},
}
Figure