Taylor Expansion Policy Optimization (TayPO) provides a unified mathematical framework that links two foundational pillars of reinforcement learning: trust-region methods (like TRPO and PPO) and off-policy corrections (like V-trace). By applying higher-order Taylor expansions to the RL objective, TayPO yields a new class of algorithms that are more robust to delays and perform better in large-scale distributed settings.
The Challenge: Off-Policy Discrepancies
In reinforcement learning (RL), practitioners frequently encounter a discrepancy between the behavior policy (the policy used to collect data) and the target policy (the policy currently being optimized). This discrepancy typically arises for two reasons:
- Trust Regions: To ensure stable learning, algorithms intentionally constrain the new policy to remain close to the old one.
- Distributed Lag: In large-scale systems, "actors" collect data using an older version of the policy while a central "learner" updates the latest version. This inherent delay creates "off-policy" data.
Historically, researchers handled these issues with two distinct toolsets: trust-region constraints to limit policy changes, and importance sampling (IS) to correct for off-policy data. This paper addresses the lack of a unified theory connecting these approaches, demonstrating that they are actually two sides of the same Taylor-expanded coin.
The TayPO Framework
The authors propose viewing the RL objective function through the lens of Taylor expansions.
- The Discovery: They prove that the standard "surrogate objective" used in popular algorithms like TRPO and PPO is actually just a first-order Taylor expansion of the true RL objective.
- TayPO-k: They generalize this objective to the $k$-th order. While first-order methods only look at the immediate ratio between policies, higher-order expansions (like TayPO-2) look at the correlations between policy ratios across multiple time steps.
- The Connection: The paper demonstrates a formal link between Taylor expansions and the $Q(\lambda)$ operator. Specifically, a $K$-th order Taylor expansion is equivalent to applying a return-based off-policy evaluation operator $K$ times.
Based on this, they introduce TayPO-2, which adds a second-order correction term to the policy gradient. This term uses importance sampling ratios from two different time steps to provide a tighter approximation of the true objective without the extreme variance typically associated with full importance sampling.
Key Insights
- TRPO is a Special Case: The realization that TRPO/PPO are simply first-order Taylor approximations suggests that these famous algorithms are just the beginning of a hierarchy of possible objectives.
- The "Trust Region" is Built-In: Taylor expansions only converge within a certain "radius." This radius of convergence mathematically justifies why we need trust regions: if the policy changes too much, the Taylor approximation (and thus our gradient) becomes highly inaccurate.
- Bias-Variance Trade-off: While a full infinite-order expansion would be perfectly accurate, it would have infinite variance. TayPO-2 hits a "sweet spot"—it is more accurate than PPO but more stable than full importance sampling.
Empirical Results
The authors evaluated their approach on the Atari-57 benchmark, comparing TayPO against state-of-the-art baselines including PPO, IMPALA (using V-trace), and R2D2.
Better Asymptotics
In "near on-policy" settings with low delay, TayPO-2 achieved higher mean human-normalized scores on Atari-57 compared to PPO and zero-order baselines.
Graceful Degradation Under Lag
In distributed systems, there is always a delay between actors and learners. In experiments where the authors artificially injected "lag" (delaying the policy update to actors), TayPO-2's performance decayed much more slowly than V-trace or first-order methods. It remained effective even when data was significantly off-policy.
Value-Based Gains
The benefits of TayPO are not limited to policy gradients. When integrated into the R2D2 architecture, second-order corrections outperformed the standard "Retrace" algorithm, showing that these insights apply to Q-learning as well.
Limitations and Future Work
While TayPO offers significant improvements, it comes with trade-offs. Computing second-order terms is computationally more expensive than first-order terms. Additionally, while the authors primarily focused on $K=2$, they note that as $K$ increases, the variance of the estimator grows. Finding the optimal $K$ for a specific environment, as well as tuning the weighting parameter ($\eta$) for the second-order term, remain open questions for future research.
Summary updated