Abstract: In this work, we investigate the application of Taylor expansions in reinforcement learning. In particular, we propose Taylor expansion policy optimization, a policy optimization formalism that generalizes prior work (e.g., TRPO) as a first-order special case. We also show that Taylor expansions intimately relate to off-policy evaluation. Finally, we show that this new formulation entails modifications which improve the performance of several state-of-the-art distributed algorithms.
Introduction
In Deep Reinforcement Learning (RL), achieving stable and efficient policy updates is a primary challenge. Researchers typically rely on two distinct strategies: trust-region methods (like TRPO and PPO) to prevent catastrophic policy shifts, and off-policy corrections (like V-trace) to handle the "lag" inherent in distributed systems where actors and learners operate asynchronously.
Historically, these strategies have been treated as separate algorithmic techniques. Taylor Expansion Policy Optimization (TayPO) provides a unified mathematical framework that bridges these concepts. By viewing policy optimization through the lens of Taylor expansions, we can derive tighter approximations of the true reward objective, leading to improved performance and robustness.
Methodology
The core of the TayPO framework is expressing the performance difference between a new policy ($\pi$) and an old behavior policy ($\mu$) as a $K$-th order Taylor series expansion.
- TayPO-1 (First-Order): Taking only the first term of the expansion mathematically recovers the surrogate objectives used in standard algorithms like TRPO and PPO. This reveals that these popular methods are actually first-order Taylor approximations.
- TayPO-2 (Second-Order): The primary contribution of this work is the second-order expansion. By incorporating the second-order term, the algorithm accounts for the interactions between importance sampling ratios across different time steps. This results in a significantly more accurate approximation of the true reward surface.
- Theoretical Unification: We demonstrate that a $K$-th order Taylor expansion is equivalent to applying a standard off-policy evaluation operator (the $Q(\lambda)$ operator) $K$ times. This formally links the fields of policy optimization and off-policy evaluation.
Experimental Evaluation
We evaluated TayPO using synthetic Random MDPs for initial validation and the Atari 57 suite for large-scale benchmarking. The Atari experiments utilized distributed architectures similar to IMPALA and R2D2.
Approximation Accuracy
A key insight of this work is that higher-order expansions provide a much tighter fit to the objective function. As shown in Figure 1, while first-order methods provide a reasonable estimate for small policy changes, second-order expansions drastically reduce the relative error as the distance between the behavior and target policies increases.
Robustness to Off-Policy Lag
In distributed RL, there is an unavoidable delay between data collection by actors and optimization by the learner. This "lag" makes the data increasingly off-policy. TayPO-2 demonstrates remarkable robustness to this stale data. While the performance of first-order methods and standard corrections like V-trace tends to collapse under high lag, the second-order correction remains stable and effective.
Key Results
- Superior Performance: On the Atari 57 suite, TayPO-2 achieved higher mean and median human-normalized scores compared to PPO and V-trace baselines.
- Efficiency: TayPO-2 reached higher peak scores more quickly, demonstrating better sample efficiency within the same budget of actor steps.
- Distributed Stability: In high-latency scenarios, TayPO-2 significantly outperformed V-trace, maintaining high performance in games where other methods failed.
Limitations and Future Work
While higher-order terms reduce bias, they involve products of importance sampling ratios which can increase variance. We address this using a hyperparameter ($\eta$) to balance these terms. Future research could explore automated methods for tuning this parameter. Additionally, while we focused on $K=2$, the framework theoretically allows for even higher-order expansions, though the computational complexity and variance may eventually yield diminishing returns.
Summary updated