Introduction
In practical reinforcement learning (RL), there is often a "technical gap" between the objective we train on and the objective we actually care about. Most real-world tasks evaluate performance based on the undiscounted cumulative return over a long horizon (e.g., $T=1000$ steps). However, to ensure mathematical convergence and stable learning, practitioners typically use a "myopic" discount factor ($\gamma$), such as $\gamma = 0.99$, which effectively looks only 100 steps into the future.
Simply increasing $\gamma$ to $0.999$ to match the evaluation horizon usually fails; it causes massive variance in gradient estimates and "collapses" the difference in value between different actions. To cope, practitioners often rely on heuristics—like dropping the discount factor from policy gradient updates entirely. While effective, these lack a solid theoretical foundation. This paper introduces a principled way to bridge this gap by applying Taylor expansions to discount factors, deriving a family of value function estimators that interpolate between stability and accuracy.
Methodology: Taylor Expansions for Value Functions
The authors propose using Taylor expansions to express a value function with a high, target discount factor ($\gamma'$) as a series of terms involving a value function with a lower, more stable discount factor ($\gamma$).
- K-th Order Expansion: They derive a formula where $V_{\gamma'}$ is an infinite sum of terms based on $V_{\gamma}$. By truncating this series at a specific order $K$, they create a "K-th order expansion" ($V_{K, \gamma, \gamma'}$). Here, $K=0$ is simply the myopic value function. Higher $K$ values progressively "look further" into the future, reducing bias but increasing variance.
- Sample-Based Estimation: The paper provides an algorithm to estimate these expansions using trajectories and geometrically distributed random time steps.
- Policy Gradient Decomposition: The authors show that the full policy gradient can be split into two parts. Crucially, they prove that the common "heuristic" of dropping the $\gamma^t$ term is actually the first partial gradient of the true objective.
- Algorithmic Integration: These expansions are integrated into Proximal Policy Optimization (PPO), creating PPO(K), which uses the Taylor expansion as a plug-in estimator for the advantage function.
Key Insights: The Bias-Variance Trade-off of $K$
One of the most striking insights is that the common practice of ignoring the discount factor in policy gradients isn't just a "hack"—it's a mathematically identifiable partial gradient of the long-term objective.
Furthermore, Taylor expansions provide a principled "slider" ($K$) to navigate the bias-variance trade-off. Instead of choosing between a myopic $\gamma=0.99$ and an unstable $\gamma=0.999$, you can use a stable $\gamma$ and "expand" it toward the target horizon.
As shown above, there is a "sweet spot" for $K$. As the base estimates of $V_{\gamma}$ become noisier, the optimal $K$ decreases. This explains why very high-order expansions aren't always better in practice; the noise compounds, making lower-order expansions (like $K=1$) more robust.
Empirical Results
The researchers evaluated their methods on standard continuous control benchmarks, including OpenAI Gym (MuJoCo) and Bullet Physics environments (e.g., HalfCheetah, Ant, Walker2d).
- Superior Performance: PPO(K) with $K=1$ consistently outperformed standard PPO ($\gamma=0.99$) and PPO with a high discount factor ($\gamma=0.999$) across almost all tasks.
- Stability and Speed: Unlike simply increasing $\gamma$, which often led to learning failure or high variance, the Taylor expansion approach remained stable while achieving higher asymptotic returns. In environments like HalfCheetah and Walker2d, PPO(K) reached higher reward levels significantly faster than the baselines.
- Update Weighting: Their "Taylor expansion update weighting" also showed more stable improvements compared to other state-of-the-art modifications like "PPO with time limits."
Limitations and Future Work
While promising, the method introduces a new hyperparameter $\eta$ (a mixture constant between the standard and Taylor-expanded estimators) that requires tuning, though $\eta=0.01$ generally works well. Additionally, while $K=1$ is computationally cheap, higher-order expansions require more sampling and computation per update.
Future work remains to thoroughly explore Taylor expansions in purely off-policy settings (like SAC and TD3) and to better understand the complex interactions between Taylor expansions and the errors introduced by neural network function approximation.
Summary updated