Abstract: In practical reinforcement learning (RL), the discount factor used for estimating value functions often differs from that used for defining the evaluation objective. In this work, we study the effect that this discrepancy of discount factors has during learning, and discover a family of objectives that interpolate value functions of two distinct discount factors. Our analysis suggests new ways for estimating value functions and performing policy optimization updates, which demonstrate empirical performance gains. This framework also leads to new insights on commonly-used deep RL heuristic modifications to policy optimization algorithms.
Introduction
In Reinforcement Learning (RL), agents are typically trained to maximize a discounted sum of rewards. However, a fundamental tension exists: while we often evaluate agents on long-horizon or undiscounted returns, training with a discount factor $\gamma$ near 1 is numerically unstable. High discount factors lead to high variance in gradient estimates and the phenomenon of "action gap collapse," where the value differences between actions become indistinguishable.
To circumvent this, practitioners often use a smaller $\gamma$ (e.g., 0.99) for training and employ various heuristics, such as omitting the $\gamma^t$ term in policy gradient updates. This paper provides a formal framework to bridge this gap using Taylor expansions of the discount factor.
Methodology: Taylor Expansions of Value Functions
The core proposal is to express the value function of a high discount factor $\gamma'$ as a Taylor series involving the value function of a lower, more stable discount factor $\gamma$.
- Value Function Expansion: The authors show that a value function for a high discount factor $\gamma'$ can be expressed as a series of terms involving the value function of a lower discount factor $\gamma$.
- Sample-based Estimation: They provide a way to estimate these expansions using trajectories. By sampling random time steps from a geometric distribution, they can "bootstrap" far-sighted value estimates from myopic ones.
- PPO(K): This leads to a modified Proximal Policy Optimization algorithm that uses a $K^{th}$-order expansion for advantage estimation.
The Bias-Variance Trade-off
The expansion order $K$ serves as a tunable parameter for the bias-variance trade-off. A low $K$ (e.g., $K=0$) corresponds to the standard, stable but biased approach. As $K$ increases, the estimate becomes less biased (approaching the true long-horizon objective) but more susceptible to noise. Finding the "sweet spot" for $K$ allows for improved learning speed and final performance.
Key Insights and Results
A striking theoretical result is that the common practice of ignoring the discount factor in policy gradients is not merely a heuristic; the authors prove it is the first partial gradient of the true undiscounted objective. This provides a rigorous justification for a technique previously considered a "hack."
Empirically, the researchers evaluated PPO(K) on standard continuous control benchmarks, including MuJoCo and PyBullet tasks. Key findings include:
- PPO(K) Superiority: PPO with a first-order Taylor expansion ($K=1$) consistently outperformed both standard PPO and PPO with a high discount factor across almost all tasks.
- Stability: Unlike simply increasing $\gamma$, which often leads to instability, the Taylor expansion approach maintains training stability while capturing long-horizon dependencies.
Limitations and Future Work
While PPO(K) shows significant promise, the interpolation constant $\eta$ requires careful tuning. Additionally, while $K=1$ is computationally cheap, higher-order expansions introduce complexity that may yield diminishing returns. Future work will focus on automating hyperparameter selection and exploring the interaction between Taylor expansions and deep function approximation errors in off-policy settings.
Summary updated