Taylor Expansion of Discount Factors

Yunhao Tang, Mark Rowland, Rémi Munos, Michal Valko

Abstract: In practical reinforcement learning (RL), the discount factor used for estimating value functions often differs from that used for defining the evaluation objective. In this work, we study the effect that this discrepancy of discount factors has during learning, and discover a family of objectives that interpolate value functions of two distinct discount factors. Our analysis suggests new ways for estimating value functions and performing policy optimization updates, which demonstrate empirical performance gains. This framework also leads to new insights on commonly-used deep RL heuristic modifications to policy optimization algorithms.

Introduction

In Reinforcement Learning (RL), agents are typically trained to maximize a discounted sum of rewards. However, a fundamental tension exists: while we often evaluate agents on long-horizon or undiscounted returns, training with a discount factor $\gamma$ near 1 is numerically unstable. High discount factors lead to high variance in gradient estimates and the phenomenon of "action gap collapse," where the value differences between actions become indistinguishable.

To circumvent this, practitioners often use a smaller $\gamma$ (e.g., 0.99) for training and employ various heuristics, such as omitting the $\gamma^t$ term in policy gradient updates. This paper provides a formal framework to bridge this gap using Taylor expansions of the discount factor.

Methodology: Taylor Expansions of Value Functions

The core proposal is to express the value function of a high discount factor $\gamma'$ as a Taylor series involving the value function of a lower, more stable discount factor $\gamma$.

  1. Value Function Expansion: The authors show that a value function for a high discount factor $\gamma'$ can be expressed as a series of terms involving the value function of a lower discount factor $\gamma$.
  2. Sample-based Estimation: They provide a way to estimate these expansions using trajectories. By sampling random time steps from a geometric distribution, they can "bootstrap" far-sighted value estimates from myopic ones.
  3. PPO(K): This leads to a modified Proximal Policy Optimization algorithm that uses a $K^{th}$-order expansion for advantage estimation.

The Bias-Variance Trade-off

The expansion order $K$ serves as a tunable parameter for the bias-variance trade-off. A low $K$ (e.g., $K=0$) corresponds to the standard, stable but biased approach. As $K$ increases, the estimate becomes less biased (approaching the true long-horizon objective) but more susceptible to noise. Finding the "sweet spot" for $K$ allows for improved learning speed and final performance.

The bias-variance trade-off of K: (a) Log relative error vs. expansion order K; (b) Optimal K vs. noise level.

Key Insights and Results

A striking theoretical result is that the common practice of ignoring the discount factor in policy gradients is not merely a heuristic; the authors prove it is the first partial gradient of the true undiscounted objective. This provides a rigorous justification for a technique previously considered a "hack."

Empirically, the researchers evaluated PPO(K) on standard continuous control benchmarks, including MuJoCo and PyBullet tasks. Key findings include:

  • PPO(K) Superiority: PPO with a first-order Taylor expansion ($K=1$) consistently outperformed both standard PPO and PPO with a high discount factor across almost all tasks.
  • Stability: Unlike simply increasing $\gamma$, which often leads to instability, the Taylor expansion approach maintains training stability while capturing long-horizon dependencies.
Comparative performance of PPO(K) against baselines on continuous control benchmarks.

Limitations and Future Work

While PPO(K) shows significant promise, the interpolation constant $\eta$ requires careful tuning. Additionally, while $K=1$ is computationally cheap, higher-order expansions introduce complexity that may yield diminishing returns. Future work will focus on automating hyperparameter selection and exploring the interaction between Taylor expansions and deep function approximation errors in off-policy settings.

Summary updated

Abstract

In practical reinforcement learning (RL), the discount factor used for estimating value functions often differs from that used for defining the evaluation objective. In this work, we study the effect that this discrepancy of discount factors has during learning, and discover a family of objectives that interpolate value functions of two distinct discount factors. Our analysis suggests new ways for estimating value functions and performing policy optimization updates, which demonstrate empirical performance gains. This framework also leads to new insights on commonly-used deep RL heuristic modifications to policy optimization algorithms.
BibTeX
@inproceedings{tang2021taylor,
  author = {Yunhao Tang and Mark Rowland and Rémi Munos and Michal Valko},
  title = {{Taylor Expansion of Discount Factors}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=hCMqNgUmCX},
}
Figure