Beyond Exponentially Discounted Sum: Automatic Learning of Return Function

Yufei Wang, Qiwei Ye, Tie-Yan LiuarXiv 2019 · OpenReview

Abstract

In reinforcement learning, Return, which is the weighted accumulated future rewards, and Value, which is the expected return, serve as the objective that guides the learning of the policy. In classic RL, return is defined as the exponentially discounted sum of future rewards. One key insight is that there could be many feasible ways to define the form of the return function (and thus the value), from which the same optimal policy can be derived, yet these different forms might render dramatically different speeds of learning this policy. In this paper, we research how to modify the form of the return function to enhance the learning towards the optimal policy. We propose to use a general mathematical form for return function, and employ meta-learning to learn the optimal return function in an end-to-end manner. We test our methods on a specially designed maze environment and several Atari games, and our experimental results clearly indicate the advantages of automatically learning optimal return functions in reinforcement learning.

Introduction

In standard Reinforcement Learning (RL), an agent's objective is to maximize the return, which is almost universally defined as the exponentially discounted sum of future rewards ($G_t = \sum_{k=0}^{\infty} \gamma^k r_{t+k}$). While this formulation is mathematically convenient and grounded in economic theory, it isn't necessarily the most efficient objective for learning.

Consider a maze where the agent only receives a reward at the exit. With exponential discounting, the learning signal becomes incredibly weak for states far from the goal. This "vanishing signal" problem makes credit assignment difficult and learning slow. The authors argue that while many different return functions can lead to the same optimal policy, some make that policy much easier to discover than others. They set out to find a way to learn these "learning-efficient" return functions automatically.

Methodology

The core innovation is the Return Generating Model (RGM). Instead of a fixed $\gamma$, the return is calculated as a general linear combination of future rewards: $G^g(s_t, a_t) = \sum_{l=t}^{\infty} \beta_l^\eta r_l$.

  • Transformer-based Architecture: To determine the coefficients $\beta_l^\eta$, the authors use a Multi-Head Attention (Transformer) encoder. This allows the model to look at the entire trajectory as a whole and perform relational reasoning—identifying which past states were truly critical for a future reward, regardless of how many time steps lie between them.
  • Meta-Learning (Meta-Gradients): The RGM is trained in an end-to-end "meta" fashion. The objective for the RGM is to produce returns that, when used to update the agent's policy, result in the greatest improvement in the agent's performance (measured by the original, true environment rewards).
The relationship between the Return Generating Model and standard policy gradient RL. The dashed red lines indicate the meta-learning update path.

Experiments and Results

The authors evaluated their approach on two main benchmarks: an illustrative 8x8 grid world featuring "portals" that transport the agent across the map, and 12 Atari games from the Arcade Learning Environment (ALE), including both rich-reward and sparse-reward titles.

In the portal maze, the RGM-enhanced agent (A2C + RGM) learned dramatically faster than vanilla A2C. Visualization shows that RGM creates a smooth, high-value gradient along the entire successful path, whereas vanilla A2C's value function drops off almost immediately away from the goal. On Atari, the method showed massive gains in games with delayed rewards. In Bowling and Tennis, improvements were over 1000% compared to the baseline.

Visualization of the maze environment (left) and the comparison of learned value functions between A2C + RGM (middle) and vanilla A2C (right).

Analysis suggests that the RGM effectively "backs up" rewards. It learns to assign high weights to critical rewards even for distant states, effectively propagating the learning signal back through time with minimal loss, unlike the rapidly decaying $\gamma^t$.

Conclusion

This paper highlights a crucial distinction: while the "true" return function defines what the optimal policy is, a "learned" return function can define how fast we find it. While arbitrary learned return functions don't necessarily satisfy the Bellman equation—limiting the RGM to Monte-Carlo style updates—the results on Atari and maze environments clearly demonstrate the potential of meta-learning the objective function itself. Future work will explore reconciling these learned objectives with temporal difference (TD) learning methods like Q-learning.

Summary updated

Abstract

In reinforcement learning, Return, which is the weighted accumulated future rewards, and Value, which is the expected return, serve as the objective that guides the learning of the policy. In classic RL, return is defined as the exponentially discounted sum of future rewards. One key insight is that there could be many feasible ways to define the form of the return function (and thus the value), from which the same optimal policy can be derived, yet these different forms might render dramatically different speeds of learning this policy. In this paper, we research how to modify the form of the return function to enhance the learning towards the optimal policy. We propose to use a general mathematical form for return function, and employ meta-learning to learn the optimal return function in an end-to-end manner. We test our methods on a specially designed maze environment and several Atari games, and our experimental results clearly indicate the advantages of automatically learning optimal return functions in reinforcement learning.
BibTeX
@misc{wang2019beyond,
  author = {Yufei Wang and Qiwei Ye and Tie-Yan Liu},
  title = {{Beyond Exponentially Discounted Sum: Automatic Learning of Return Function}},
  year = {2019},
  eprint = {1905.11591},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.11591},
}
Figure