Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learning

Harm van Seijen, Mehdi Fatemi, Arash Tavakoli

Introduction

In reinforcement learning (RL), agents typically aim to maximize the total undiscounted reward, such as the final score in a game. However, to ensure mathematical stability and simplify the optimization process, practitioners heavily rely on a discount factor ($\gamma < 1$).

While it is widely known that low discount factors (e.g., $\gamma = 0.1$) often lead to poor performance, the underlying reasons have historically been misunderstood. This paper reveals that the failure of low discount factors is not due to universally small "action-gaps," but rather the massive variance of these gaps across an environment. To address this, the authors introduce Logarithmic Q-learning, a novel method that maps value estimates into a logarithmic space to homogenize action-gaps and enable stable learning even at low discount factors.

The Problem: Action-Gap Deviation

In standard RL, the "action-gap" is the difference in estimated value between the best and second-best actions. Common wisdom suggests that low discount factors shrink these gaps so much that neural networks (function approximators) cannot reliably distinguish between them.

The authors challenge this notion. They demonstrate that simply scaling up rewards—which proportionally increases the magnitude of the action-gaps—does not fix the failure of low discount factors. The true culprit is action-gap deviation: the phenomenon where an action-gap might be 1.0 in one state but $10^{-5}$ in another. Standard optimizers relying on $L_2$ loss struggle to resolve these multi-scale differences simultaneously, causing the network to prioritize states with larger gaps while ignoring others.

Logarithmic Q-Learning

To solve the issue of action-gap deviation, the authors propose Logarithmic Q-learning, which performs updates in a transformed space using three key mechanisms:

  1. Logarithmic Mapping: Instead of predicting the standard value $Q$ directly, the network predicts a mapped value $\tilde{Q} = f(Q)$. The mapping function used is $f(x) = c \ln(x + \gamma^k) + d$. This logarithmic transformation "squashes" the range of values, turning exponential decay into linear decay. Consequently, the action-gaps across different states become much more similar in size (homogeneous), allowing the network to learn effectively across the entire state space.
  2. Handling Stochasticity: Mapping a random variable into log-space introduces bias due to Jensen's Inequality (the average of the logarithms is not equal to the logarithm of the average). To correct this, the authors employ a two-step update: they perform a small averaging step in the standard space before mapping the result back to the logarithmic space.
  3. Reward Decomposition: Because logarithms cannot handle negative numbers, environments with both positive and negative rewards pose a challenge. The authors decompose the reward into two distinct components ($r^+$ and $r^-$) and train two separate value functions, $Q^+$ and $Q^-$. These are then combined during action selection.

Key Insights and Results

The authors evaluated their approach on a custom 50-state "Chain Task" to isolate optimization effects, as well as on 55 Atari 2600 games using the Dopamine framework.

  • Refuting the "Small Gap" Myth: By proving that reward magnitude scaling doesn't fix low-$\gamma$ failures, the authors successfully shifted the focus from the magnitude of the learning signal to the consistency of the signal across the state space.
  • Task-Dependent Discounting: The research highlights that the "optimal" discount factor is highly task-dependent. In some environments, a low $\gamma$ actually has a higher theoretical performance ceiling than a high $\gamma$, provided the optimization challenges can be overcome.
Grid environments (left) and their corresponding performance across different discount factors (right), illustrating task-dependent discounting.
  • Chain Task Performance: In controlled grid environments, standard Q-learning with function approximation failed completely for $\gamma < 0.8$. In contrast, Logarithmic Q-learning maintained near-perfect performance across the entire range of discount factors.
  • Atari Performance: The authors' deep RL implementation, LogDQN (using $\gamma = 0.96$), outperformed standard DQN (using $\gamma = 0.99$) on the majority of the 55 Atari games tested. Even without the complex additions of agents like Rainbow, LogDQN's median human-normalized score was significantly higher than standard DQN and approached the performance of more sophisticated distributional RL methods.

Limitations and Future Work

While Logarithmic Q-learning represents a significant step forward, it introduces new hyperparameters (such as $k$, $c$, and $\beta_{reg}$). The authors noted that while $k=100$ worked well for Atari, optimal settings may vary by task. Furthermore, training two separate networks for positive and negative rewards increases the parameter count, though this overhead was mitigated by sharing internal convolutional layers.

Finally, while the method enables much lower discount factors than traditional approaches, performance still dipped in deep RL settings when $\gamma$ was set extremely low (e.g., 0.1 or 0.5). The authors suggest this may be because standard DQN hyperparameters—like learning rates and epsilon-greedy schedules—are heavily tuned for high-$\gamma$ regimes and require re-evaluation for low-$\gamma$ training.

Summary updated

Abstract

In an effort to better understand the different ways in which the discount factor affects the optimization process in reinforcement learning, we designed a set of experiments to study each effect in isolation. Our analysis reveals that the common perception that poor performance of low discount factors is caused by (too) small action-gaps requires revision. We propose an alternative hypothesis that identifies the size-difference of the action-gap across the state-space as the primary cause. We then introduce a new method that enables more homogeneous action-gaps by mapping value estimates to a logarithmic space. We prove convergence for this method under standard assumptions and demonstrate empirically that it indeed enables lower discount factors for approximate reinforcement-learning methods. This in turn allows tackling a class of reinforcement-learning problems that are challenging to solve with traditional methods.
BibTeX
@inproceedings{seijen2019using,
  author = {Harm van Seijen and Mehdi Fatemi and Arash Tavakoli},
  title = {{Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=72R5qhmxiB},
}
Figure