Abstract
In an effort to better understand the different ways in which the discount factor affects the optimization process in reinforcement learning, we designed a set of experiments to study each effect in isolation. Our analysis reveals that the common perception that poor performance of low discount factors is caused by (too) small action-gaps requires revision. We propose an alternative hypothesis that identifies the size-difference of the action-gap across the state-space as the primary cause. We then introduce a new method that enables more homogeneous action-gaps by mapping value estimates to a logarithmic space. We prove convergence for this method under standard assumptions and demonstrate empirically that it indeed enables lower discount factors for approximate reinforcement-learning methods. This in turn allows tackling a class of reinforcement-learning problems that are challenging to solve with traditional methods.
Introduction
In Reinforcement Learning (RL), the discount factor ($\gamma$) is a crucial hyperparameter that determines the agent's temporal horizon, making it prioritize immediate rewards over distant ones. While high discount factors (typically around 0.99) are standard, they are not always optimal for every task. Some environments are actually easier to solve with a lower discount factor. However, approximate RL methods—those using neural networks—often suffer from a significant performance collapse when $\gamma$ is reduced. This paper investigates why this happens and proposes a novel solution: Logarithmic Q-learning.
The Action-Gap Deviation Hypothesis
The "action-gap" is the difference in value between the best possible action and the next best alternative in a given state. A long-held belief in the RL community was that low discount factors make this gap so small that neural networks cannot distinguish between actions, leading to poor performance.
The authors debunk this "small gap" myth by showing that simply scaling rewards (which increases the absolute size of the gap) does not fix the issue. Instead, they propose the Homogeneity Hypothesis: the problem is not the absolute size of the action-gap, but its variation across the state space. In many tasks, the gap might be 0.001 in one state and 100 in another. Because standard neural networks optimize for Mean Squared Error (MSE), they naturally focus on states with massive gaps and ignore those with tiny ones, creating an inconsistent and difficult optimization landscape.
Logarithmic Q-Learning (LogDQN)
To solve the problem of action-gap deviation, the authors introduce a method that performs updates in a logarithmic space. By applying the transformation $f(x) = \ln(x + \gamma^k)$ to reward targets, the range of values is "squashed." This naturally makes the action-gaps more homogeneous across the entire state space, allowing the neural network to learn effectively from all states regardless of their raw value scale.
Implementing this idea requires overcoming two main hurdles:
- Stochasticity and Jensen's Inequality: Averaging in log-space is not the same as averaging in linear space. To ensure the agent correctly calculates expected values, the authors use two separate step-sizes ($\beta_{reg}$ and $\beta_{log}$) during updates.
- Handling Negative Rewards: Since logarithms are not defined for negative numbers, the method decomposes rewards into two streams: $Q^+$ for positive rewards and $Q^-$ for negative rewards. These are learned separately and subtracted to find the final value.
Experimental Results
The researchers evaluated their approach on a synthetic Chain Task and the Arcade Learning Environment (Atari 2600).
- Chain Task: Logarithmic Q-learning completely resolved the performance drop-off seen in standard Q-learning at low discount factors, validating the homogeneity hypothesis.
- Atari Performance: LogDQN outperformed standard DQN on a majority of the 55 games tested. Crucially, while standard DQN performed best at $\gamma=0.99$, LogDQN achieved its best results at $\gamma=0.96$. This confirms that the logarithmic mapping successfully enables the use of lower discount factors, which can be beneficial for specific tasks.
Conclusion
This work identifies action-gap deviation as a primary reason for the failure of low-discount RL and provides a mathematically grounded solution. By equalizing the optimization signal across the state space, Logarithmic Q-learning allows agents to be more flexible with their temporal horizons, potentially opening the door to solving complex tasks where long-term discounting is a hindrance.
Summary updated