Introduction
In reinforcement learning (RL), agents typically aim to maximize the total undiscounted reward, such as the final score in a game. However, to ensure mathematical stability and simplify the optimization process, practitioners heavily rely on a discount factor ($\gamma < 1$).
While it is widely known that low discount factors (e.g., $\gamma = 0.1$) often lead to poor performance, the underlying reasons have historically been misunderstood. This paper reveals that the failure of low discount factors is not due to universally small "action-gaps," but rather the massive variance of these gaps across an environment. To address this, the authors introduce Logarithmic Q-learning, a novel method that maps value estimates into a logarithmic space to homogenize action-gaps and enable stable learning even at low discount factors.
The Problem: Action-Gap Deviation
In standard RL, the "action-gap" is the difference in estimated value between the best and second-best actions. Common wisdom suggests that low discount factors shrink these gaps so much that neural networks (function approximators) cannot reliably distinguish between them.
The authors challenge this notion. They demonstrate that simply scaling up rewards—which proportionally increases the magnitude of the action-gaps—does not fix the failure of low discount factors. The true culprit is action-gap deviation: the phenomenon where an action-gap might be 1.0 in one state but $10^{-5}$ in another. Standard optimizers relying on $L_2$ loss struggle to resolve these multi-scale differences simultaneously, causing the network to prioritize states with larger gaps while ignoring others.
Logarithmic Q-Learning
To solve the issue of action-gap deviation, the authors propose Logarithmic Q-learning, which performs updates in a transformed space using three key mechanisms:
- Logarithmic Mapping: Instead of predicting the standard value $Q$ directly, the network predicts a mapped value $\tilde{Q} = f(Q)$. The mapping function used is $f(x) = c \ln(x + \gamma^k) + d$. This logarithmic transformation "squashes" the range of values, turning exponential decay into linear decay. Consequently, the action-gaps across different states become much more similar in size (homogeneous), allowing the network to learn effectively across the entire state space.
- Handling Stochasticity: Mapping a random variable into log-space introduces bias due to Jensen's Inequality (the average of the logarithms is not equal to the logarithm of the average). To correct this, the authors employ a two-step update: they perform a small averaging step in the standard space before mapping the result back to the logarithmic space.
- Reward Decomposition: Because logarithms cannot handle negative numbers, environments with both positive and negative rewards pose a challenge. The authors decompose the reward into two distinct components ($r^+$ and $r^-$) and train two separate value functions, $Q^+$ and $Q^-$. These are then combined during action selection.
Key Insights and Results
The authors evaluated their approach on a custom 50-state "Chain Task" to isolate optimization effects, as well as on 55 Atari 2600 games using the Dopamine framework.
- Refuting the "Small Gap" Myth: By proving that reward magnitude scaling doesn't fix low-$\gamma$ failures, the authors successfully shifted the focus from the magnitude of the learning signal to the consistency of the signal across the state space.
- Task-Dependent Discounting: The research highlights that the "optimal" discount factor is highly task-dependent. In some environments, a low $\gamma$ actually has a higher theoretical performance ceiling than a high $\gamma$, provided the optimization challenges can be overcome.
- Chain Task Performance: In controlled grid environments, standard Q-learning with function approximation failed completely for $\gamma < 0.8$. In contrast, Logarithmic Q-learning maintained near-perfect performance across the entire range of discount factors.
- Atari Performance: The authors' deep RL implementation, LogDQN (using $\gamma = 0.96$), outperformed standard DQN (using $\gamma = 0.99$) on the majority of the 55 Atari games tested. Even without the complex additions of agents like Rainbow, LogDQN's median human-normalized score was significantly higher than standard DQN and approached the performance of more sophisticated distributional RL methods.
Limitations and Future Work
While Logarithmic Q-learning represents a significant step forward, it introduces new hyperparameters (such as $k$, $c$, and $\beta_{reg}$). The authors noted that while $k=100$ worked well for Atari, optimal settings may vary by task. Furthermore, training two separate networks for positive and negative rewards increases the parameter count, though this overhead was mitigated by sharing internal convolutional layers.
Finally, while the method enables much lower discount factors than traditional approaches, performance still dipped in deep RL settings when $\gamma$ was set extremely low (e.g., 0.1 or 0.5). The authors suggest this may be because standard DQN hyperparameters—like learning rates and epsilon-greedy schedules—are heavily tuned for high-$\gamma$ regimes and require re-evaluation for low-$\gamma$ training.
Summary updated