Introduction
Standard Reinforcement Learning (RL) agents typically plan for a single, fixed future horizon defined by a discount factor ($\gamma$). This "one-size-fits-all" approach can be brittle, especially in procedurally generated environments where level dynamics vary significantly. If an agent's estimate of the "expected lifetime" within a level is biased, its policy may fail to generalize.
Think of it like human planning: we make decisions based on today (lunch), next year (career goals), and decades from now (retirement). If an RL agent only looks at one of these, it becomes either too short-sighted (myopic) or too focused on a distant future that might never happen. This paper addresses the brittleness of single-horizon learning by introducing a multi-horizon approach where an agent learns to evaluate rewards across multiple timescales simultaneously.
Methodology
The proposed framework builds upon a streamlined version of the Rainbow algorithm, a high-performance ensemble of RL improvements. The core innovation is a Multi-Horizon Architecture that features multiple output "heads" or blocks. Each block predicts Q-values (the expected value of an action) for a distinct discount factor $\gamma$, ranging from 0.906 to 0.99.
As shown in the architecture diagram, the network utilizes residual blocks and noisy fully connected (FC) layers within a dueling architecture. To decide on an action, the agent employs Advantage-Based Action Selection. Instead of using raw Q-values directly, it leverages the "advantage" signal—how much better an action is compared to the average—to determine the optimal move. Furthermore, the authors implement Hyperbolic Discounting, which more closely mimics human behavior by valuing rewards across a spectrum of timescales. This is computationally approximated as a weighted Riemann sum of multiple exponential horizons.
Experimental Setup
The researchers evaluated the multi-horizon agent on the Procgen benchmark, a suite of 16 procedurally generated environments such as Bigfish, Miner, and Leaper. Procgen is specifically designed to test generalization: agents are trained on a limited set of levels and then tested on entirely new, unseen scenarios. This setup prevents simple path memorization and requires the agent to learn robust game logic that applies across diverse visual and structural variations.
Results and Discussion
The multi-horizon agents generally outperformed the single-horizon "Vanilla Rainbow" baseline across the Procgen suite. A key insight from the study is that "learning long but acting short" is highly effective; agents that learn about long-term horizons provide a richer internal representation of the environment, even when choosing actions based on more immediate advantages.
As illustrated in the aggregate performance profiles, the multi-horizon (hyperbolic) variant demonstrates a significantly higher success rate across various normalized score thresholds ($\tau$) compared to both single-horizon and "largest gamma" baselines. This suggests that averaging across multiple timescales acts as a safety net, preventing the agent from being misled by a single, inappropriate horizon for a specific level's risk profile. The multi-horizon approach also showed superior sample efficiency, reaching higher test scores faster than traditional methods.
Conclusion and Future Work
This work demonstrates that multi-horizon learning significantly enhances generalization in complex, procedurally generated environments. By moving away from a fixed discount factor, agents can better navigate the diverse risks and reward structures inherent in varying levels.
While promising, the authors note that the specific $\gamma$ values were pre-defined. Future research could explore adaptive horizon selection, where the agent dynamically learns which timescale to prioritize based on the current state of the game. Additionally, optimizing how these horizons are weighted based on the agent's "prior beliefs" about environmental risk remains an open area for improvement.
Summary updated