Non-exponential Reward Discounting in Reinforcement Learning

Raja Farrukh AliAAAI 2023 · DOI

Abstract

Standard Reinforcement Learning (RL) frameworks typically rely on exponential reward discounting, which assumes a constant risk of failure over time. However, real-world environments often exhibit complex, time-varying hazard rates. This paper explores the integration of hyperbolic discounting and survival analysis into RL agents. By linking an agent's temporal preferences to the statistical hazard rate of its environment, we can improve generalization in procedurally generated tasks and enhance coordination in multi-agent systems.

Introduction

In Reinforcement Learning, the objective is to maximize the expected return—the cumulative sum of future rewards. To ensure mathematical convergence and prioritize immediate gains, researchers traditionally employ exponential discounting, defined by the factor $\gamma^t$. This approach implicitly assumes a constant "hazard rate," or a fixed probability that an episode will terminate at any given time step.

While computationally convenient, exponential discounting often fails to capture the dynamics of complex, uncertain environments. In contrast, humans and animals frequently exhibit hyperbolic discounting, characterized by high impatience in the short term and increased patience for rewards further in the future. This research argues that shifting away from fixed exponential decay toward survival-informed discounting allows agents to better navigate environments where the risk of termination is unknown or non-constant.

Methodology

Hyperbolic Discounting Formulation

The proposed framework utilizes a hyperbolic discounting function: $$\Gamma_k(t) = \frac{1}{1+kt}$$ where $k > 0$. Unlike exponential decay, hyperbolic functions allow for "preference reversals." This means an agent's relative valuation of two future rewards can change as time progresses, a behavior that more accurately reflects decision-making under uncertainty about environmental duration.

Multi-Horizon Learning

Building on recent advancements in Deep RL, the agent is designed to learn multiple Q-values or advantage functions simultaneously. Each function corresponds to a different discount factor and, consequently, a different time horizon. This multi-horizon approach serves as a powerful auxiliary task, helping the agent develop a more robust internal representation of temporal dynamics.

Hazard Rate Integration and Survival Analysis

The core innovation lies in treating episode termination as a statistical event. By employing survival analysis techniques—such as Kaplan-Meier estimation and Cox regression—the agent can empirically learn the hazard rate of its environment. This allows the discount rate to be dynamically adjusted based on the observed risk of "death" or episode end, rather than relying on a static, pre-defined prior.

Diverse Multi-Agent Teams

In Multi-Agent Reinforcement Learning (MARL), the research suggests that homogeneity in temporal preference may be a weakness. By equipping different agents within a team with varying discount factors—ranging from highly myopic to far-sighted—the collective policy becomes more resilient to environmental uncertainty and varied risk profiles.

Evaluation and Results

Generalization in Procedural Environments

The framework was evaluated using procedurally generated environments where agents encounter novel levels during testing. Results indicate that hyperbolic-based advantage estimation significantly improves generalization compared to standard policy-gradient baselines. The uncertainty inherent in unseen levels makes the hyperbolic prior more effective than the rigid constant-risk assumption of exponential discounting.

Behavioral Impact and Feasibility

The study confirms that the specific behavior policy enforced by the discounting function (hyperbolic vs. exponential) has a material impact on performance in hazardous conditions. Furthermore, the work demonstrates that approximating these complex, non-exponential functions is computationally feasible using modern temporal difference (TD) learning methods, dispelling concerns regarding the overhead of non-standard discounting.

Discussion

Discounting as a Prior

Exponential discounting should be viewed as a "prior belief" that the environment possesses a constant hazard rate. When an agent is uncertain about the underlying risk structure, hyperbolic discounting emerges as the mathematically appropriate choice. By framing RL through the lens of survival analysis, episode termination is transformed from a technical boundary into a statistical signal that dictates how future rewards should be valued.

Limitations and Future Work

While promising, the application of non-exponential discounting requires careful implementation to avoid instability. Future research will focus on:

  • Identifying Edge Cases: Determining specific scenarios where certain discounting functions may lead to suboptimal or unstable behavior.
  • Advanced Statistical Priors: Exploring more nuanced hazard rate models, such as the Gamma or Erlang distributions, to provide a more sophisticated representation of environmental risk.
  • Empirical Hazard Mapping: Developing representations that allow agents to explicitly map and respond to the empirical hazard rate in real-time.

Conclusion

Rethinking the temporal "clock" in RL by moving toward non-exponential discounting offers a path toward more adaptive and human-like agent behavior. By integrating survival analysis, RL agents can move beyond rigid assumptions and learn to navigate the inherent uncertainties of complex, time-varying environments.

Summary updated

BibTeX
@inproceedings{ali2023non,
  author = {Raja Farrukh Ali},
  title = {{Non-exponential Reward Discounting in Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.26916},
}

Loading related content...

Figure