State-Wise Adaptive Discounting from Experience (SADE): A Novel Discounting Scheme for Reinforcement Learning (Student Abstract)

Milan Zinzuvadiya, Vahid BehzadanAAAI 2021 · DOI

Abstract

Traditional Reinforcement Learning (RL) relies on a fixed global discount factor ($\gamma$) to weigh future rewards against immediate ones. This research introduces State-wise Adaptive Discounting from Experience (SADE), a novel scheme where each state maintains an individual discount rate that adapts based on the agent's experience. By increasing the discount rate for promising trajectories and decreasing it for underperforming ones, SADE achieves significantly faster convergence and improved sample efficiency. Empirical evaluations in Grid World environments demonstrate that SADE consistently outperforms standard fixed-rate discounting methods across various complexities.

Introduction

In standard Reinforcement Learning, the discount factor $\gamma$ is typically a fixed hyperparameter applied globally across the entire environment. However, a "one-size-fits-all" approach ignores the experiential information an agent gains during exploration. Human cognition naturally adjusts temporal horizons based on context; similarly, an RL agent might benefit from valuing future rewards differently depending on its current state. SADE addresses this by allowing the discount factor to evolve dynamically on a per-state basis.

Methodology

SADE introduces a discount function $\lambda(s)$ that maps each state to a specific discount rate. The core mechanism involves an iterative adjustment process:

  1. Initialization: States are initialized with a baseline discount value (e.g., 0.5).
  2. Experience Gathering: The agent explores the environment in episodes.
  3. Adaptive Update: After each episode, the discount rate $\lambda_s$ for states in the trajectory is updated based on the expected return. If the return increases—suggesting progress toward an optimal path—$\lambda_s$ is increased to encourage a longer-term horizon. Conversely, if the return decreases, $\lambda_s$ is decreased.
  4. Constraints: Stability is maintained by enforcing upper and lower bounds ($\lambda_{min}$ and $\lambda_{max}$) to prevent the discount factor from reaching 0 or 1.

Experimental Evaluation

The researchers evaluated SADE in a Grid World benchmark using six configurations ranging from a simple $3 \times 3$ grid to a complex $10 \times 11$ environment with obstacles. SADE was compared against two baselines: Fixed-Exponential Discounting (the standard RL approach) and Fixed-Hyperbolic Discounting (often used to model human behavior).

Figure 1: Comparison of wins across different Gridworld complexities for SADE and fixed discounting baselines.

Key Results

The experimental results highlight several advantages of the SADE approach:

  • Higher Success Rates: As shown in Figure 1, SADE consistently achieves a higher number of wins across all tested grid complexities compared to both exponential and hyperbolic baselines.
  • Faster Convergence: SADE often outperforms fixed schemes by at least a factor of two in terms of convergence speed, requiring fewer interactions to reach optimal performance.
  • Robustness to Hyperparameters: The performance of SADE is influenced by the "adjustment rate" (the percentage by which $\lambda_s$ changes). However, sensitivity analysis indicates that SADE maintains a performance advantage over a wide range of adjustment settings.
Figure 2: Performance sensitivity of SADE compared to baselines across various adjustment rates.

Discussion and Future Work

SADE demonstrates that state-wise adaptive discounting can significantly improve learning efficiency by mimicking the adaptive nature of human temporal discounting. While highly effective in discrete Grid World environments, future research is needed to explore how SADE scales to high-dimensional, continuous state spaces. Additionally, integrating SADE with Deep RL architectures, such as Deep Q-Networks (DQN), represents a promising avenue for enhancing the sample efficiency of modern deep learning agents.

Summary updated

BibTeX
@inproceedings{zinzuvadiya2021state,
  author = {Milan Zinzuvadiya and Vahid Behzadan},
  title = {{State-Wise Adaptive Discounting from Experience (SADE): A Novel Discounting Scheme for Reinforcement Learning (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i18.17973},
}
Figure