The Short-Sightedness of Discounted Reinforcement Learning

55 concepts38 papers

In reinforcement learning (RL), the discount factor $\gamma \in 0, 1)$ is a ubiquitous mathematical tool. It serves a vital pragmatic purpose: it ensures that infinite-horizon cumulative rewards sum to a finite number, providing mathematical stability and guaranteeing convergence during training.

However, this mathematical convenience comes with a severe cognitive cost for the agent: discount-induced myopia. By exponentially decaying the value of future rewards, standard RL frameworks inherently bias agents toward short-term gratification. This temporal bias often renders long-term goals, steady-state safety constraints, and distant catastrophic risks completely invisible to the learning agent.

The Mathematics of Invisibility

To understand why discounted RL fails over long horizons, we must look at how quickly future rewards vanish. If an agent operates with a standard discount factor of $\gamma = 0.99$, a reward or penalty occurring 500 steps in the future is multiplied by $\gamma^{500} \approx 0.006$. By step 1,000, its influence shrinks to $\approx 0.00004$.

As argued by [Andrew C. Li et al. in Challenges to Solving Combinatorially Hard Long-Horizon Deep RL Tasks, this exponential decay creates an artificial mathematical horizon. Beyond this horizon, distant rewards become effectively invisible, forcing the agent to prioritize immediate, smaller gains over critical long-term objectives, such as meeting distant deadlines in combinatorial routing problems. Even when value scaling issues are resolved, Hado van Hasselt et al. note in Learning values across many orders of magnitude that this temporal bias fundamentally prevents the agent from prioritizing high-value outcomes that require extended planning.

The Safety Blind Spot

This short-sightedness becomes dangerous when applied to safety constraints. In safety-critical applications—like autonomous driving or industrial control—a failure 1,000 steps from now is just as catastrophic as a failure tomorrow. Yet, when safety penalties are discounted, the agent perceives distant hazards as negligible.

Standard Discounting Horizon:
[Action] ---> [Immediate Reward (High Weight)] ---> ... ---> [Distant Catastrophic Risk (Zero Weight)]

Akhil Agnihotri et al. demonstrate in Average-Constrained Policy Optimization that using a discount factor as a proxy for safety leads to policies that optimize for immediate rewards while ignoring long-term catastrophic risks. This makes discounted models fundamentally unsuitable for tasks requiring permanent stability.

This issue stems from a structural mismatch in the optimization problem:

  • Physical safety constraints are typically undiscounted and finite.
  • Standard RL frameworks discount future costs to maintain mathematical stability.

As highlighted by Qisong Yang et al. in CEM: Constrained Entropy Maximization for Task-Agnostic Safe Exploration, this discrepancy causes agents to ignore long-term safety violations that occur far in the future.

This creates what Daniel Tabas et al. describe in Interpreting Primal-Dual Algorithms for Constrained Multiagent Reinforcement Learning as a "near-term effective horizon." Safety is only rigorously enforced within this narrow window; beyond it, protection diminishes exponentially. To counteract this, Pierre-François Massiani et al. prove in Safe Value Functions that the required penalty to keep a system safe must scale up drastically as the system's time-to-failure grows relative to the discount rate. Without such adjustments, or without moving to undiscounted cost returns as proposed by Ao Zhou et al. in CHPO: Constrained Hybrid-action Policy Optimization for Reinforcement Learning, agents will routinely exploit unsafe trajectories for short-term gains.

The Paradox of Rapid Failure

Discounting can also warp the logical structure of a task, leading to bizarre behavioral pathologies where the agent prefers failing quickly over succeeding slowly.

Consider a complex task where an agent receives a minor penalty for every step it takes until it succeeds. If the task takes too long, the discounted reward for eventual success can become numerically smaller than the discounted penalty for failing quickly. As observed by Mathias Jackermeier et al. in DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications for Multi-Task RL, this mathematical quirk causes a preference for rapid failure over delayed success when satisfying complex temporal constraints.

This misalignment makes discounted reward structures fundamentally incompatible with reachability objectives. Suguman Bansal et al. demonstrate in A Framework for Transforming Specifications in Reinforcement Learning that the temporal decay of rewards can cause an agent to prefer a fast but highly risky path over a slower, guaranteed optimal path. Consequently, traditional discounted reward functions cannot reliably guarantee the eventual completion of long-horizon logical specifications across arbitrary environments.

Biases in Action Selection and Exploration

The temporal bias of discounting does not just affect how agents value states; it directly distorts how they explore and which actions they select.

Execution Time Bias

In environments where actions take variable amounts of time to execute, agents develop a strong preference for speed over quality. Sherry Yang et al. show in Reinforcement Learning for Machine Learning Engineering Agents that standard RL algorithms exhibit a bias toward actions with shorter execution times because they provide faster feedback loops. This leads agents to favor computationally cheap but suboptimal solutions over intensive, high-performing strategies that require significant processing time.

Geographical and Goal Bias

Discounting also restricts an agent's exploratory footprint. Rishi Saket et al. explain in Dense and Diverse Goal Coverage in Multi Goal Reinforcement Learning that a discount factor naturally biases agents toward goals that are geographically or temporally closer to the starting state. This inherent preference makes it exceptionally difficult to discover or value distant goal states without explicit diversity objectives to counteract the bias.

Reasoning and Token-Level Pitfalls

In complex reasoning tasks (such as those performed by Large Language Model agents), applying traditional temporal discounting can be actively counterproductive. Jingcheng Hu et al. note in Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model that in reasoning domains, only the final answer truly matters, and discounting prevents the model from learning the long-term dependencies required for multi-step problem-solving.

Furthermore, when optimizing at the token level, the authors of Reinforcing LLM Agents via Policy Optimization with Action Decomposition show that standard discounting introduces a mathematical artifact where tokens appearing later in a sequence are treated as more important. This directly conflicts with linguistic structures, where early tokens often dictate the success of the entire sequence.

Moving Beyond the Discount Factor

To build agents capable of true long-term planning and steady-state safety, researchers are increasingly abandoning or modifying the standard discounted proxy.

One prominent alternative is the average reward criterion, which evaluates policies based on their long-term steady-state performance rather than near-term gains, eliminating the short-termism that plagues traditional models (Akhil Agnihotri et al.; Dong-Ki Kim et al.). Other approaches leverage value-distributional methods without discounting to preserve long-horizon visibility (Andrew C. Li et al.) or strictly enforce safety via undiscounted cost returns (Ao Zhou et al.), ensuring that the future is never sacrificed for the present.

Key Foundational Work

Learning values across many orders of magnitude (2016) — contributed 1 concept elements to this direction.
Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Application (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_short_term_reward_bias_ee3c3417,
  title = {short-term reward bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/short-term-reward-bias/rdr_ee3c3417e435422a96d1ecbff0a8821b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_short_term_reward_bias_ee3c3417_v1,
  title = {short-term reward bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/short-term-reward-bias/rdr_ee3c3417e435422a96d1ecbff0a8821b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake