Why do we make a New Year's resolution to exercise every day, only to skip the gym on January 2nd? In classical economics and standard reinforcement learning (RL), this behavior is an anomaly. Standard sequential decision-making models rely on exponential discounting, where a future reward is devalued by a constant factor $\gamma^t$ based on its distance in time.
As Lattimore et al. mathematically characterize, geometric or exponential discounting is essentially the only "sliding" scale that guarantees time-consistency. Time-consistency means that if an agent prefers Plan A over Plan B when looking into the future, they will still prefer Plan A when that future arrives. The passage of time alone does not alter the optimal strategy.
However, real-world biological agents do not behave this way. In a foundational manifesto for behavioral economics, Thaler points out that human self-control is fundamentally time-inconsistent. We are highly patient when planning for the distant future, but highly impulsive when faced with immediate rewards. This phenomenon, known as present bias, creates a profound challenge for computational planning: how do we model, predict, and optimize behavior when an agent's own preferences change over time?
The Mathematics of Preference Reversal
To understand why standard planning algorithms break down, we must look at how rewards are devalued over time. In standard RL, the relative value between a reward at time $t$ and time $t+1$ is always constant ($\gamma$). Schultheis et al. argue that this reliance on exponential discounting fails to capture the true nature of human and animal behavior, preventing artificial agents from accurately modeling real-world decision-making.
When biological agents evaluate the future, they tend to use non-exponential discounting (such as hyperbolic or power-law functions). Gershman et al. demonstrate that animal behavior aligns with a scale-invariant, logarithmically compressed timeline.
Exponential Discounting (Time-Consistent):
Today: [Take $10 today] > [Wait for $11 tomorrow]
In 1 Year: [Take $10 in 365 days] > [Wait for $11 in 366 days]
Hyperbolic Discounting (Time-Inconsistent / Preference Reversal):
Today: [Take $10 today] > [Wait for $11 tomorrow] (Impulsive)
In 1 Year: [Take $10 in 365 days] < [Wait for $11 in 366 days] (Patient)
Under hyperbolic discounting, the discount rate is steep for immediate delays but flattens out for distant delays. This mathematical structure guarantees preference reversals. As time glides forward and a once-distant choice becomes immediate, the relative value of the options flips. This invalidates the Bellman optimality principle, meaning we can no longer use standard dynamic programming to find a single, permanently optimal policy.
Graph-Theoretic Models of Procrastination
To computationally analyze present-biased agents, researchers often map their behavior to pathfinding over directed graphs. In this framework, developed by Tang et al., an agent navigates a graph toward a goal state. At any given node, the agent chooses the next immediate step that minimizes the sum of the immediate edge cost and the discounted estimated cost of the remaining path.
This setup elegantly explains procrastination and task abandonment:
- At the start: The agent looks at a multi-stage task. The immediate cost of step 1 is clear, but the costs of steps 2, 3, and 4 are safely in the future and heavily discounted. The total path looks manageable.
- At the next step: When the agent arrives at step 2, its cost is no longer devalued by the future discount factor. It is now an immediate cost. Suddenly, the effort feels prohibitively expensive, and the agent abandons the path or delays execution.
Tang et al. proved that finding optimal ways to "nudge" these agents—either by placing rewards or restricting their choices to keep them on track—is NP-hard.
To make these models more representative of real-world volatility, Gravin et al. extended this framework by treating the present-bias factor as a stochastic variable that changes at each time step, capturing how fluctuating willpower alters daily planning. For continuous-time environments, Kim et al. introduced a tractable mathematical model using the variational principle to smoothly predict exactly when an agent will hit a breaking point and abandon a task as a deadline approaches.
How Agents Plan for Their Own Inconsistency
When an agent has time-inconsistent preferences, how does it formulate a plan? Skalse et al. formalize a taxonomy of three distinct planning strategies based on the agent's level of self-awareness:
- Naïve Agents: A naïve agent is completely oblivious to its future preference shifts. At node $A$, it assumes its future self will perfectly execute the currently optimal plan. However, upon reaching node $B$, its preferences change, and it deviates from its own plan.
- Sophisticated Agents: A sophisticated agent is fully self-aware. It knows its future self will be tempted to procrastinate or abandon tasks. Therefore, it plans backwards using a subgame-perfect equilibrium, choosing actions today based on what its future self will actually do, rather than what it wishes its future self would do.
- Resolute Agents: A resolute agent chooses an initial optimal trajectory and rigidly commits to it, completely overriding any local temptations or preference reversals that arise later.
This distinction is vital for Inverse Reinforcement Learning (IRL), where an observer tries to infer an agent's underlying rewards by watching its behavior. As Skalse et al. demonstrate, if an AI system assumes a human is a time-consistent exponential discounter when they are actually a sophisticated hyperbolic discounter, the inferred reward function will be fundamentally flawed, leading to catastrophic failures in preference elicitation.
Structural Inconsistency in Risk and Multi-Objective AI
Time-inconsistency is not just a psychological quirk of biological brains; it is also an inherent mathematical property that emerges when optimizing complex, multi-variable AI objectives.
Multi-Objective Aggregation
When an autonomous agent balances multiple goals with different time horizons (e.g., a short-term safety goal vs. a long-term mission goal), a common approach is to linearly combine their rewards. However, Pitis proved that aggregating objectives with diverse time preferences using static weights inherently causes dynamic inconsistency. As time passes, the relative remaining horizons of the goals shift, causing the agent to perpetually delay long-term tasks. To maintain consistency, the authors show that agents must use non-Markovian rewards—reward functions that track the historical trajectory of the agent rather than just the current state.
Risk-Sensitive Planning
A similar structural failure occurs when optimizing for safety and risk. Traditional Markov Decision Processes evaluate risk locally at individual time steps. However, Osogami and Osogami et al. demonstrated that applying risk-sensitive utility functions to cumulative rewards results in time-inconsistent behavior. An agent might choose a strategy at the start of a trajectory but reject it midway through, even if no new information has been revealed.
To resolve this, Ku et al. developed actor-critic frameworks that optimize static risk measures across the entire trajectory as a unified whole. By avoiding step-by-step local risk evaluations, these models prevent erratic policy shifts and ensure stable, long-term autonomous planning in high-stakes domains like finance and robotics.
Key Foundational Work
Behavioral Economics (2000) — contributed 2 concept elements to this direction.