People and animals discount the future steeply over short delays and gently over long ones. One consequence is that they reverse their preferences: the plan that looked best this morning stops looking best by afternoon, with no new information in between. Once the discount curve is anything other than exponential, the standard reinforcement learning machinery stops working, because the Bellman recursion assumes that today's preferences and tomorrow's agree. The problem gets harder in continuous time, meaning physical systems that evolve smoothly rather than in fixed steps. Ordinary value learning already degrades there as the control step shrinks. The methods that do handle present bias in continuous time either need the equations of motion or need a differentiable simulator.
This note proposes a model-free, continuous-action actor-critic that learns the sophisticated equilibrium: the behaviour of an agent who knows its future selves will re-optimize, and who therefore picks, at every state, the action its current self prefers given how its future selves will act. The construction rests on one observation. If the discount curve is written as a weighted mixture of exponentials (hyperbolic discounting is exactly such a mixture), then the equilibrium condition in continuous time reduces to acting greedily on a weighted sum of ordinary exponentially-discounted evaluation critics of the agent's own current policy. Each critic is a standard exponential object, so it can be trained with existing continuous-time losses that are robust to small control steps. All of the time-inconsistency is concentrated in one place: the policy has to be a fixed point of its own critics.
Core Claim
Let the discount function be $\varphi(\tau)=\sum_{i=1}^K w_i e^{-\rho_i \tau}$, with weights $w_i\ge 0$ summing to one and rates $\rho_i>0$. For a stationary policy $\pi$, let $V_i^\pi(x)$ be its ordinary value at rate $\rho_i$. Let $q_i^\pi(x,a)=r(x,a)+(\mathcal{L}^a V_i^\pi)(x)-\rho_i V_i^\pi(x)$ be its advantage rate, where $\mathcal{L}^a$ is the generator of the controlled diffusion $ds_t=f(s_t,a_t)dt+\sigma(s_t)dW_t$, that is, the instantaneous expected rate of change of a function when action $a$ is held. A stationary feedback policy is a continuous-time sophisticated equilibrium exactly when
$$\pi(x)\in\arg\max_a \sum_{i=1}^K w_i\, q_i^\pi(x,a)\quad\text{for every state } x.$$
The derivation is short. Consider a self at state $x$ that deviates to action $a$ for a short window $\epsilon$ and then hands control back to $\pi$. Its payoff is $\int_0^\epsilon \varphi(s) r\,ds+\mathbb{E}\big[\sum_i w_i e^{-\rho_i\epsilon}V_i^\pi(X_\epsilon)\big]$. Expanding this to first order in $\epsilon$ gives $\epsilon\sum_i w_i q_i^\pi(x,a)$ plus terms that do not depend on $a$.
Hyperbolic discounting fits this form exactly, because $1/(1+kt)=\int_0^\infty e^{-\rho t}\,k^{-1}e^{-\rho/k}\,d\rho$. A $K$-node quadrature over $\rho$ therefore gives an approximation whose error is controlled by the quadrature rule rather than by an arbitrary curve fit. The heads must be evaluation critics of one shared policy. They are not optimal critics for their own rates, and that coupling is what separates this proposal from multi-horizon RL.
Relation to Prior Work
Two papers motivate the gap by solving continuous-time non-exponential control with privileged model access. Reinforcement Learning with Non-Exponential Discounting recasts hyperbolic discounting as a decreasing termination hazard. It solves a time-augmented HJB equation (the continuous-time Bellman partial differential equation) by deep collocation, which requires the known dynamics $f$, and it restricts the maximization to a finite action set. It leaves the model-free version explicitly open. Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discounting avoids value functions altogether. It estimates costates through a differentiable simulator and maximizes a local Hamiltonian to obtain equilibrium controls, and it reports large accuracy gains on Merton, survival-target and resource problems. Our method needs neither a model nor gradients through the simulator. It also learns a stationary feedback policy, whereas their approach runs Monte Carlo rollouts at execution time.
Two papers supply the method. Continuous Soft Actor-Critic replaces the temporal-difference loss with martingale orthogonality conditions, which avoids the trap where TD error converges to diffusion noise as the control step shrinks. It also derives learning-rate scaling with the control step. A Temporal Difference Method for Stochastic Continuous Dynamics estimates drift and diffusion terms model-free, with a linear-cost Hessian trick, and scales to Humanoid. Both assume a single constant rate. The research file lists "martingale conditions under time-dependent hazard" as unsolved. In our formulation that problem never comes up, because each head has a constant rate, so the existing losses apply head by head.
One paper constrains the actor design. Reinforcement Learning with Quasi-Hyperbolic Discounting shows in discrete time that policy gradients on the present-biased objective converge to non-equilibria, and that equilibria can be multiple. It finds equilibria with a two-timescale actor-critic on inventory control. We keep its two-timescale structure and its warning. Our actor ascends the local mixed advantage rate, not the gradient of a time-0 return. Our setting is the $\delta t\to 0$ limit with continuous actions, which that work does not treat. Continuous Q-Score Matching shows that $Q(s,a)\to V(s)$ as the step shrinks. This is why the critics must output advantage rates rather than step-$Q$ values.
One paper appears to contradict us. The Ko & Huh paper argues that linear combinations of exponential value functions do not resolve time-inconsistency. That argument applies to combining heads that are each optimal for their own rate. Such heads correspond to different policies, so their weighted sum is not the value of any single agent. Our heads all evaluate the same policy, and the policy is required to be a fixed point. We test this distinction directly with an ablation below.
Two further papers bear on the design. Non-exponential Reward Discounting in Reinforcement Learning uses multi-horizon heads mainly for representations and hazard estimation, and does not target the sophisticated equilibrium. A Continuous-time Tractable Model for Present-biased Agents gives closed-form abandonment thresholds for deterministic one-dimensional progress tasks, which we use as ground truth.
Research Program
On the theory side, we will prove the equilibrium characterization above for mixture discounts under standard regularity conditions. We will bound the change in equilibrium behaviour caused by replacing the hyperbolic density with a $K$-node quadrature. We will also state honestly where the result does not apply. Pure hyperbolic discounting with persistent rewards diverges over an infinite horizon, since the mixing density does not vanish near $\rho=0$. We will therefore impose a minimum rate $\rho_{\min}$ (a background hazard) or a finite horizon, and report how sensitive results are to that choice. Discounting that depends on calendar time, not just on delay, falls outside this time-homogeneous construction. The Ko & Huh resource benchmark has this property, and we treat it only as a stress test.
On the algorithm side, a shared network trunk outputs $K$ value heads $V_i$ and $K$ advantage-rate heads $q_i$. Each pair is trained on replay data with the martingale orthogonality loss of Continuous Soft Actor-Critic at its own rate. Diffusion-dominated tasks can optionally use the second-order terms of β-dTD. A Gaussian actor ascends $\sum_i w_i q_i(x,a)$, with an optional entropy term. The critics run on the faster timescale, following the two-timescale logic of Eshwar et al., and learning rates scale with $\delta t$ as CSAC prescribes. Because equilibria can be multiple, we will report which equilibrium is reached across seeds and initializations, rather than assuming uniqueness.
On the diagnostics side, we define an equilibrium gap: the largest improvement in the current self's objective available from a short deviation window, estimated by Monte Carlo branching rollouts from sampled states. This lets us certify equilibrium on tasks where no analytic solution exists.
Baselines
The naive reference point is exponential Continuous Soft Actor-Critic with a single rate, chosen so that its effective horizon matches the hyperbolic curve's. It runs on the same replay data, network size and tuning budget. It shows what an agent loses by ignoring present bias entirely. Beating it means matching the analytic equilibrium on Merton and the investment task in cases where it cannot, for example by reproducing the early-spend-then-invest reversal.
The strongest existing method is PG-DPO from Beyond the Bellman Recursion. It is the strongest competitor because it is the only method in the file reported to track analytic equilibria in continuous state and action spaces, up to 100 dimensions. Its stated results are qualitative: it outperformed PPO, PINNs and Deep BSVIE "by orders of magnitude in terms of $L_1$ error," with inference under 0.02 seconds on a CPU. For its policy $L_1$ error, no reported result was found on Lacuna. We will rerun it on its own three benchmarks with its differentiable simulator and report its errors. It receives strictly more information than our method, so we treat it as an informed upper reference. The success criterion is $L_1$ error within a small factor of PG-DPO's, using only black-box transitions, at a small fraction of its per-step inference cost. Matching it outright would be a strong result, not a requirement.
The model-based reference is the deep collocation solver of Schultheis et al., given the true dynamics and a finely discretized action grid. We compare on the investment and line problems, where its preference reversals are documented.
The discrete-time references are the two-timescale quasi-hyperbolic actor-critic of Eshwar et al. and discrete SAC with a hyperbolic return. Both run at a sweep of control steps. The relevant evidence from the file concerns exponential discounting: on VMAS Sampling, discrete MASAC falls from a median of 0.66 to 0.43 when $\delta t$ shrinks from 0.1 to 0.01, while CMASAC rises from 0.84 to 0.92 (Continuous Soft Actor-Critic). Beating these references means a flat accuracy curve across the sweep while theirs degrades.
The key-ingredient ablation keeps the same $K$ heads but trains each as an optimal critic for its own rate. The actor is then greedy on their weighted sum. This is the multi-horizon reading of Far et al.. A second ablation swaps the martingale loss for plain temporal-difference error. A third varies $K$.
Why not just…?
Why not just discretize time finely and run discrete quasi-hyperbolic RL? The known failures of discrete critics as the step shrinks, namely action dependence vanishing (Continuous Q-Score Matching) and TD error converging to noise (Continuous Soft Actor-Critic), are properties of the critic, not of the discount. We have no direct evidence that they carry over to present-biased objectives, so the $\delta t$ sweep tests this. If the Eshwar actor-critic stays accurate at $\delta t=0.001$ on Merton, the continuous-time component of our proposal is unnecessary, and we would say so.
Why not just approximate the hyperbolic curve with a mixture of exponentials? We do approximate it this way, so this alternative is the real threat. The open question is what each head should represent. If each head learns its own optimal value, the heads describe different agents, and the objection Ko & Huh raise to combining them applies. Our claim is that the heads must evaluate one shared policy that is greedy on their sum. The optimal-heads ablation settles this. If it reaches the same equilibrium gap and $L_1$ error, our contribution shrinks to the theorem alone.
Why not just run PPO on the hyperbolic return? That procedure optimizes the plan as seen from time 0. This is the resolute, precommitted policy. Eshwar et al. show it leaves later selves with an incentive to deviate, and Ko & Huh report that PPO is biased with high variance on Merton. We include PPO as a baseline and measure its equilibrium gap. We expect that gap to be large even when its time-0 return is higher, which is exactly the signature that distinguishes a precommitted policy from an equilibrium.
Proposed Experiments
The main accuracy test uses Merton with hyperbolic discounting and survival-discounted target control from Ko & Huh. Both have analytic equilibria. The metric is policy $L_1$ error against ground truth, swept over $\delta t\in\{0.1,0.01,0.001\}$ and over diffusion level. The investment and line problems from Schultheis et al. are run with continuous actions and test whether the method reproduces the documented preference reversal. We add a stochastic version of the progress task from Akagi et al.. In its zero-noise limit, a learned agent should quit midway at the closed-form threshold for large $k$, and an exponential agent should never quit midway. Both are checks on behaviour, not on reward.
For scale, the resource task with dimension swept from 10 to 100 tests whether our error grows with dimension, as the Ko & Huh paper reports for amortized methods, or stays roughly flat. We also run Brax Hopper, Ant and Humanoid, used by β-dTD, with a hyperbolic reward discount. These tasks have no ground truth, so we report the equilibrium gap, seed-level equilibrium multiplicity, and wall-clock time against exponential CSAC.
Two checks decide whether the core claim holds. First, on Merton the mixed-evaluation critic must beat the optimal-heads ablation in both $L_1$ error and equilibrium gap. Second, its accuracy must stay flat as $\delta t$ shrinks while the discrete references degrade. If either check fails, the proposal reduces to a known baseline.
References
- Reinforcement Learning with Non-Exponential Discounting
- Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discounting
- Continuous Soft Actor-Critic: An Off-Policy Learning Method Robust to Time Discretization
- A Temporal Difference Method for Stochastic Continuous Dynamics
- Reinforcement Learning with Quasi-Hyperbolic Discounting
- Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control
- Non-exponential Reward Discounting in Reinforcement Learning
- A Continuous-time Tractable Model for Present-biased Agents