Equilibrium Feedback Control Under Hyperbolic Discounting via Coupled Exponential Heads

People and animals value the near future far more steeply than the distant future. They are impatient about a one-day delay but patient about the difference between day 300 and day 301. In reinforcement learning, this is modeled with "hyperbolic" or other non-exponential discounting. Under it, a plan that looks best today stops looking best tomorrow, so dynamic programming (the backward recursion behind almost every value-based controller) no longer applies. The right target is then a sophisticated equilibrium: a feedback policy that no future version of the agent wants to deviate from. Computing such policies for continuous, high-dimensional systems is hard. Grid-based solutions of the Hamilton-Jacobi-Bellman (HJB) equation (the PDE whose solution gives the optimal value and controller) scale exponentially with dimension. Neural PDE solvers produce unreliable gradients. The best current method, which needs no value function at all, has to run fresh Monte Carlo simulations every time it picks an action.

We propose a different route. A hyperbolic discount is a weighted average of ordinary exponential discounts. Building on that fact, we show how to turn one intractable, non-recursive equilibrium problem into a family of ordinary exponential-discount policy-evaluation problems that share a single policy. Each member of the family is again recursive, so it can be solved with standard deep forward-backward stochastic differential equation solvers, which already scale to 12-state quadcopters. The policy is fixed by maximizing a Hamiltonian built from the weighted sum of the heads' gradients. The result is an amortized feedback controller that is evaluated with one forward pass at deployment time. It needs a model of the dynamics but not a differentiable simulator.

Core Claim

Consider a time-homogeneous discount that can be written as an exponential mixture, $D(t)=\int_0^\infty e^{-\rho t}\,p(\rho)\,d\rho$. This covers hyperbolic $1/(1+kt)$ and power-law discounts. Hyperbolic Discounting and Learning over Multiple Horizons used this representation to build multi-head Q-networks for Atari. For a fixed feedback policy $u(x)$, define one head per rate $\rho$. Each head $f_\rho(x)$ is the expected exponentially discounted reward of running $u$ from state $x$. The equilibrium value is then $V(x)=\int f_\rho(x)\,p(\rho)\,d\rho$, and each head satisfies a linear policy-evaluation equation:

$$\rho f_\rho(x) = c(x,u(x)) + \nabla f_\rho(x)\cdot b(x,u(x)) + \tfrac12 \mathrm{tr}\big(\sigma\sigma^\top \nabla^2 f_\rho(x)\big).$$

Here $b$ is the drift, $\sigma$ the diffusion coefficient, and $c$ the running reward. In the extended HJB system for sophisticated agents, the type of non-local PDE system studied abstractly in Equilibria for Time-Inconsistent Singular Control Problems, the correction term that separates equilibrium from naive optimization does not depend on the current action. So the equilibrium control is a pointwise Hamiltonian maximization using the aggregated derivatives:

$$u(x)=\arg\max_{u}\Big\{c(x,u)+\nabla V(x)\cdot b(x,u)+\tfrac12\mathrm{tr}\big(\sigma\sigma^\top\nabla^2 V(x)\big)\Big\}.$$

The difference from multi-horizon RL is precise. In Hyper-Rainbow-style methods each head is trained toward its own optimum, or the aggregate is greedily maximized as if it were Bellman-consistent. In our scheme no head optimizes anything. Every head evaluates the same shared policy at its own rate, and the equilibrium is the fixed point where the shared policy is greedy with respect to the mixture. Because each head discounts exponentially, the backward martingale property that breaks down in Volterra-type formulations holds again head by head. This is exactly the property the research file identifies as the obstacle for deep FBSDE solvers under time inconsistency.

How This Relates to Prior Work

Two papers motivate the gap. Reinforcement Learning with Non-Exponential Discounting derives a generalized HJB by treating discounting as a time-varying hazard rate and adding time to the state. It solves that HJB by collocation, a PINN-like method that minimizes the PDE residual at sampled points, but only in one-dimensional problems with discrete actions. Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discounting proves that Bellman recursion requires both multiplicativity and time homogeneity of the discount. Its method, PG-DPO, drops value functions entirely and computes costates (the gradients of future reward with respect to the current state) by backpropagating through Monte Carlo rollouts at every decision. It scales to 100 dimensions, but at the cost of online simulation and a differentiable simulator.

PG-DPO's finding that PINNs fail through compounding errors in $\nabla_x V$ also constrains our design. We never fit $V$ by global PDE residuals. We learn each head's gradient directly as the "$Z$" process of a backward SDE along simulated paths.

Deep Forward-Backward SDEs for Min-max Control supplies the method: LSTM-based FBSDE solvers with importance sampling that reach 12-state systems. We reuse that machinery per head. Reinforcement Learning with Quasi-Hyperbolic Discounting supplies a cautionary result. Its two-timescale actor-critic for discrete quasi-hyperbolic equilibria shows that equilibria can be non-unique. We therefore report which equilibrium we reach and test whether it is stable rather than assuming uniqueness.

A Nash-Type Fictitious Game Framework to Time-Inconsistent Stochastic Control Problems gives equilibria only for open-loop linear-quadratic problems. Our closed-loop linear-quadratic special case reduces to coupled Riccati-type equations, which gives an exact check.

Research Program

The work has three parts.

Construction. We discretize $p(\rho)$ by Gauss-Laguerre quadrature into $M$ heads. The heads share a trunk network with per-head gradient outputs, and the rate $\rho$ is fed in as an input, so $M$ can be changed at test time. Training alternates two steps. First, forward-simulate under the current policy and fit all heads with a per-head deep BSDE loss. Second, update a policy network toward the pointwise Hamiltonian argmax computed from the aggregated gradient (and, where diffusion depends on control, the aggregated Hessian-vector products). This requires known $b$ and $\sigma$, but no differentiation through the simulator.

Analysis. We will prove that fixed points of the scheme are equilibria in the extended-HJB sense under smoothness, inheriting the "regular equilibrium" assumptions of Equilibria for Time-Inconsistent Singular Control Problems. We will bound the policy error from truncating the quadrature at $M$ heads. For hyperbolic $p(\rho)$, the tail at small $\rho$ (long horizons) is the dangerous part, and we expect the error to be dominated by how slowly $D(t)$ decays.

Scope. The method covers time-homogeneous discounts with mixture representations. Purely multiplicative but non-stationary discounts, such as survival-discounted tasks, are already recursive after adding time to the state, and we treat them as a sanity case. For time-varying impatience that is neither multiplicative nor stationary, we test a variant whose mixture weights depend on calendar time. We state plainly that this variant may fail.

Baselines

Naive reference points. The first is an agent that ignores time inconsistency: a single exponential discount matched to the hyperbolic discount's mean horizon, solved with the same deep FBSDE solver. Any gain over it measures the value of modeling the right preferences at all. The second is the naive re-planning agent, which re-optimizes the hyperbolic objective as if its future selves will comply. A Continuous-time Tractable Model for Present-biased Agents predicts this agent abandons tasks partway through. We run it to confirm that our benchmarks expose that failure.

PG-DPO. The strongest existing method is PG-DPO from Beyond the Bellman Recursion, run with the authors' two-stage procedure on the same simulators. Its online Monte Carlo budget is matched so that its training plus per-step compute equals our training compute plus inference. The research file reports that PG-DPO beats PPO, PINNs, and Deep BSVIE "by orders of magnitude" in $L_1$ error, stays flat from 10 to 100 dimensions, and runs under 0.02 seconds per decision on CPU. No exact $L_1$ values are reported on Lacuna for PG-DPO on the Merton, survival-discounted, or resource benchmarks, so we will re-run it ourselves. Beating it means matching its $L_1$ policy error within its confidence band at every dimension while using at least 10x lower per-decision latency. Failing that, it means beating it on equilibrium-deviation gain or under perturbed initial states it did not simulate from.

Distilled PG-DPO. This is the obvious amortized competitor. We will train a feedforward policy on PG-DPO's actions, using the same total simulator calls as our method.

Learned-value baselines. We also compare against Deep BSVIE, the time-augmented collocation solver of Reinforcement Learning with Non-Exponential Discounting extended to continuous actions, and PPO with a hyperbolic weighting lookup. All receive equal tuning budgets.

Key ablation. We keep the architecture but train each head to optimize its own exponential objective, or maximize the aggregate greedily, as multi-horizon RL does. This isolates the "evaluate a shared policy" ingredient.

Why not just…?

Why not just distill PG-DPO into a feedforward policy? This is the most serious threat, and we treat it as a co-primary baseline rather than dismissing it. Distillation inherits PG-DPO's need for a differentiable simulator during training. It also only learns actions on states PG-DPO visited, and it yields no value or costate field to check for equilibrium. Our method needs only forward simulation with known drift and diffusion. The deciding experiments are three: accuracy at matched simulator calls, accuracy on initial states outside the training distribution, and a non-differentiable variant of the resource benchmark that adds clipped and discontinuous dynamics. If distillation matches us on all three, the contribution reduces to the equilibrium certificate, and we will say so.

Why not just sum exponential HJB solutions, as in multi-horizon RL? The research file notes that the argmax of a sum of Hamiltonians is not a combination of per-head argmaxes. It also cites Reinforcement Learning with Quasi-Hyperbolic Discounting for the point that greedy action selection on aggregated values does not produce Markov Perfect Equilibria. Our ablation is exactly this alternative. It should show measurable deviation gains even when its $L_1$ error looks reasonable.

Why not just add time to the state and solve one HJB with a PINN? Time augmentation makes survival-type discounting recursive, but not hyperbolic discounting of a sophisticated agent. Even when it applies, PG-DPO reports that PINN gradient errors corrupt the resulting controls. We include it as a baseline. We expect it to fail beyond low dimensions.

Why not just use PPO with hyperbolic weights? The file reports PPO diverging or settling into high-variance policies on these benchmarks. On the separate, exponentially discounted PackCooling problem, PPO reaches cumulative reward -3970.02 versus -881.55 for the physics-informed HJBPPO, per Actor-Critic Methods using Physics-Informed Neural Networks. We run PPO for completeness only.

Proposed Experiments

Exact linear-quadratic check. We first test on closed-loop linear-quadratic problems with hyperbolic discounting, where the coupled Riccati-type equations give the true equilibrium. We sweep $M$ and dimension to measure quadrature and approximation error separately.

PG-DPO benchmarks. We then run on the three benchmarks of Beyond the Bellman Recursion, with the 10-to-100-dimension sweep. We report $L_1$ policy error against their ground truth, per-decision CPU latency, and training simulator calls.

Stress test on a physical system. We modify the 12-state quadcopter of Deep Forward-Backward SDEs for Min-max Control to use a hyperbolic running-cost discount. No reported result is found on Lacuna for any non-exponential version of that task, so all numbers there will be new.

Equilibrium deviation gain. The decisive check is the equilibrium deviation gain, a direct test of sophistication. From states sampled along the learned closed-loop trajectories, we let a future self deviate optimally over a short window $\epsilon$ and then return to the policy. We measure its improvement in hyperbolic value, normalized by $\epsilon$, using independent high-sample Monte Carlo. A true equilibrium drives this toward zero as $\epsilon$ shrinks.

Predicted outcome. Our method and PG-DPO should both show near-zero deviation gain. The "optimize each head" ablation and naive re-planning should show clearly positive gain. Our method should match PG-DPO's error at a small fraction of its deployment cost. If the ablation achieves near-zero deviation gain too, the shared-policy coupling is not what matters and the claim fails. If distilled PG-DPO matches us at equal compute, the practical case for the method narrows to black-box simulators.