Some decision problems have two features at once. The first is randomness whose size depends on where the system is. In a portfolio, the dollar swings grow with how much is invested. In a muscle, the jitter grows with how hard it contracts. The second is an agent that discounts the future non-exponentially, for example hyperbolically. Such an agent's plan made today will look wrong to its own future self, so the standard recursive (Bellman) planning machinery no longer applies. The best current method for such problems, Pontryagin-Guided Direct Policy Optimization, avoids Bellman recursion altogether. It backpropagates through simulated rollouts and finds a "time-consistent equilibrium" policy, meaning one that no future self wants to deviate from. It does this only when an exact, differentiable simulator is available (Beyond the Bellman Recursion). In practice the state-dependent noise has to be learned from data. Learned stochastic models are known to underestimate noise (Improving the noise estimation of latent neural SDEs).
This proposal asks a narrow, testable question: how does error in a learned noise model propagate into the equilibrium policy of a present-biased agent? We conjecture that the propagation is worse than for an ordinary exponentially discounting agent. The reason is that each "self" optimizes against future selves whose behavior also depends on the misestimated noise, so the error enters twice. If this holds, the noise model should be judged by its effect on the downstream equilibrium, not by likelihood. The fitting method should then change when the world model will feed a time-inconsistent planner. If it does not hold, then the existing direct-optimization method can be paired with any reasonable noise estimator, and that is also worth knowing.
Core Claim
Let $\sigma(x)$ be the true state-dependent diffusion (noise scale). Let $\pi^*_{\hat\sigma}$ be the equilibrium policy computed by Pontryagin-guided optimization through a model whose diffusion is $\hat\sigma$. We define the policy sensitivity to a relative diffusion error $\epsilon$ as
$$E(\epsilon) = \mathbb{E}_{x \sim \rho}\,\big|\pi^*_{(1+\epsilon)\sigma}(x) - \pi^*_{\sigma}(x)\big|,$$
where $\rho$ is the state distribution visited under the true equilibrium policy. This is the same $L_1$ policy error used by Beyond the Bellman Recursion.
The claim has two parts. First, the slope of $E(\epsilon)$ near zero is larger under hyperbolic discounting than under exponential discounting with a matched effective horizon, and the gap grows with the present-bias strength. Second, choosing the noise estimator to minimize downstream $E$ yields policies closer to the true-simulator equilibrium than choosing it by likelihood.
The mechanism is a hypothesis, not a derivation. In the Merton investment-consumption problem with constant relative risk aversion, the risky allocation does not depend on the discount function. Any amplification should therefore show up in consumption. There, the equilibrium consumption rule integrates the whole discount curve against wealth growth rates that themselves depend on $\sigma$. This prediction is specific and falsifiable. Amplification should appear in consumption error and not in allocation error.
Research Program
The work has three parts.
Measuring sensitivity directly. We perturb the true diffusion by a controlled factor $(1+\epsilon)$ and rerun Pontryagin-guided optimization through the perturbed simulator. This traces $E(\epsilon)$ for exponential, quasi-hyperbolic, and hyperbolic discounting across a range of present-bias strengths. No learning is involved, so this experiment isolates the core claim.
Running the optimizer through learned models. We train world models from trajectories using four estimators:
- a standard variational latent SDE;
- the same model with the explicit noise penalty of Improving the noise estimation of latent neural SDEs;
- simulation-free SDE Matching (SDE Matching), which fits state-dependent volatility without backpropagating through a solver;
- a direct regression of $\sigma(x)^2$ on squared increments, which estimates noise from observed quadratic variation.
We then run the Pontryagin adjoint projection through each model. Strong state-dependent noise makes explicit Euler rollouts blow up, which would corrupt the costates (the gradients of value with respect to state that the method backpropagates). For that reason the rollouts use the tamed explicit schemes of Strong convergence for stochastic Burgers equations. We separately measure the bias that taming introduces into the costates.
Fitting to the downstream target. We fit the diffusion so that the predicted equilibrium is stable under small perturbations of $\hat\sigma$. Concretely, we tune the noise-penalty weight, whose lack of a calibration rule is an open problem noted in the file, by held-out policy-consistency error rather than by likelihood. This is the only new estimation ingredient. It is deliberately small, because the main contribution is the measurement.
Baselines
Naive reference points. The do-nothing reference is the analytic Merton policy computed with exponential discounting at a matched horizon and applied as if the agent had no present bias. This is the policy a practitioner would deploy by ignoring time inconsistency. The second reference is the certainty-equivalent convex reformulation of A Certainty Equivalent Merton Problem, run as model predictive control. That paper reports solving a 500-asset, 50-period instance in about two seconds with ECOS. Its exact equivalence to the stochastic problem holds only under exponential discounting. Beating it means lower $L_1$ error against the true hyperbolic equilibrium at comparable compute.
Strongest existing method, as an upper bound. This is Pontryagin-guided optimization with the exact simulator (Beyond the Bellman Recursion). It reports beating PPO, physics-informed networks, and deep backward-stochastic-Volterra solvers "by orders of magnitude in terms of $L_1$ error," with under 0.02 seconds of CPU latency per step. No exact numeric $L_1$ value was found on Lacuna for this benchmark. We will therefore reproduce it and use our reproduced number as the ceiling. Success for a learned-model variant means closing most of the gap between this ceiling and the naive references, at equal trajectory budget.
Model-free baselines. We run PPO and the two-timescale quasi-hyperbolic actor-critic of Reinforcement Learning with Quasi-Hyperbolic Discounting. The latter is the only model-free method in the file with convergence guarantees to equilibria. It was shown only on discrete inventory control, so we discretize the action space. Both get the same number of environment transitions as the world-model pipeline and the same hyperparameter search budget.
Estimator comparison. All four noise estimators share one architecture, one data budget, and one number of tuning trials. For noise recovery itself, the noise-penalty paper reports matching bistable transition rates in the Energy Balance climate model at penalty weight 200, but gives no numeric table on Lacuna. SDE Matching reports a Motion Capture test MSE of 4.50 ± 0.32 versus 4.03 ± 0.2 for the adjoint latent SDE.
Ablations. Removing taming isolates its effect on the costates. Swapping the downstream fitting criterion back to likelihood isolates the calibration ingredient.
Why not just…?
Why not just estimate the noise from squared increments and plug it in? This is the most serious threat. With full-state observation at high frequency, quadratic variation identifies $\sigma(x)$ almost exactly, and the latent-model bias may not matter at all. We will settle this by sweeping the observation interval and adding observation noise. Our proposal earns its place only in regimes, such as sparse sampling or partially observed states, where the simple regression's $E$ is measurably worse. If no such regime appears on these benchmarks, we will report that the simple estimator suffices.
Why not just use PPO, perhaps with multiple discount heads? Beyond the Bellman Recursion reports that PPO is biased or high-variance on exactly the hyperbolic Merton benchmark. Reinforcement Learning with Quasi-Hyperbolic Discounting shows that standard actor-critics adapted to quasi-hyperbolic discounting fail to reach equilibria. The multi-horizon heads of Hyperbolic Discounting and Learning over Multiple Horizons learn values for a hyperbolic objective. They do not compute an equilibrium among selves. Its 0.002 Pathworld value MSE concerns value estimation, not the policy. We still run PPO as a baseline.
Why not just inflate the estimated noise by a safety margin? This is cheap and may be competitive. Our sensitivity curve $E(\epsilon)$ answers it directly. If $E$ is roughly symmetric in $\epsilon$, overestimating noise costs as much as underestimating it, and a margin does not help. If $E$ is asymmetric, a margin is a legitimate fix, and we will report it as one.
Why not just certainty-equivalent model predictive control? A Certainty Equivalent Merton Problem itself notes that its exactness depends on exponential discounting, and that model predictive control is otherwise a heuristic. We include it as a reference to quantify how large that heuristic gap is.
Proposed Experiments
Primary benchmark. The main testbed is the hyperbolic Merton problem from Beyond the Bellman Recursion, in 1 to 100 dimensions. We use its ground-truth equilibrium and report $L_1$ policy error separately for consumption and allocation.
Noise-recovery checks. We validate the estimators on the Energy Balance model and the Ornstein-Uhlenbeck process from the noise-penalty paper. We report transition rates and Kramers-Moyal coefficients, which are the drift and diffusion estimates obtained from the moments of observed increments.
Secondary testbed. We take the 2-link, 6-muscle arm from Deep 2FBSDEs, where noise grows with control effort, and replace its objective with quasi-hyperbolic discounting. This tests whether the effect extends beyond portfolios. No equilibrium ground truth exists there, so we report the gap to exact-simulator Pontryagin-guided optimization.
Decisive checks. There are three:
- The ratio of the slope of $E$ at small $\epsilon$ under hyperbolic versus exponential discounting must exceed one and increase with present bias. Otherwise the core claim is false.
- That amplification must appear mainly in consumption, as the mechanism predicts.
- Downstream-calibrated estimators must beat likelihood-trained ones in at least the sparse-sampling regime.
Seeds. All results use 10 seeds, with confidence bands.
References
- Beyond the Bellman Recursion: A Pontryagin-Guided Framework for Non-Exponential Discounting
- Improving the noise estimation of latent neural stochastic differential equations
- SDE Matching: Scalable and Simulation-Free Training of Latent Stochastic Differential Equations
- Strong convergence for explicit space–time discrete numerical approximation methods for stochastic Burgers equations
- A Certainty Equivalent Merton Problem
- Reinforcement Learning with Quasi-Hyperbolic Discounting
- Hyperbolic Discounting and Learning over Multiple Horizons
- Deep 2FBSDEs for Systems with Control Multiplicative Noise