Where Model Errors Bite: Value-Function Kinks and Rollout Horizons

Model-based reinforcement learning trains a policy on imagined trajectories. A learned simulator is rolled forward several steps by feeding its own predictions back in, and the predicted cost along that imagined path is used as a training target. Errors in those predictions compound with each step. The usual remedy is to shorten the imagined rollout wherever the simulator is inaccurate. We argue that this targets the wrong quantity. In optimal control, the value function (the best achievable cost-to-go from each state) is typically not smooth. It has creases, or "kinks," along the surfaces where the best decision flips, for example swinging a pendulum left versus right. Away from a kink, a small blur in the predicted state changes the value target only by an amount proportional to the square of the blur. On a kink, the change is proportional to the blur itself. So the same simulator error that is harmless in open regions becomes the dominant source of bias near decision boundaries, even if the simulator is equally accurate everywhere.

The proposal makes this precise and uses it to build a rollout-horizon rule. The connecting idea comes from stochastic control. Averaging a value function over random perturbations is what noise does to an optimal control problem: it adds a second-order "viscosity" term to the Hamilton–Jacobi–Bellman (HJB) equation, the partial differential equation the value function satisfies. Accumulated rollout error therefore behaves like extra artificial noise. Its effect on the value is the classical vanishing-viscosity error, which is quadratic in smooth regions and linear at concave kinks. We measure this effect directly with a cheap Jensen-gap estimator. The rule then picks, state by state, the longest horizon whose predicted kink-induced bias stays below a tolerance. We test this on problems where the exact non-smooth value function is known.

Core Claim

Let $V$ be the value function, $x$ a state, and $s$ the scale of accumulated prediction error after $H$ rollout steps. Write that error as $s\,\xi$, where $\xi$ is a standardized random direction. The bias that the error injects into a bootstrapped value target is governed by the Jensen gap

$$ J(x,s) = \mathbb{E}_{\xi}\big[V(x + s\,\xi)\big] - V(x). $$

Where $V$ is twice differentiable, $J(x,s) \approx \tfrac{s^2}{2}\,\mathrm{Tr}\,\nabla^2 V(x)$. This is exactly the diffusion term of the second-order HJB generator. At a concave kink, where $V$ is locally the minimum of two smooth branches, $J(x,s)$ is negative and of order $s$. A concave kink is the typical shape for semiconcave cost-to-go functions, which can be written as pointwise minima of smooth functions, as in Representation of practical nonsmooth control Lyapunov functions by piecewise affine functions and neural networks. The order-$s$ scaling follows from $\mathbb{E}[\min(a,b)]$ falling below $\min(\mathbb{E}a,\mathbb{E}b)$ by an amount proportional to the spread when the two branches tie.

Suppose rollout errors grow roughly like a random walk, $s_H \approx \sqrt{H}\,\delta$, where $\delta$ is the one-step error. Then bias grows like $H\delta^2$ away from kinks and like $\sqrt{H}\,\delta$ on them. The sign also matters. For cost minimization the bias near cut loci is optimistic, so imagined rollouts make decision boundaries look cheaper than they are.

The claim has three parts. The first is that compounding-error bias in value targets concentrates on the non-smooth set of $V$ and has these two distinct scalings. The second is that a horizon rule which caps $|J(x, s_H)|$ outperforms rules based on simulator error alone. The third is that true process noise of size $\sigma$ smooths kinks over a width proportional to $\sigma$, so the special kink effect should disappear once $s_H \ll \sigma$. That crossover gives a falsifiable prediction tied to the stochastic HJB.

Relation to Prior Work

Three works motivate the gap. A Study of Reinforcement Learning in the Continuous Case by the Means of Viscosity Solutions shows two things. Value functions are non-smooth even for smooth dynamics. Among the infinitely many almost-everywhere solutions of the HJB equation, only the viscosity solution is the true value; the viscosity solution is the unique weak solution singled out by a test-function condition that tolerates kinks. That uniqueness is why we measure bias against the viscosity solution and not against any residual-minimizing fit. On the Nonsmooth Geometry and Neural Approximation of the Optimal Value Function of Infinite-Horizon Pendulum Swing-up reconstructs the exact non-smooth pendulum value, including its spiral cut-locus curves. It does this by integrating Pontryagin's principle backward from a local quadratic regulator. The authors explicitly leave closed-loop rollout drift unstudied. We reuse their reconstruction as ground truth. Learning with Imperfect Models: When Multi-step Prediction Mitigates Compounding Error proves how single-step and multi-step predictors trade bias against variance, but only for linear systems with quadratic, smooth values. Our analysis covers exactly the regime their analysis excludes.

The closest method is Learning to Combat Compounding-Error in Model-Based Reinforcement Learning, called AdaMVE. It learns a cumulative simulator-error function by temporal-difference learning, using a Wasserstein discrepancy as the per-step error, and shortens horizons where that error is large. AdaMVE weights rollouts by how wrong the simulator is. We weight them by how much a given amount of wrongness corrupts the value, which depends on the local geometry of $V$. The two coincide only if simulator error happens to peak at kinks.

Hamilton-Jacobi-Bellman Equations for Q-Learning in Continuous Time supplies the action-aware value with unique viscosity characterization for a deterministic continuous-time variant. Post-hoc denoising such as Thermalizer: Stable autoregressive neural emulation of spatiotemporal chaos constrains the design space. It relies on a stationary invariant measure, which goal-directed controlled trajectories lack, so we do not pursue state-correction.

Path-dependent extensions are out of scope beyond one stretch experiment. The uniqueness theory of Equivalence of minimax and viscosity solutions of path-dependent Hamilton-Jacobi equations is first-order only, so any delayed, stochastic result would be empirical.

Research Program

The analytic part comes first. We will prove the two-regime bound on $J(x,s)$ for value functions that are semiconcave and piecewise $C^1$, with constants given by the curvature of the branches and the angle between branch gradients at the kink. We will then relate it to the vanishing-viscosity error of the corresponding stochastic HJB, so that the "rollout error equals artificial viscosity" statement holds as an inequality and not just an analogy. The random-walk growth assumption for $s_H$ will be checked empirically, not assumed. Where error grows super-linearly, as in the misspecified regime of Somalwar et al., the horizon rule uses the measured $s_H$ directly.

The algorithm, which we call Jensen-gap value expansion, keeps the standard model-based value expansion machinery. It adds two estimated quantities. The first is $s_H(x)$, the calibrated spread of $H$-step predictions, obtained from validation residuals or ensemble disagreement. The second is $\hat J(x, s)$, computed by sampling a few perturbations around rollout endpoints and evaluating the current value network. The horizon at each state is the largest $H$ with $|\hat J(x, s_H)| \le \tau$, where $\tau$ is a single tolerance in cost units. The value network is parameterized as a minimum over a small number of smooth heads, following the min-of-smooth representation result, so that kinks are representable and not smeared. Two diagnostics come for free: the ratio $\hat J(x,s)/s^2$ evaluated across several $s$, and the tie gap between the top two heads. Both flag the non-smooth set.

Baselines

The naive reference is fixed-horizon model-based value expansion with a hand-tuned horizon. It shows whether any state dependence helps. A pure model-free reference, Proximal Policy Optimization, is included because On the Nonsmooth Geometry and Neural Approximation of the Optimal Value Function of Infinite-Horizon Pendulum Swing-up reports that it reaches higher trajectory cost than the optimal-value controller on swing-up. That paper also reports that model predictive control comes close; no reported numerical cost for any baseline was found on Lacuna.

The strongest competitor is AdaMVE from Learning to Combat Compounding-Error in Model-Based Reinforcement Learning. It is the only existing method that adapts horizon per state using a learned estimate of compounding error. Its reported advantage over fixed-horizon value expansion and STEVE on HalfCheetah and Swimmer is shown only graphically, so no reported numerical result was found on Lacuna for AdaMVE. We will therefore rerun it ourselves.

We also include an ensemble-uncertainty horizon rule that truncates when ensemble spread exceeds a threshold, and a simulator trained with the hybrid multi-step rollout loss of Somalwar et al. combined with fixed horizons.

To keep comparisons fair, all methods share the same simulator architecture, data, policy learner, environment-step budget and number of value-network evaluations. Each gets the same hyperparameter-search budget, about twenty configurations, over its own horizon or threshold parameters. AdaMVE's error function and our Jensen-gap estimate receive matched compute.

Two ablations isolate the key ingredient. The first keeps our per-state horizons but replaces $\hat J$ with the smooth-curvature estimate $\tfrac{s^2}{2}\mathrm{Tr}\nabla^2 V$; this estimate is blind to kinks, so it should underperform if the kink term matters. The second keeps $\hat J$ but redistributes the same average horizon uniformly across states. Beating AdaMVE means two things on pendulum swing-up and car-on-the-hill. Value-target bias near cut loci must be lower with non-overlapping confidence intervals over seeds. Final trajectory cost or sample efficiency must improve at matched environment steps, with no loss on the MuJoCo tasks.

Why not just…?

Why not just use AdaMVE's learned simulator-error function? This is the real threat. Where simulator error happens to peak at decision boundaries, AdaMVE should already capture most of the benefit. AdaMVE's own doorway examples in FourRoom suggest this often happens. The proposal settles the question with a controlled experiment. We use a perfect simulator with injected isotropic Gaussian error. In that setting the per-step Wasserstein error is the same everywhere, so AdaMVE has no signal to adapt on, while our theory predicts the bias is concentrated on the cut locus. If AdaMVE matches us there, or if in natural settings the two horizon maps are nearly identical, our contribution reduces to an explanation of why AdaMVE works rather than a better method. We will report it that way.

Why not just train the simulator with a multi-step loss, or predict multi-step directly? Learning with Imperfect Models: When Multi-step Prediction Mitigates Compounding Error shows this reduces error growth with horizon. It does not remove residual error, and any residual of size $s$ still produces order-$s$ bias at kinks. The approach is therefore complementary. We test it in two ways: as a baseline, and with the hybrid-loss simulator plugged into our horizon rule. The prediction is that gains stack.

Why not just use a short horizon everywhere? A uniformly short horizon avoids kink bias but discards model rollouts in smooth regions, where bias is only quadratic in $s$. The uniform-redistribution ablation and the fixed-horizon sweep measure exactly this trade-off. If some fixed horizon matches our method's cost-versus-sample curve, the state dependence is not earning its keep.

Why not just fit the HJB residual with a neural network and skip rollouts? Munos shows that residual minimization can converge to spurious almost-everywhere solutions with zero residual. PDEBench: An Extensive Benchmark for Scientific Machine Learning reports that physics-informed networks struggle on problems with sharp interfaces. A residual-trained value network is included only as a diagnostic reference on pendulum, scored against the exact viscosity solution.

Proposed Experiments

The decisive check comes first. It uses the pendulum swing-up with the Han–Yang reconstructed value as ground truth and a perfect simulator corrupted by injected Gaussian error of known $\delta$. We measure value-target bias as a function of horizon $H$ and of distance to the cut locus. The theory predicts log–log slopes near 1 in $H$ away from the cut locus and near 1/2 on it, with negative (optimistic) sign at the kink. If bias does not localize on the cut locus or the slopes do not separate, the core claim is wrong and we stop.

Second, we add Brownian process noise of size $\sigma$ to the pendulum and car-on-the-hill dynamics; the latter is the benchmark from Munos with steep value discontinuities. We verify the crossover: the kink-specific bias should vanish once $s_H$ falls well below $\sigma$, because the true stochastic value is already smoothed over that width.

Third, we run the full comparison with learned simulators on pendulum swing-up, car-on-the-hill, and cart-pole, to which Han and Yang report their approach generalizes. Metrics are trajectory cost from a grid of initial states, swing-up success rate, and environment steps to a cost threshold.

Fourth, to check that the method does not hurt where kinks are less prominent, we run HalfCheetah and Swimmer under the AdaMVE protocol.

As a stretch, we add a fixed input delay to the pendulum with a history-window state. This probes whether the same localization appears when the value lives on path space; no second-order comparison theory exists there, so the result would be purely empirical.