Turnpikes for Risk-Averse Stochastic Control Under Model Uncertainty

When an optimal controller plans over a long horizon, its trajectory usually hurries to a best operating point, stays near it for most of the horizon, and leaves only at the end. This is the turnpike property. It is the reason a controller that repeatedly re-plans over a short window can still behave well without any terminal cost. In deterministic continuous time the theory is settled. The property is equivalent to strict dissipativity: there is an energy-like "storage function" such that any deviation from the best steady state strictly worsens long-run cost (On turnpike and dissipativity properties of continuous-time optimal control problems). The theory stops working once the plant is driven by noise and the noise model is itself uncertain. Noisy states never settle at a point. Gros and Zanon showed that the natural linear storage functions then cannot certify stability. Their fix is to move to nonlinear functionals on probability distributions, but only in discrete time and with a single known noise model (Economic MPC of Markov Decision Processes).

This note proposes a continuous-time theory for stochastic plants whose drift and volatility are only known to lie in a set, with cost evaluated by a worst-case (sublinear) expectation. It rests on two claims. First, in continuous time the right notion of "strict" is strictness in the control, not in the state: the rotated cost must penalize deviation from an optimal feedback law. With that change, an ordinary state-space storage function works despite the Gros–Zanon impossibility result, and convergence of the state distribution comes from ergodicity rather than from the storage function. Second, the certificate only means something when the set of plausible models is stable under pasting. Pasting stability means that switching from one plausible model to another at a random time still gives a plausible model. Nutz and Soner show this is exactly what makes worst-case conditional expectations obey the Bellman tower property (Constructing sublinear expectations on path space). We will test whether dropping it breaks the turnpike of a controller that re-plans in closed loop.

Core Claim

Consider a controlled diffusion $dX_t = b(X_t,u_t,\theta_t)\,dt + \sigma(X_t,u_t,\theta_t)\,dW_t$. Here $X_t$ is the state, $u_t$ the control, and $\theta_t \in \Theta$ an unknown model parameter that an adversary may switch at any time. The admissible laws of $X$ form a pasting-stable set. The long-run objective is the worst-case expected running cost $\ell(x,u)$. We call the system robustly feedback-strictly dissipative if there are a storage function $v$ on the state space, a constant $\lambda$ (the optimal long-run risk-adjusted cost), a feedback law $\kappa$, and a positive-definite function $\alpha$ such that, for all $x$ and $u$,

$$ \ell(x,u) - \lambda + \max_{\theta\in\Theta}\Big[b(x,u,\theta)\cdot\nabla v(x) + \tfrac12 \mathrm{tr}\big(\sigma\sigma^\top(x,u,\theta)\nabla^2 v(x)\big)\Big] \;\ge\; \alpha\big(|u-\kappa(x)|\big), $$

with equality when $u=\kappa(x)$. The bracketed term is the worst-case Itô generator applied to $v$. The second-derivative term absorbs the persistent noise variance that defeats linear storage functions. For the stochastic linear-quadratic regulator (LQR), a quadratic $v$ reduces the left side to a completed square in $u-\kappa(x)$. This is the simplest instance of the claim, and it is linear in the state distribution, unlike the nonlinear functional of Economic MPC of Markov Decision Processes.

We conjecture three consequences. The finite-horizon optimal feedback differs from $\kappa$ by more than $\epsilon$ only on a set of times whose Lebesgue measure is bounded independently of the horizon $T$. This is a policy analogue of the measure-turnpike of Integral and measure-turnpike properties for infinite-dimensional optimal control systems. The closed-loop state laws approach the set of worst-case stationary laws in Wasserstein distance. And an economic model predictive controller (a controller that re-optimizes over a sliding window and applies only the first move) without terminal ingredients has suboptimality vanishing as the window grows.

The pasting condition enters because the pointwise maximum over $\theta$ equals the dynamic worst-case evaluation only when the model set is rectangular in time. A non-rectangular set, such as "one unknown constant parameter for all time", makes the objective time-inconsistent. We predict that re-planning then keeps changing its target, and the turnpike guarantee for the receding-horizon loop fails.

Research Program

The theoretical part has three pieces. The first is the sufficiency theorem above, proved with viscosity solutions of the robust ergodic Hamilton–Jacobi–Bellman–Isaacs equation. That equation's solution is the storage function, so we do not treat it as a competitor. The proof follows the deterministic continuous-time template of Faulwasser et al.. Their exponential-reachability assumption is replaced by a uniform ergodicity (small-set) assumption on the closed loop.

The second piece is a partial converse, mirroring their Theorems 7–8: under uniform ergodicity, a policy turnpike implies feedback-strict dissipativity. We also want to determine whether pasting stability is necessary for any time-invariant storage function to certify the receding-horizon loop, or only sufficient. We regard necessity as the riskiest claim and will report it as sufficiency only if counterexamples appear.

The third piece is a discretization analysis. We will check whether the discrete-time functional dissipativity of Functional Stability of Discounted Markov Decision Processes converges to our condition as the time step shrinks. We suspect a naive discrete certificate loses strictness at rate proportional to the step. That would be a control-theoretic version of the finite-approximation trap reported in Evolutionary Games on Infinite Strategy Sets.

Computationally, we will certify the inequality with sum-of-squares programming, as Faulwasser et al. did for a degree-5 storage function on a stirred-tank reactor. The Itô generator stays polynomial for polynomial $v$, and for box-shaped $\Theta$ with affine dependence the maximum is attained at vertices. For comparison we will solve the ergodic Isaacs equation on grids in dimensions one to three. For jump noise, the running evaluation is a backward stochastic differential equation with a sublinear generator, following Dynamic Risk Measures for Processes via BSDEs Associated with Lévy Processes. That work links generator convexity to time-consistency but never adds control or stability.

Baselines

The naive reference is certainty-equivalent economic MPC: it uses the nominal model, re-plans over a sliding window, and has no terminal cost. It shows what happens when uncertainty is simply ignored.

The first strong competitor is tube-based robust economic MPC without terminal conditions (Robust Economic Model Predictive Control without Terminal Conditions). It is the existing turnpike-based method for uncertain plants, and we will run both its average-cost and worst-case-cost variants. To keep the comparison fair, it gets the same bounded disturbance support, truncated at a matched quantile of our noise, and the same window lengths. On the growth model, that paper reports only qualitative results: the average-cost variant "yields better overall economic performance on average", with no scalar number found on Lacuna. We count it as beaten if our controller matches its average cost while achieving lower realized worst-case risk under unbounded noise.

The second strong competitor is discrete-time functional dissipativity from Gros and Zanon, applied after discretizing time. It gets the same model, step-size sweeps, and tuning budget. On stochastic LQR that work reports monotone decrease of the rotated value function but no scalar result was found on Lacuna. We count it as beaten if its certificate fails, or its certified region shrinks, as the step goes to zero while ours holds.

The third is static continuous-time CVaR control (Optimal Control of Conditional Value-at-Risk in Continuous Time). It reports roughly a 50% CVaR reduction versus buy-and-hold at equal expected return, with 11% drift and 20% volatility. We will reproduce that setting. We do not expect to beat its terminal CVaR. The goal is to show that its re-planned closed loop drifts from the committed plan, while ours does not.

The key ablation uses the same controller with a non-rectangular model set, holding the size of $\Theta$ fixed. It isolates the pasting ingredient.

Why not just…?

Why not just solve the robust average-cost Isaacs equation and skip turnpike theory? Its solution is our storage function, so this is not a rival. What it lacks is any statement about finite windows. Practitioners run short-window re-planning, and our theorem is what bounds the loss from doing so. The ablation will show that, without pasting, the ergodic solution still exists but no longer describes what the receding-horizon loop does.

Why not just use tube-based robust economic MPC? It works for bounded disturbances. With Gaussian or Lévy noise, the tubes must be truncated, so tail events go uncertified, and it ignores probability mass. The Schwenkel et al. comparison already shows the worst-case tube cost is conservative. Our experiments measure both average cost and tail risk under unbounded noise to settle whether this conservatism matters in practice.

Why not just discretize time and apply the existing discrete functional dissipativity? This is a real threat. For smooth low-dimensional plants it may work fine at practical step sizes. The step-size sweep decides the question. If the certified strictness constant does not degrade with the step, we will report that the continuous-time theory mainly adds the pasting insight and the linear-in-distribution storage, not better certificates.

Why not just add a running-cost state and optimize static CVaR? Miller and Yang already show that state lifting causes the curse of dimensionality. Static CVaR is also time-inconsistent, so repeated re-planning optimizes a moving target. Our time-consistency-gap metric will measure this directly.

Proposed Experiments

We will use four systems, all taken from the file. The first is stochastic LQR with box uncertainty on the drift and volatility; the closed form checks our quadratic storage against the Gros–Zanon functional. The second is the Faulwasser stirred-tank reactor with added diffusion and uncertain reaction rate, to test whether sum-of-squares certification scales beyond degree 5. The third is the Schwenkel growth model with Gaussian instead of bounded shocks. The fourth is the Miller–Yang portfolio with rectangular drift ambiguity.

The metrics are the following. The measure of the policy-deviation time set as a function of $T$ is predicted to plateau; that is the decisive turnpike check. We also track Wasserstein distance of closed-loop laws to the stationary set, realized worst-case risk evaluated by nested Monte Carlo, and the time-consistency gap between planned and executed controls.

Three outcomes are decisive. The central one is the pasting ablation. If a non-rectangular set of matched size yields a policy-deviation measure that grows with $T$, or a non-vanishing time-consistency gap, while the rectangular set plateaus, then pasting is doing real work. If both plateau, we downgrade the claim to sufficiency. The second is the step-size sweep against discrete functional dissipativity. As a secondary deterministic sanity check, the discounted version of our certificate should recover a minimum certifiable discount factor near the 0.3109 reported for LQR in A New Dissipativity Condition for Asymptotic Stability of Discounted Economic MPC, rather than the earlier 0.846.