Abstract
In stochastic control theory, practitioners typically design an optimal control policy based on a nominal model of the system. However, when this policy is applied to the actual (true) system, a performance loss occurs due to the mismatch between the assumed and true models. This paper investigates the robustness of stochastic optimal control for non-degenerate controlled diffusion processes. We establish that as the approximate model converges to the true model, both the optimal value and the performance of the approximate control policy converge to the true optimum. This robustness is demonstrated across four major cost criteria: finite horizon, infinite-horizon $\alpha$-discounted, ergodic (average cost), and optimal control up to an exit time. By leveraging the regularity properties of Hamilton-Jacobi-Bellman (HJB) equations and the theory of uniformly elliptic PDEs, we provide a unified framework for robustness in continuous-time systems.
Introduction
In real-world engineering and financial applications, the mathematical models used to describe system dynamics are rarely perfect. Whether derived from physical principles or estimated from noisy data, these "nominal" models are approximations of the underlying "true" system. A fundamental question in control theory is whether a controller designed for an approximate model will remain effective when deployed on the actual system.
While this problem of model mismatch and performance loss has been studied extensively in discrete-time Markov Decision Processes (MDPs), the continuous-time case—where systems are governed by Stochastic Differential Equations (SDEs) or controlled diffusions—presents unique challenges and opportunities. This research provides a rigorous mathematical proof that stochastic optimal control is robust to model approximations in these continuous-time settings.
The Robustness Problem
The core of the study revolves around the "Robustness Index," which quantifies the performance loss incurred by applying a policy $v^*_n$ (optimized for an approximate model $n$) to the true system. The goal is to prove that as the approximate model $n$ approaches the true model, this index vanishes.
Unlike discrete-time studies that often rely on value iteration or transition kernel arguments, this research utilizes a PDE-centric approach. By focusing on the Hamilton-Jacobi-Bellman (HJB) equations, the authors exploit the strong regularity properties of solutions to uniformly elliptic PDEs. A surprising insight from this work is that the continuous-time framework is often "nicer" than the discrete-time one; the underlying PDEs provide a level of smoothness and stability that discrete-time transition kernels frequently lack.
Methodology and Convergence Criteria
The authors define a "continuous convergence" requirement where the drift ($b$) and cost ($c$) functions of the approximate models must converge to the true ones uniformly over the set of possible control actions. To establish the convergence of value functions, the study employs $W^{2,p}$ (Sobolev) space estimates, which allow for the analysis of the compactness and stability of the solutions to the HJB equations.
Stability in Ergodic Control
The ergodic (long-run expected average cost) case is particularly complex because HJB equations for average costs can admit uncountably many solutions. To ensure robustness in this regime, the authors analyze two specific stability conditions:
- Near-monotonicity: A condition that penalizes the system state for wandering toward infinity, effectively keeping the process within a manageable region.
- Lyapunov Stability: A condition ensuring the system naturally tends to return to a central state, which guarantees the "tightness" of invariant measures.
Key Results across Cost Criteria
The paper provides a unified treatment for four distinct cost evaluation criteria:
- Finite Horizon: Control over a fixed, specific time window.
- $\alpha$-discounted: Infinite horizon where future costs are discounted by a factor $\alpha$.
- Ergodic: The long-term average cost per unit of time.
- Exit Time: Control until the system state leaves a predefined boundary (e.g., a safety or operational region).
For each of these criteria, the research establishes two primary types of convergence:
- Value Convergence: The optimal value calculated using the approximate model ($V_n$) converges to the true optimal value ($V$).
- Policy Robustness: The actual cost resulting from applying the approximate optimal policy ($v^*_n$) to the true system converges to the true minimum cost as the model approximation improves ($n \to \infty$).
Discussion and Future Work
While this work provides a robust foundation for continuous-time control, it focuses on "non-degenerate" diffusions, where noise affects all dimensions of the state space. This assumption ensures the elliptic nature of the governing PDEs. Future research is needed to address "degenerate" systems, where noise may be restricted to specific dimensions, potentially requiring different analytical tools.
Additionally, the study assumes system noise follows a standard Wiener process (Gaussian noise). Extending these robustness results to systems with wide-bandwidth noise or jump processes—which can introduce non-Markovian behavior—remains an important open challenge for the field. Nevertheless, by utilizing the Vasicek interest rate model as a motivating example, the authors demonstrate that these theoretical findings have significant implications for practical model-based control and reinforcement learning in continuous time.
Summary updated