Non-Smooth Value Functions in Continuous Optimal Control

41 concepts36 papers1 proposal

Imagine driving a car toward a fork in the road. To the left is a smooth, clear highway; to the right is a bumpy shortcut. Depending on your exact starting position, the optimal choice abruptly flips from left to right. If you plot your minimum travel time as a function of your starting coordinates, this sudden switch creates a sharp ridge—a "kink"—where the slope changes instantaneously.

Crucially, this happens even if the physics of the car and the geometry of the roads are perfectly smooth. As Remi Munos demonstrated in A Study of Reinforcement Learning in the Continuous Case by the Means of Viscosity Solutions, optimal value functions in continuous spaces naturally lack differentiability. This inherent non-smoothness is not a failure of modeling; it is a fundamental geometric property of optimal decision-making.

Why Classical Calculus Breaks Down

In continuous-time optimal control and reinforcement learning, the standard mathematical tool used to find an optimal policy is the Hamilton-Jacobi-Bellman (HJB) equation. The HJB is a partial differential equation that relies directly on the gradient of the value function ($\nabla V(x)$) to determine the best action at any given state $x$.

When a value function contains kinks, its gradient becomes undefined at those exact points. Classical calculus requires functions to be continuously differentiable to solve differential equations. Without differentiability, the standard analytical framework collapses. As Lasserre et al. point out in Inverse optimal control with polynomial optimization, assuming continuous smoothness in systems that naturally produce non-smooth value functions—such as minimum-time objectives—leads to severe approximation errors where the control logic changes abruptly.

Mathematical Frameworks for Non-Smoothness

To bypass the limitations of classical calculus, researchers utilize generalized solution concepts that can accommodate sharp corners and discontinuities.

The most prominent framework is the viscosity solution, which allows differential equations to accept non-differentiable functions by evaluating them using smooth "test functions" that touch the value function from above or below. Yang et al. applied this to reinforcement learning in Hamilton-Jacobi-Bellman Equations for Q-Learning in Continuous Time, proving that continuous-time Q-functions can be rigorously modeled as viscosity solutions despite their kinks.

Similarly, Plaksin et al. demonstrated in Equivalence of minimax and viscosity solutions of path-dependent Hamilton–Jacobi equations that viscosity solutions provide a stable foundation for infinite-dimensional systems with memory delay. In competitive environments, Bouchard et al. leverage regularized viscosity solutions in Stochastic Target Games and Dynamic Programming via Regularized Viscosity Solutions to characterize non-differentiable value functions within non-linear partial differential equations.

When systems involve moving boundaries or discontinuous costs, alternative frameworks are required:

What Causes Value Functions to Lose Smoothness?

Non-smoothness arises from several distinct physical and mathematical catalysts across control problems:

1. Hard Constraints and Safety Boundaries

When an agent must avoid an obstacle or stay within a safe region, the value function changes dramatically at the boundary. Buckdahn et al. showed in Regularity Properties for General HJB Equations: A Backward Stochastic Differential Equation Method that value functions lose Lipschitz continuity as they approach terminal times or state barriers. In competitive or multi-agent settings, Ghimire et al. note in State-Constrained Zero-Sum Differential Games with One-Sided Information that strict safety boundaries introduce sharp discontinuities that break standard numerical solvers.

2. Discontinuous Dynamics and Operations

Systems that switch modes (such as a vehicle shifting gears or a thermostat clicking on and off) inherently introduce non-smoothness. Tan et al. explain in Verifying Switched System Stability With Logic that operations like $\max$ or $\min$ complicate the derivation of smooth Lyapunov functions needed for formal stability proofs. In differential games with time delays, Anton Plaksin notes in On Hamilton–Jacobi–Isaacs–Bellman equation for neutral type systems that value functions routinely lack classical derivatives due to the complex interaction of delayed states.

3. Stochastic Noise and Non-Quadratic Costs

Randomness does not always smooth out a control landscape. Ito et al. show in Sparse optimal stochastic control that random noise can render the value function non-differentiable, preventing first-order analysis. Furthermore, when actions directly influence the system's noise coefficient, Siska et al. note in Entropy annealing for policy mirror descent in continuous time and space that proving differentiability becomes exceptionally complex due to a lack of operator regularity.

These dynamics are further complicated by the choice of objective function; Hawkins et al. observe in Forward-Backward RRT: Branched Sampled FBSDEs for Stochastic Optimal Control that real-world applications like minimum-fuel problems introduce non-smooth costs that defeat traditional quadratic optimization techniques. Similarly, Jiangyan Pu notes in Dynamic programming principle and associated Hamilton-Jacobi-Bellman equation for stochastic recursive control problem with non-Lipschitz aggregator that connecting value functions to optimality equations becomes difficult when cost aggregators depend on risk variables in a non-smooth manner.

The Impact on Function Approximation

Modern reinforcement learning and control rely heavily on neural networks or polynomials to approximate value functions. However, non-smooth landscapes present severe practical challenges for these smooth approximators.

The Failure of Global Smooth Approximators

In Nonlinear optimal control synthesis via occupation measures, Lasserre et al. demonstrate that global polynomial approximations struggle to capture the sharp transitions and kinks of optimal value functions, leading to localized degradation. Because polynomials are inherently smooth, they cannot perfectly replicate a sharp corner. To find stable subsolutions to the HJB equation without introducing artificial noise, Berthier et al. propose infinite-dimensional Sum-of-Squares frameworks in Infinite-Dimensional Sums-of-Squares for Optimal Control.

Neural Network Optimization Failures

When deep neural networks are used to represent value functions, their jagged or non-convex landscapes can cause gradient-based controllers to fail. Karnchanachari et al. observe in Practical Reinforcement Learning For MPC: Learning from sparse objectives in under an hour on a real robot that integrating neural network value functions into Model Predictive Control (MPC) loops requires explicit regularization to prevent optimization solvers from getting stuck in local, non-smooth artifacts.

To address this in safe continuous-time multi-agent reinforcement learning, Wang et al. introduce an epigraph-based reformulation in Safe Continuous-time Multi-Agent Reinforcement Learning via Epigraph Form to handle the sharp gradients introduced by hard safety constraints without causing optimization jitter.

Theoretical Guarantees

Despite these practical hurdles, recent theoretical work confirms that deep networks possess the mathematical capacity to navigate these landscapes. In Universal Approximation Theorem of Deep Q-Networks, Qian Qi proved that ResNet-style architectures can universally approximate non-smooth optimal Q-functions, providing a vital theoretical bridge for deploying deep reinforcement learning safely in continuous physical systems.

Key Foundational Work

A Study of Reinforcement Learning in the Continuous Case by the Means of Viscosity Solutions (2000) — contributed 1 concept elements to this direction.
On-Line Estimation of the Optimal Value Function: HJB- Estimators (1992) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_value_function_non_smoothness_d722a399,
  title = {value function non-smoothness},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/value-function-non-smoothness/rdr_d722a39950c14f0996fc667a9add814c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_value_function_non_smoothness_d722a399_v1,
  title = {value function non-smoothness},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/value-function-non-smoothness/rdr_d722a39950c14f0996fc667a9add814c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake