In classical control theory, we often assume that the world is smooth. We imagine that the "value function"—the mathematical map telling us the minimum cost to reach a goal from any given point—is a landscape with gentle slopes where derivatives are always well-defined.
But real-world systems are rarely so polite. When a robot must avoid a wall, a power grid hits a voltage limit, or a trader faces a sudden market jump, the value function develops "kinks" or discontinuities. At these points, traditional calculus fails because the gradient simply does not exist. This research direction focuses on viscosity solutions, a generalized mathematical framework that allows us to solve the Hamilton-Jacobi-Bellman (HJB) equations even when the underlying landscape is jagged and non-differentiable.
The Geometry of the Kink
The core of optimal control is the Bellman principle: the optimal action today is the one that minimizes the current cost plus the expected future cost. Mathematically, this involves a min or max operator over all possible controls.
As Lijun Zhang and Markus N. Rabe point out in Efficient Approximation of Optimal Control for Markov Games, these operators are the primary source of non-smoothness. When two different strategies become equally optimal at a certain state, the value function forms a "kink"—a point where the slope changes abruptly. Standard numerical solvers for differential equations assume smoothness and essentially "trip" over these kinks, leading to instability or incorrect solutions.
Why Viscosity Solutions Work
The intuition behind a viscosity solution is to stop looking for a function that satisfies the PDE at every single point. Instead, we look for a function that is "consistent" with the PDE when tested against smooth functions.
Imagine trying to measure the slope of a jagged mountain peak. You can't measure the slope at the tip, but you can slide a smooth, curved bowl over the peak. If the bowl touches the peak from above and its slope is consistent with the physics of the problem, you’ve found a way to bound the behavior of the non-smooth surface. Qingmeng Wei demonstrates in Optimal control problems of fully coupled FBSDEs and viscosity solutions of Hamilton-Jacobi-Bellman equations that this framework is the only way to rigorously define the value function for complex stochastic systems where noise depends on the system's own sensitivity.
Smoothing the Landscape
One way to bypass the "kink" problem is to force the system to be smooth. Insoon Yang and Jeongho Kim 0002 explore this in Hamilton-Jacobi-Bellman Equations for Maximum Entropy Optimal Control. By adding an entropy penalty—essentially rewarding the controller for being "uncertain" or exploring—they derive a "Soft" HJB equation. This mathematical trick effectively "blurs" the sharp edges of the value function, making it easier to solve while providing more robust control.
Similarly, Tobia Marcucci and colleagues in Smooth Model Predictive Control with Applications to Statistical Learning use log-barrier functions to "soften" hard constraints. This ensures that the resulting control policy is Lipschitz continuous, which is a fancy way of saying the robot won't suddenly jerk its motors when it gets close to an obstacle.
Scaling to High Dimensions
While viscosity theory provides the mathematical "permission" to handle non-smoothness, actually calculating these solutions in high-dimensional space is notoriously difficult—the "curse of dimensionality."
Recent work has turned to sophisticated approximation architectures:
- Neural Networks: Xingjian Li and Deepanshu Verma use the Stochastic Pontryagin Maximum Principle to guide neural networks, focusing their learning capacity only on the trajectories that actually matter, rather than trying to learn the entire non-smooth state space (A Neural Network Approach for Stochastic Optimal Control).
- Tensor Trains: Konstantin Fackeldey and Reinhold Schneider use a "Tensor Train" format to approximate the value function. This method breaks a high-dimensional problem into a chain of low-dimensional interactions, allowing them to solve 6D problems that would normally require billions of grid points (Approximative Policy Iteration for Exit Time Feedback Control Problems).
- Random Trees: Kelsey P. Hawkins and Ali Pakniyat merge robotic path planning with control theory in Forward-Backward Rapidly-Exploring Random Trees for Stochastic Optimal Control. Their FBRRT algorithm uses space-filling trees to explore the state space, naturally handling non-quadratic and non-smooth costs that cause traditional methods to fail.
Real-World Impact
This isn't just abstract math; it's a requirement for safety-critical systems. In Learning to Pursue AC Optimal Power Flow Solutions with Feasibility Guarantees, Damola Ajeyemi and Jorge Cortés 0001 show that by emulating a "Safe Gradient Flow," they can keep power grids within strict physical limits while achieving a 297x speedup over traditional solvers.
By embracing the non-smooth nature of the world through viscosity solutions and generalized gradients, researchers are building controllers that don't just work in idealized simulations, but remain stable and optimal in the face of the sharp transitions and unpredictable jumps of reality.