In classical optimal control theory, the gold standard for finding an optimal strategy is the Hamilton-Jacobi-Bellman (HJB) equation. This partial differential equation (PDE) relies fundamentally on the Markov property: the assumption that the future behavior of a system depends solely on its current state.
However, real-world systems rarely enjoy such perfect amnesia. From network communication lags in cyber-physical systems to driver reaction delays in traffic flow, future states often depend explicitly on the system's history. As noted by Shankar A. Deka et al., these time-delay terms enter governing equations implicitly through the system's history, breaking standard state-space techniques and rendering classical HJB equations insufficient.
To solve control problems with memory, researchers have developed a generalized framework that extends HJB equations to infinite-dimensional history spaces. By leveraging coinvariant derivatives and viscosity solutions, this approach guarantees that path-dependent control problems possess unique, mathematically tractable solutions.
The Challenge of Infinite-Dimensional Memory
Why does memory break classical calculus? In a standard system, the state at time $t$ is a single point $x(t) \in \mathbb{R}^n$. In a time-delay or path-dependent system, the state is no longer a point; it is an entire trajectory segment—a history window $x_t(\theta) = x(t + \theta)$ for $\theta \in [-r, 0]$.
Because the state space is now a function space (an infinite-dimensional Hilbert or Banach space), taking standard partial derivatives of the value functional becomes problematic. A classical derivative requires the functional to be smooth under arbitrary perturbations in any direction of this function space. In control applications, where trajectories are driven by non-smooth or adversarial inputs, expecting such widespread smoothness (like Fréchet differentiability) is unrealistic.
The Mechanism: Sliding the History Window
To bypass the need for full Fréchet differentiability, researchers utilize coinvariant derivatives (also known as path-dependent or structural derivatives).
The core intuition behind a coinvariant derivative is that it restricts how we perturb the history. Instead of changing the historical path arbitrarily, it tracks how a functional changes as the time-indexed history window naturally slides forward in time. As Anton Plaksin demonstrates, these derivatives are calculated along the "right extensions" of a system's history. This matches the physical reality of time: we cannot rewrite the past; we can only append the unfolding present to the front of our history window.
By focusing exclusively on this time-sliding operation, coinvariant derivatives act analogously to partial derivatives in standard systems, as shown by Anton Plaksin. This mechanism allows researchers to formulate path-dependent HJB equations that account for both the current state and the accumulated history without requiring overly restrictive smoothness. For instance, in neutral-type systems where dynamics depend on past rates of change, Anton Plaksin shows that the coinvariant framework successfully incorporates an additional term representing the interaction between the neutral component's derivative and the value functional's gradient.
Guaranteeing Uniqueness: Viscosity and Minimax Equivalence
Value functions in optimal control are rarely smooth enough to satisfy PDEs classically; they often develop kinks or non-differentiable boundaries. To handle this, the field relies on viscosity solutions—a generalized solution concept that uses smooth "test functions" to bound the value functional from above and below.
For a path-dependent HJB equation to be practically useful, it must satisfy a comparison principle. This mathematical guarantee ensures that the value functional derived from the control problem is the unique viscosity solution to the corresponding PDE. Without uniqueness, numerical algorithms might converge to solutions that do not represent the true optimal control strategy.
Jianjun Zhou established this rigorous theoretical basis by proving the existence and uniqueness of viscosity solutions for first-order path-dependent HJB equations in infinite-dimensional Hilbert spaces. Similarly, Mingshang Hu et al. demonstrated that under specific conditions, the value function can be robustly characterized as a unique viscosity solution.
A major theoretical unification occurred when Anton Plaksin et al. proved that minimax solutions (which rely on geometric invariance properties along trajectories) and viscosity solutions (which rely on test-function gradients) are mathematically equivalent for path-dependent Hamilton-Jacobi equations. This equivalence allows researchers to seamlessly switch between geometric and analytical tools depending on the requirements of the problem, yielding stronger comparison results and cementing the uniqueness of solutions for systems with memory (Anton Plaksin).
Broad Applications across Control Theory
This unified framework of coinvariant differentiation and viscosity uniqueness has unlocked solutions to several complex control paradigms:
- Differential Games: In zero-sum games where an adversary exploits communication lags or distribution shifts, the value functions are proven to be unique viscosity solutions to infinite-dimensional Hamilton-Jacobi-Isaacs (HJI) equations, whether operating in history spaces (Anton Plaksin) or over probability distributions in Wasserstein space (Jun Moon).
- Stochastic Systems and Jumps: When systems face random shocks and information delays, frameworks like functional Itô calculus are used to derive nonlinear master equations (Yuri Saporito et al.). Moreover, Qingmeng Wei bridged stochastic calculus and PDEs by showing that value functions derived from fully coupled forward-backward stochastic differential equations (FBSDEs) with jumps are deterministic viscosity solutions to their associated HJI equations.
- State Constraints and Non-Smooth Objectives: In real-world scenarios where systems must stay within safe boundaries or optimize non-smooth financial utilities, Bruno Bouchard et al. and Jiangyan Pu demonstrated that probabilistic relaxation and regularized viscosity solutions maintain uniqueness even at the boundaries of constraint sets.
By shifting the perspective from static states to sliding history windows, this research direction provides a mathematically rigorous foundation for controlling complex, real-world systems that are bound by their past.
Key Foundational Work
Weak Dynamic Programming for Generalized State Constraints (2011) — contributed 1 concept elements to this direction.
Dynamic Non-Cooperative Games as a Foundation for Modeling Dynamic User Equilibrium (2006) — contributed 1 concept elements to this direction.