Abstract
In this letter, we study dynamic game optimal control with imperfect state observations and introduce an iterative method to find a local Nash equilibrium. The algorithm consists of an iterative procedure combining a backward recursion similar to minimax differential dynamic programming and a forward recursion resembling a risk-sensitive Kalman smoother. A coupling equation renders the resulting control dependent on the estimation. In the end, the algorithm is equivalent to a Newton step but has linear complexity in the time horizon length. Furthermore, a merit function and a line search procedure are introduced to guarantee convergence of the iterative scheme. The resulting controller reasons about uncertainty by planning for the worst case disturbances. Lastly, the low computational cost of the proposed algorithm makes it a promising method to do output-feedback model predictive control on complex systems at high frequency. Numerical simulations on realistic robotic problems illustrate the risk-sensitive behavior of the resulting controller.
Introduction
Traditional robotic control often relies on the "certainty equivalence" principle, which separates state estimation from control action. While effective in predictable environments, this approach can fail when faced with significant sensor noise or unpredictable external disturbances. Dynamic game control offers a more robust framework by modeling disturbances as an adversarial player. This leads to a minimax optimization problem where the controller seeks to minimize costs while accounting for a "worst-case" adversary.
Despite its theoretical advantages, solving minimax problems for nonlinear systems with imperfect state observations is computationally intensive. Existing solvers often struggle with real-time constraints or lack robust convergence properties. This paper introduces the Stagewise Newton Method, an iterative algorithm designed to bridge this gap by providing a fast, convergent solution for dynamic game control.
Methodology
The proposed method leverages the temporal sparsity of the problem's Hessian matrix to achieve a computational complexity that scales linearly with the time horizon ($O(T)$). This efficiency allows for high-frequency control even in complex systems.
The Algorithm
The Stagewise Newton Method iterates through three key phases: an estimation forward pass, a control backward pass, and a unique coupling step. As shown in Figure 1, the coupling equation explicitly links the two passes, ensuring the control command explicitly incorporates the uncertainty of the state estimate.
To ensure global convergence to a local Nash equilibrium, the authors introduce a merit function and a line search procedure, preventing the solver from diverging in highly nonlinear regions.
Experimental Results
The algorithm's performance was evaluated using two simulated robotic benchmarks: a 6D Planar Quadrotor and a 14D KUKA LWR iiwa industrial manipulator.
Risk-Sensitive Navigation
In the quadrotor task, the drone must navigate to a target while avoiding an obstacle represented by a high-cost exponential field. As illustrated in Figure 2, the risk-sensitive controller ($\mu=6$) adopts a safer trajectory compared to the neutral controller. By planning for worst-case disturbances, the algorithm maintains a larger safety margin, effectively "pushing" the trajectory further from the obstacle to account for potential estimation errors and disturbances.
Scalability to High-Dimensional Systems
The KUKA robot experiment demonstrated the algorithm's scalability. Handling a 14D state space, the controller exhibited "cautious" behavior, slowing down reaching tasks when uncertainty was high. This confirms that the linear complexity of the Stagewise Newton Method makes it suitable for modern, high-DOF industrial robots.
Conclusion
The Stagewise Newton Method provides a computationally efficient and theoretically grounded approach to dynamic game control. By unifying risk-sensitive estimation and minimax control, it enables robots to reason about uncertainty and plan for robustness in real-time. Future work will explore the implementation of this method in high-frequency Model Predictive Control (MPC) loops on physical hardware and the integration of full second-order dynamics for even faster convergence.
Summary updated