Neural Time-Reversed Generalized Riccati Equation

Alessandro Betti, Michele Casoni, Marco Gori, Simone Marullo, Stefano Melacci, Matteo Tiezzi

Abstract

This paper introduces a novel neural-based framework for optimal control that operates entirely forward-in-time, circumventing the need for traditional forward-backward iterations. By employing a "time-reversed" generalized Riccati equation to estimate costate variables, the system stabilizes Hamiltonian dynamics. This approach enables neural networks to learn and adapt to complex signals in real-time, providing a theoretical bridge between classical control theory and biologically inspired machine learning.

Introduction

Optimal control of dynamical systems typically relies on the Pontryagin Maximum Principle or Dynamic Programming, both of which utilize Hamiltonian equations. These equations involve "costates"—auxiliary variables representing the sensitivity of the cost function to the system state. However, solving these equations usually results in a two-point boundary value problem, requiring global-in-time computations where the system is integrated forward and then backward across the entire time horizon. This requirement is a significant bottleneck for real-time adaptation, long-term tasks, and the development of biologically plausible learning algorithms that must operate strictly forward in time.

Methodology

The proposed Neural Time-Reversed Generalized Riccati Equation addresses this limitation by estimating costates locally and forward-in-time. The architecture consists of several key components:

  1. Neural State Model: A Continuous Time Recurrent Neural Network (CTRNN) represents the underlying system dynamics.
  2. Neural Costate Estimator: A Feed-forward Neural Network (FNN) estimates the costate $\lambda(t)$ as a function of the current state $x(t)$ and a set of learnable parameters $\theta$.
  3. The Time-Reversal Trick: The core innovation involves reversing the sign of the temporal derivative of the costate network's parameters. In classical control, the Riccati equation is solved backward to ensure stability; the authors conjecture that flipping the sign of the parameter dynamics induces a similar stabilizing effect while moving forward in time.
  4. Consistency Loss: A local loss function $\Omega_t$ is defined at each time step to enforce consistency between the neural costate estimate and the theoretical requirements prescribed by the Hamilton equations. Parameters $\theta$ are updated via gradient descent on this local loss.

Experimental Evaluation

The method was evaluated using three synthetic control case studies designed to test tracking, non-linear adaptation, and classification:

  • Sinusoidal Signal Tracking: A fundamental test of the network's ability to follow a simple periodic target.
  • Sign Prediction: A non-linear task requiring the model to predict the sign of an input sinusoidal signal.
  • Wave-shape Classification: A task where the model must distinguish between sine and square wave inputs in real-time.

Key Results

The system demonstrated rapid adaptation and high accuracy across all test cases. In the sinusoidal tracking task (Case a), the model successfully converged to the target signal from random initial weights, maintaining high fidelity throughout the trajectory.

Sinusoidal signal tracking (Case a)

In the sign prediction task (Case b), the network effectively integrated current inputs with previous states to track the non-linear target. The model correctly interleaved information to produce a square-wave output from a sinusoidal input, demonstrating the ability to handle non-linearities.

Sign prediction of a sinusoidal input (Case b)

Furthermore, in the wave-shape classification task (Case c), the model successfully adapted its parameters in real-time to distinguish between different input signals. The authors also noted instances of "generalization," where the network continued to produce correct outputs even after the target signal was masked, suggesting it had learned the underlying dynamics rather than simple mimicry.

Discussion and Conclusion

The results suggest that the inherent instability of forward-running Hamiltonian equations can be mitigated by reversing the parameter dynamics of the costate estimator. This local-in-time learning approach allows for continuous adaptation without the need to store or re-process past trajectories, offering a potential mechanism for biologically plausible backpropagation.

However, several limitations remain. The model's performance is highly sensitive to hyperparameters, and explicit generalization (performing the task in the complete absence of a target) is not yet fully robust. Future research will focus on scaling this approach to high-dimensional state spaces and exploring its application in lifelong learning scenarios where models must continuously adapt to new tasks without catastrophic forgetting.

Summary updated

Abstract

Optimal control deals with optimization problems in which variables steer a dynamical system, and its outcome contributes to the objective function. Two classical approaches to solving these problems are Dynamic Programming and the Pontryagin Maximum Principle. In both approaches, Hamiltonian equations offer an interpretation of optimality through auxiliary variables known as costates. However, Hamiltonian equations are rarely used due to their reliance on forward-backward algorithms across the entire temporal domain. This paper introduces a novel neural-based approach to optimal control. Neural networks are employed not only for implementing state dynamics but also for estimating costate variables. The parameters of the latter network are determined at each time step using a newly introduced local policy referred to as the time-reversed generalized Riccati equation. This policy is inspired by a result discussed in the Linear Quadratic (LQ) problem, which we conjecture stabilizes state dynamics. We support this conjecture by discussing experimental results from a range of optimal control case studies.
BibTeX
@inproceedings{betti2024neural,
  author = {Alessandro Betti and Michele Casoni and Marco Gori and Simone Marullo and Stefano Melacci and Matteo Tiezzi},
  title = {{Neural Time-Reversed Generalized Riccati Equation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i8.28630},
}
Figure