LQGNET: Hybrid Model-Based and Data-Driven Linear Quadratic Stochastic Control

Solomon Goldgraber Casspi, Oliver Hüsser, Guy Revach, Nir Shlezinger

TL;DR

This paper introduces LQGNet, a novel hybrid control system that combines the strengths of traditional model-based control with the adaptability of deep learning. LQGNet effectively learns to control systems with partially known dynamics, outperforming classical methods when models are mismatched and approaching optimal performance even when the system is fully understood.

What Problem Does This Paper Address?

Stochastic control is crucial for systems operating under uncertainty, with the Linear Quadratic Gaussian (LQG) framework being a cornerstone for linear systems with Gaussian noise. The standard LQG controller, derived from the separation principle, relies on a perfectly known system model. However, in real-world applications, system dynamics are often only partially understood or are approximations, leading to significant performance degradation for traditional LQG controllers. While reinforcement learning offers a data-driven alternative, it often lacks the interpretability and ability to incorporate existing domain knowledge. This paper addresses the challenge of designing controllers that can leverage existing (even if imperfect) model knowledge while also learning from data to overcome uncertainties and model mismatches, bridging the gap between model-based and purely data-driven approaches.

Methods

LQGNet is a hybrid model-based and data-driven controller that builds upon the optimal LQG policy. The core innovation lies in augmenting the state estimation module of the traditional LQG controller with a deep learning component. Specifically, the Kalman Filter (KF), responsible for state estimation, is replaced by a trainable recurrent neural network (RNN) inspired by the KalmanNet architecture. This trainable KF computes the Kalman Gain (KG) without requiring knowledge of noise statistics, a key limitation of traditional KFs. The rest of the controller, the Linear Quadratic Regulator (LQR), remains model-based, utilizing the available (potentially mismatched) system matrices.

The training process for LQGNet is end-to-end, optimizing the overall quadratic control loss. This is achieved by formulating a regularized LQG loss function that includes a term for the control objective and a regularization term for the RNN weights. Gradient-based learning is then used to train the network, allowing LQGNet to learn to cope with both state estimation inaccuracies and LQR mismatches simultaneously.

Datasets

The paper does not explicitly mention specific datasets for training. Instead, it utilizes a simulator (denoted by Γ) that emulates the underlying system dynamics. This simulator is used to generate random trajectories of inputs, observations, and state values, which are then used for training and evaluating LQGNet. The evaluation involves comparing LQGNet's performance against a model-based KF-LQR controller under various noise levels and model mismatch scenarios.

Key Insights and Ideas

  • Bridging the Gap: LQGNet successfully merges the interpretability and structure of model-based control with the adaptability of data-driven methods, offering a powerful hybrid approach.
  • Learning the Kalman Gain: By using a trainable RNN to compute the Kalman Gain, LQGNet bypasses the need to know noise statistics, a significant practical advantage.
  • End-to-End Learning for Robustness: Training the entire system end-to-end, optimizing for the final control objective, allows LQGNet to learn state estimates that, while potentially not optimal for estimation alone, lead to better overall control performance in the presence of model mismatches.
  • Preserving Simplicity: Despite incorporating deep learning, LQGNet retains the interpretable structure of the LQG controller, making it easier to understand and debug compared to black-box RL approaches.

Key Results

The empirical evaluations demonstrate that:

  • Without Mismatches: LQGNet achieves performance very close to the optimal LQG controller, validating its ability to learn the underlying dynamics when they are well-represented.
  • With Mismatches: LQGNet significantly outperforms the classical KF-LQR controller when the system dynamics (either state evolution or observation mapping) are mismatched.
  • Improved Control Despite Inaccurate State Estimation: Even when LQGNet's state estimation (MSE) is not as accurate as the model-based KF, its learned control inputs lead to more reliable and better overall system performance. This is highlighted in Figure 2c, where a trajectory controlled by LQGNet closely resembles that of the optimal controller despite model mismatches.

Limitations and Future Work

  • Limitations: The paper focuses on linear Gaussian state-space models. While the approach is promising, its direct applicability to non-linear dynamics or non-Gaussian noise is not explored. The performance gains are demonstrated through simulations, and real-world deployment challenges are not discussed.
  • Future Work: The authors suggest several avenues for future research:
  • Extending LQGNet to handle non-linear dynamics.
  • Adapting the framework for alternative control objectives, such as model predictive control.
  • Investigating the use of emerging convex optimization control policies.

Summary updated

Abstract

Stochastic control deals with finding an optimal control signal for a dynamical system in a setting with uncertainty, playing a key role in numerous applications. The linear quadratic Gaussian (LQG) is a widely-used setting, where the system dynamics is represented as a linear Gaussian statespace (SS) model, and the objective function is quadratic. For this setting, the optimal controller is obtained in closed form by the separation principle. However, in practice, the underlying system dynamics often cannot be faithfully captured by a fully known linear Gaussian SS model, limiting its performance. Here, we present LQGNet, a stochastic controller that leverages data to operate under partially known dynamics. LQGNet augments the state tracking module of separation-based control with a dedicated trainable algorithm. The resulting system preserves the operation of classic LQG control while learning to cope with partially known SS models without having to fully identify the dynamics. We empirically show that LQGNet outperforms classic stochastic control by overcoming mismatched SS models.
BibTeX
@inproceedings{casspi2023lqgnet,
  author = {Solomon Goldgraber Casspi and Oliver Hüsser and Guy Revach and Nir Shlezinger},
  title = {{LQGNET: Hybrid Model-Based and Data-Driven Linear Quadratic Stochastic Control}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095181},
}
Figure