Data-Driven Mean Field Equilibrium Computation in Large-Population LQG Games

Zhenhui Xu, Jiayu Chen, Bing-Chang Wang, Tielong Shen

TL;DR

This paper introduces a novel data-driven method to compute approximate Nash equilibria in large-scale, continuous-time linear quadratic Gaussian (LQG) games. By leveraging integral reinforcement learning and system identification techniques on collected agent data, the approach bypasses the need for explicit knowledge of agent dynamics, offering a practical solution for complex multi-agent systems.

What Problem Does This Paper Address?

Mean Field Games (MFGs) are a powerful framework for analyzing systems with a large number of interacting agents. Linear Quadratic Gaussian (LQG) MFGs, in particular, have attracted significant attention due to their tractability, often allowing for closed-form solutions. However, a major hurdle in applying these theoretical solutions to real-world scenarios is the requirement for complete knowledge of the agents' dynamics and cost functions. This paper tackles the challenge of computing an $\varepsilon$-Nash equilibrium in continuous-time LQG games without this a priori knowledge. The core difficulty lies in solving coupled algebraic Riccati equations (AREs) and ordinary differential equations (ODEs) that define the equilibrium, especially when the system dynamics are unknown and the AREs become non-symmetric due to inter-agent coupling.

Methods

The proposed solution is a two-pronged, data-driven approach:

  1. Integral Reinforcement Learning (IRL) for Feedback Control: The authors adapt an IRL technique to solve one of the key algebraic Riccati equations (AREs) that determines the feedback gain matrix. This is achieved by transforming the system to overcome challenges posed by unknown dynamics and then using collected state and input samples to iteratively learn the optimal feedback gain.
  2. System Identification for Feedforward Control and Dynamics: Once the feedback gain is approximated, the paper employs system identification methods. This involves using the same collected data to estimate the drift coefficients of the agents' dynamics. This estimation then allows for the computation of the feedforward gain matrix and the mean field state trajectory by solving the remaining ARE and ODE.

A special case is also addressed where specific conditions on the system matrices simplify the nonsymmetric ARE into a symmetric indefinite ARE, allowing for a data-driven iterative solution similar to the feedback control part.

Datasets

The paper does not introduce new benchmark datasets. Instead, it relies on simulated data generated from the defined LQG game models for evaluation. The core of the methodology is its ability to work with any collected state and input samples from the agents, provided certain persistence of excitation conditions are met.

Key Insights and Ideas

  • Data-Driven Equilibrium without Models: The most significant insight is the demonstration that $\varepsilon$-Nash equilibria in complex LQG games can be approximated using only observed agent behavior (state and input trajectories), eliminating the need for explicit system models.
  • Bridging IRL and System Identification: The paper elegantly combines IRL for learning control policies with system identification for estimating unknown dynamics, creating a comprehensive data-driven framework.
  • Handling Non-Symmetric AREs: A key technical contribution is the development of methods to tackle the non-symmetric ARE that arises in MFG problems, which is often a bottleneck for model-free approaches.
  • Error and Average Trajectories: The method cleverly utilizes both "error" trajectories (differences between specific agents) and "average" trajectories to satisfy necessary conditions for convergence and parameter estimation.

Key Results

The paper presents numerical examples demonstrating the effectiveness of the proposed algorithms.

  • Algorithm 1 (General Case): Successfully approximates the feedback gain matrix ($P_{11}, K_1$) and identifies system parameters ($A, B, G$) with reasonable accuracy. For instance, in Numerical Example I, approximation errors for key matrices and coefficients were in the range of 0.02 to 0.09. The mean field state trajectory was also closely approximated.
  • Algorithm 2 (Special Case): Shows similar convergence and accuracy for the simplified scenario, with approximation errors for $P_{12}$ and $K_2$ being around 0.02 to 0.1.
  • Convergence: The iterative algorithms demonstrate quadratic convergence, reaching the specified convergence criterion within a few iterations (e.g., 7 iterations for Algorithm 1 in Example I).

Limitations and Future Work

  • Persistence of Excitation: The data-driven methods rely on certain "persistence of excitation" conditions (Assumptions 3.1 and 3.2) being met by the collected data. This typically requires injecting exploration noise into the agents' control inputs, which is discussed as a practical implementation detail.
  • Computational Complexity: While data-driven, the method still involves solving AREs and ODEs, albeit numerically. The paper mentions that the time for computation is negligible compared to data collection.
  • Future Work: The authors express interest in extending the IRL technique beyond the LQG framework to cover more general cases, particularly those involving the solution of forward-backward partial differential equations. They also aim to explore scenarios where approximate Nash equilibria arise from these more complex structures.

Summary updated

Abstract

This paper presents a novel data-driven approach for approximating the $\varepsilon$-Nash equilibrium in continuous-time linear quadratic Gaussian (LQG) games, where multiple agents interact with each other through their dynamics and infinite horizon discounted costs. The core of our method involves solving two algebraic Riccati equations (AREs) and an ordinary differential equation (ODE) using state and input samples collected from agents, eliminating the need for a priori knowledge of their dynamical models. The standard ARE is addressed through an integral reinforcement learning (IRL) technique, while the nonsymmetric ARE and the ODE are resolved by identifying the drift coefficients of the agents' dynamics under general conditions. Moreover, by imposing specific conditions on models, we extend the IRL-based approach to approximately solve the nonsymmetric ARE. Numerical examples are given to demonstrate the effectiveness of the proposed algorithms.
BibTeX
@misc{xu2025data,
  author = {Zhenhui Xu and Jiayu Chen and Bing-Chang Wang and Tielong Shen},
  title = {{Data-Driven Mean Field Equilibrium Computation in Large-Population LQG Games}},
  year = {2025},
  eprint = {2502.19993},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.19993},
}
Figure