Abstract: Learning by experience in Multi-Agent Systems (MAS) is a difficult and exciting task, due to the lack of stationarity of the environment, whose dynamics evolves as the population learns. In order to design scalable algorithms for systems with a large population of interacting agents (e.g. swarms), this paper focuses on Mean Field MAS, where the number of agents is asymptotically infinite. Recently, a very active burgeoning field studies the effects of diverse reinforcement learning algorithms for agents with no prior information on a stationary Mean Field Game (MFG) and learn their policy through repeated experience. We adopt a high perspective on this problem and analyze in full generality the convergence of a fictitious iterative scheme using any single agent learning algorithm at each step. We quantify the quality of the computed approximate Nash equilibrium, in terms of the accumulated errors arising at each learning iteration step. Notably, we show for the first time convergence of model free learning algorithms towards non-stationary MFG equilibria, relying only on classical assumptions on the MFG dynamics. We illustrate our theoretical results with a numerical experiment in a continuous action-space environment, where the approximate best response of the iterative fictitious play scheme is computed with a deep RL algorithm.
Introduction
Learning in Multi-Agent Systems (MAS) is notoriously difficult due to environmental non-stationarity. As agents learn and adapt, the dynamics of the environment shift, creating a moving target for optimization. Mean Field Games (MFGs) address this by considering an infinite population of identical agents, where individuals react to the aggregate distribution (the "mean field") rather than every individual interaction.
While MFGs simplify the problem, traditional solutions often require perfect knowledge of game dynamics (model-based planning). This paper explores model-free reinforcement learning (RL) in MFGs, specifically addressing how approximation errors from RL algorithms propagate when learning non-stationary Nash equilibria. The authors ask a fundamental question: How do the approximation errors of any standard RL algorithm propagate when trying to learn a non-stationary Mean Field Nash Equilibrium?
The Approximate Fictitious Play Scheme
The authors propose an Approximate Fictitious Play (FP) iterative scheme. In classical game-theoretic Fictitious Play, agents play an exact best response to the historical average of their opponents' strategies. In the proposed model-free setting, agents instead compute an approximate best response using a single-agent RL algorithm, such as Deep Deterministic Policy Gradient (DDPG), at each iteration.
The core innovation lies in the theoretical analysis of error propagation. By treating the RL component as a black-box oracle that provides an approximate best response with some error $\ell_n$, the authors bound the system's total "exploitability." This metric quantifies how much an agent could gain by deviating from the learned policy to the true optimal policy. A key finding is that the Fictitious Play framework acts as a stabilizer: the global error is bounded by the average of individual learning errors plus a term that decays at a rate of $O(1/n)$.
Experimental Validation: Congestion Avoidance
To validate the theory, the researchers designed a continuous state-action MFG environment. In this "Congestion Avoidance" scenario, agents move on a 1D continuous torus (a line segment that loops back on itself) and are rewarded for avoiding crowded areas. This environment is particularly useful because it possesses a known closed-form mathematical solution in continuous time, allowing for direct comparison between learned results and ground truth.
Results and Key Insights
The experiments demonstrate that model-free learning can indeed converge to non-stationary MFG equilibria. Using DDPG for the approximate best response and Monte Carlo sampling for density estimation, the algorithm successfully matched the theoretical benchmarks.
As shown in the results, the $L_2$ error for both the population density and the control policy consistently decreased over Fictitious Play iterations. This empirical success supports the theoretical claim that the FP scheme can tolerate and average out the inherent noise and inaccuracies of deep RL algorithms. Notably, this work provides the first proof of convergence for model-free algorithms toward non-stationary MFG equilibria under standard, easily verifiable monotonicity assumptions.
Limitations and Future Directions
A primary theoretical constraint identified is the requirement for the RL algorithm's output to be "close" to the optimal policy in terms of Wasserstein distance. While standard RL typically focuses on value (reward) convergence, ensuring policy closeness is more demanding. The authors suggest that future work could employ regularized MDPs to enforce policy smoothness and uniqueness (Lipschitz continuity). Additionally, extending these centralized learning results to fully decentralized multi-agent settings remains an open and promising research path.
Summary updated