Abstract
Multi-Agent Guided Policy Search (MA-GPS) is a hybrid framework designed to stabilize reinforcement learning in non-cooperative dynamic games. By integrating local, model-based Linear-Quadratic (LQ) approximations as regularizing "priors" in the reward function, MA-GPS mitigates common training instabilities such as "limit cycles." This approach facilitates faster convergence and high-quality policy generation in complex multi-agent environments, including autonomous driving and strategic sports simulations.
Introduction
In non-cooperative dynamic games, multiple agents—such as self-driving cars or competing robots—pursue distinct and often conflicting objectives. Identifying a Nash equilibrium in these settings is a fundamental challenge. Existing solutions typically fall into two categories: model-based optimization and Multi-Agent Reinforcement Learning (MARL). Model-based methods like iLQGames offer stability but scale poorly with agent count and are computationally intensive. Conversely, MARL methods like Multi-Agent Policy Gradients (MA-PG) scale better but suffer from extreme instability due to non-stationarity; as agents learn simultaneously, they often enter "limit cycles," endlessly chasing shifting strategies without converging.
MA-GPS bridges this gap by utilizing model-based control to guide the scalability of reinforcement learning.
Methodology: The Guidance Mechanism
The core of MA-GPS is the introduction of a model-based prior into the MARL reward function as a form of regularization. A penalty term is added to each agent's reward, discouraging significant deviations from a "guiding" policy.
To generate this guidance without a pre-computed global solution, MA-GPS linearizes the game's dynamics and quadraticizes its costs around the current trajectories produced by the agents' neural network policies. This creates a "local LQ game" at every training step. Solving this local game is computationally efficient and provides a reference Nash equilibrium.
The authors theoretically demonstrate that for LQ games, this guidance shifts the system's pseudo-gradient eigenvalues, guaranteeing local exponential convergence even when the original gradient dynamics are unstable.
Experimental Evaluation
The framework was evaluated across three diverse environments:
- Synthetic LQ Games: A 2-player benchmark to verify theoretical stability.
- Nonlinear Vehicle Platooning: A three-vehicle scenario requiring a leader to guide followers into a specific lane, testing nonlinear dynamics handling.
- Six-Player Strategic Basketball: A 3-on-3 simulation in a 24-dimensional space. Agents with specialized roles (guards, centers, forwards) must coordinate and compete under non-differentiable constraints.
Key Results and Insights
- Stability and Convergence: MA-GPS achieved higher total rewards and faster convergence than baselines like IPPO, MA-PPO, and MA-DDPG. The guidance term significantly reduced variance during training.
- Real-Time Performance: While classical solvers might take ~0.64 seconds per step in the basketball environment, MA-GPS-trained neural networks execute in ~0.0016 seconds—a 400x speedup enabling real-time deployment.
- Strategic Emergence: In basketball simulations, MA-GPS was uniquely successful in training agents to adopt distinct strategic formations, such as centers protecting the basket while guards positioned at the perimeter.
- Robustness to Bias: Surprisingly, even imperfect or biased guiding policies can stabilize training. As long as the guidance weight (ρ) is sufficient, it prevents limit cycles, allowing agents to eventually refine their policies.
Discussion and Future Work
MA-GPS presents a powerful method for stabilizing MARL via structural game-theoretic information. However, it currently assumes access to an approximate environment model for LQ linearization. Future research may focus on learning these dynamics models online. Additionally, there is a trade-off between stability and bias: high guidance weights ensure convergence but may bias policies toward local LQ solutions rather than true nonlinear Nash equilibria. Unifying this guidance with traditional entropy-based exploration remains an open area for investigation.