Addressing Rotational Learning Dynamics in Multi-Agent Reinforcement Learning

Baraah A. M. Sidahmed, Tatjana Chavdarova

TL;DR

Multi-Agent Reinforcement Learning (MARL) suffers from a severe reproducibility crisis because competing agent objectives create "rotational" optimization dynamics that cause standard gradient descent to cycle endlessly or diverge. By reframing MARL through the lens of Variational Inequalities (VIs), this paper introduces LA-MARL, a computationally lightweight lookahead averaging method that neutralizes these rotational forces, drastically stabilizing training and ensuring convergence to true game-theoretic equilibria.

What Problem Does This Paper Address?

Why is training Multi-Agent Reinforcement Learning (MARL) algorithms so notoriously unstable? Even when using state-of-the-art algorithms like MADDPG or MAPPO, a change in the random seed can mean the difference between a highly coordinated team of agents and complete training failure.

The authors argue that this reproducibility crisis isn't just a minor tuning issue; it's a fundamental flaw in our choice of optimization tools. Standard optimization algorithms (like Gradient Descent and Adam) are designed to find the minimum of a single objective function. However, MARL is inherently a game played between multiple entities. When agents compete or interact, their conflicting objectives create a vector field with a heavy rotational component. Instead of moving directly toward an equilibrium point, standard gradient methods tend to orbit around it indefinitely or spiral outward into divergence.

Methods

To fix this, the authors reframe MARL as a Variational Inequality (VI) problem rather than a standard minimization problem. This mathematical shift allows them to borrow advanced optimization techniques specifically engineered to handle rotational dynamics.

They introduce LA-MARL (Lookahead MARL), a general wrapper framework that can be integrated into any standard actor-critic MARL algorithm (such as MADDPG or MATD3). The core mechanism is elegant and computationally cheap:

  1. Base Optimization: Agents update their actor and critic networks normally using a standard optimizer (like Adam) for $k$ steps.
  2. Lookahead Step ($\alpha$-averaging): Periodically, the algorithm takes a "snapshot" of the weights from $k$ steps ago and interpolates them with the current weights using a step size $\alpha \in [0,1]$.
  3. Nested Levels: LA-MARL can be applied recursively across multiple nested levels (e.g., three levels), where higher levels perform averaging over progressively longer intervals.

Because it relies purely on weight averaging rather than calculating extra gradients, LA-MARL introduces virtually zero computational overhead. The authors also explore EG-MARL, which adapts the classic Extragradient method to the MARL setting.

Datasets

The authors evaluated their methods on two distinct types of environments:

  • Classic Zero-Sum Games: Rock–Paper–Scissors (RPS) and Matching Pennies (MP). These are ideal for evaluation because their mixed Nash equilibria (MNE) can be analytically computed, providing an exact ground truth for policy convergence.
  • Multi-Agent Particle Environments (MPE): A popular benchmark for multi-agent coordination. Specifically, they used Predator-Prey (where cooperative predators must capture faster prey) and Physical Deception (where agents must reach a target while misleading an adversary).

Key Insights and Ideas

  • The Illusion of Reward Convergence: One of the paper's most striking insights is that stable, saturating rewards do not mean your agents have learned an optimal policy. In Rock-Paper-Scissors, standard MADDPG agents achieved perfectly flat, stable rewards by repeatedly picking the exact same action (resulting in constant ties). While the reward looked "converged," the policy was highly exploitable. LA-MARL, by contrast, successfully learned to randomize its actions, converging to the true mixed Nash equilibrium.
  • Contraction Beats Rotation: While the popular Extragradient (EG) method provides some benefits, lookahead averaging introduces a significantly stronger mathematical contraction. This makes it uniquely capable of dampening the severe oscillations found in deep neural network-based games.
  • Simultaneous Actor-Critic Averaging: Unlike single-agent lookahead applications, lookahead in MARL must be applied to the joint strategy space of all players simultaneously—including both actors and critics—to prevent agents from updating based on stale, un-averaged parameters.

Key Results

LA-MARL demonstrated substantial improvements over standard gradient descent (GD) baselines across all tested environments:

  • Zero-Sum Convergence: In both Rock-Paper-Scissors and Matching Pennies, standard GD-MARL completely diverged over time. LA-MARL consistently converged close to the true equilibrium policy, with MATD3 exhibiting the lowest variance.
Figure 1: Comparison between GD-(MADDPG/MATD3) and LA-(MADDPG/MATD3), on Rock–paper–scissors and Matching pennies. x-axis: training episodes. y-axis: total distance of agents’ policies to the equilibrium policy; averaged over 10 seeds.
  • Lookahead vs. Extragradient: When comparing optimization methods directly on the RPS game, multi-level Lookahead (LA) and Lookahead-Extragradient (LA-EG) combinations significantly outperformed both the baseline and standard Extragradient (EG).
Figure 2: Comparing the GD, LA, EG, and LA-EG optimiza- tion methods on the Rock–paper–scissors game. x-axis: training episodes. y-axis: squared norm of the learned policy probabilities relative to the equilibrium.
  • MPE Performance: In Predator-Prey, LA-trained agents demonstrated superior team coordination and consistently outperformed GD-trained agents in head-to-head matchups. In Physical Deception, LA-MADDPG achieved an adversary win rate closest to the ideal equilibrium value of 0.5, indicating a balanced, unexploitable strategy.

Limitations and Future Work

While LA-MARL delivers impressive empirical results, the strict theoretical convergence guarantees for VI methods typically assume the operator is monotone—a condition that rarely holds true for complex, non-convex deep reinforcement learning landscapes.

Additionally, the authors note that tuning the lookahead hyperparameters requires care: adding too many nested lookahead levels can lead to overly conservative or slow policy updates. Future work includes exploring adaptive lookahead schedules and developing more robust evaluation metrics for MARL that go beyond simple reward tracking.

Summary updated

Abstract

Multi-agent reinforcement learning (MARL) has emerged as a powerful paradigm for solving complex problems through agents' cooperation and competition, finding widespread applications across domains. Despite its success, MARL faces a reproducibility crisis. We show that, in part, this issue is related to the rotational optimization dynamics arising from competing agents' objectives, and require methods beyond standard optimization algorithms. We reframe MARL approaches using Variational Inequalities (VIs), offering a unified framework to address such issues. Leveraging optimization techniques designed for VIs, we propose a general approach for integrating gradient-based VI methods capable of handling rotational dynamics into existing MARL algorithms. Empirical results demonstrate significant performance improvements across benchmarks. In zero-sum games, Rock--paper--scissors and Matching pennies, VI methods achieve better convergence to equilibrium strategies, and in the Multi-Agent Particle Environment: Predator-prey, they also enhance team coordination. These results underscore the transformative potential of advanced optimization techniques in MARL.
BibTeX
@inproceedings{sidahmed2025addressing,
  author = {Baraah A. M. Sidahmed and Tatjana Chavdarova},
  title = {{Addressing Rotational Learning Dynamics in Multi-Agent Reinforcement Learning}},
  booktitle = {EWRL},
  year = {2025},
  url = {https://openreview.net/forum?id=RtbH7KnH0n},
}
Figure