Stabilizing Multi-Agent Learning in Non-Stationary Environments

This direction is merged into another direction: multi-agent non-stationarity. Its last version stays readable and citable here.

Version 1 of 115 papers15 conceptsfrozen 2026-10-04

In standard reinforcement learning, an agent learns by interacting with a static environment. If the agent takes action $A$ in state $S$, it can expect a consistent transition to state $S'$. However, in multi-agent reinforcement learning (MARL), this consistency vanishes. Because every agent is learning and updating its policy simultaneously, the "environment" from the perspective of any single agent is constantly shifting.

This is the problem of non-stationarity. As Cheng Li and Levi Fussell describe it in Multi-agent reinforcement learning for character control, agents face a "moving target" problem. When the rules of the world change as fast as you learn them, stable coordination becomes nearly impossible.

The Managerial Solution: Decoupling Actions from Goals

One way to anchor this shifting landscape is to introduce a hierarchy. In Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning, Peter Dayan and Sanjeevan Ahilan propose a "manager-worker" structure.

Instead of all agents trying to learn the global task at once, a high-level manager agent assigns specific subgoals to workers. This decoupling is powerful because it isolates the workers from the global non-stationarity. A worker only needs to learn how to achieve the goal set by its manager; it doesn't need to worry about how the other workers are evolving, as long as the manager can coordinate the subgoals effectively.

Modeling the Social Fabric

If the environment is non-stationary because of other agents, why not model those agents directly? Andrea Tacchetti, Matthew Botvinick, and their colleagues introduced Relational Forward Models (RFMs) to do exactly this.

By using graph-based neural networks, RFMs treat the multi-agent system as a set of entities and interactions. Instead of treating other agents as random noise in the environment, the model predicts their future behavior and rewards. When agents can "anticipate" the social dynamics of their peers, the environment feels less like a chaotic moving target and more like a predictable system of interactions.

Centralized Wisdom for Decentralized Action

A recurring strategy to bypass non-stationarity during training is Centralized Training with Decentralized Execution (CTDE). The idea is simple: during training, we have access to "global" information (the states and actions of all agents), but during execution, agents must act based only on their local observations.

  • Imitation Learning: Alex Tong Lin, Guido Montúfar, and others developed CESMA, which trains a "god-view" centralized expert first. Because the expert sees everything, it doesn't suffer from the local uncertainty of non-stationarity. Decentralized agents then use imitation learning to mimic this expert.
  • Guided Rewards: In UAV attack defense confrontation, researchers used the MADDPG algorithm to allow a swarm of drones to track targets. By using a centralized critic that sees all drone actions, the system stabilizes the learning process, allowing the swarm to converge on a cooperative path-planning strategy that would be impossible if each drone learned in isolation.

Factorization and Influence Prediction

In massive systems, centralized training becomes a bottleneck. Matthijs T. J. Spaan, Wendelin Boehmer, and their team proposed a different approach called DIALS.

Instead of one giant simulator, they factorize the world into local simulators. To handle the non-stationarity caused by neighboring regions, they use "influence predictors." These predictors model how outside agents will affect the local environment. This allows agents to train in parallel and at high speeds while still accounting for the "shifting" influence of the rest of the network.

The Future: Embodied and Generative Coordination

As the field moves toward Multi-Agent Embodied AI, researchers like Yang Yu are looking at how Large Language Models (LLMs) can mitigate non-stationarity. By using generative models for high-level reasoning and communication, agents can explicitly negotiate and allocate tasks. This moves the solution from purely mathematical stabilization (like gradient adjustments) to cognitive coordination, where agents use communication to make the environment stationary by agreement.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_stabilizing_multi_agent_learning_in_non__d6cdb93b,
  title = {Stabilizing Multi-Agent Learning in Non-Stationary Environments},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stabilizing-multi-agent-learning-in-non-stationary-environments/rdr_d6cdb93b062f4203abc510051db67a18},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_stabilizing_multi_agent_learning_in_non__d6cdb93b_v1,
  title = {Stabilizing Multi-Agent Learning in Non-Stationary Environments},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stabilizing-multi-agent-learning-in-non-stationary-environments/rdr_d6cdb93b062f4203abc510051db67a18/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}