Learning in Non-stationary Multi-Agent Environments

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2657 papers725 conceptsfrozen 2026-10-04

In standard reinforcement learning (RL), an agent learns by interacting with a stationary environment. The rules of the game—the rewards and the way the world changes—are fixed. However, in multi-agent reinforcement learning (MARL), this assumption collapses. Because all agents are learning simultaneously, the "environment" for any single agent includes the behaviors of all other agents. As Agent A improves its strategy, the world effectively changes for Agent B.

This phenomenon is known as non-stationarity. It creates a "moving target" problem where the optimal policy for one agent is constantly shifting, often leading to high gradient variance, unstable training, and a failure to converge to coordinated behaviors.

The Moving Target Problem

The core challenge of non-stationarity is the violation of the Markov property. In a single-agent setting, the probability of transitioning to a new state depends only on the current state and action. In MARL, as noted by Ryan Lowe and Yi Wu in Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments, the transition dynamics and rewards perceived by an individual agent are influenced by the evolving policies of others.

If an agent uses a standard experience replay buffer—a cornerstone of deep RL—it might try to learn from data collected hours ago. But if its teammates or opponents have changed their strategies since then, that old data becomes misleading or "stale," as explored by Jakob N. Foerster in Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning.

Stabilizing the Learning Signal

To combat this instability, researchers have developed several key mechanisms to ground the learning process:

Centralized Training, Decentralized Execution (CTDE) The most influential framework for addressing non-stationarity is CTDE. The intuition is simple: during training, we can afford to let agents see "extra" information, such as the states and actions of their peers. This global view makes the environment appear stationary from the perspective of a centralized critic. Once trained, the agents use their decentralized actors to perform tasks using only their local observations. MADDPG popularized this by using a centralized critic to provide a stable gradient for decentralized policies.

Opponent and Teammate Modeling If you cannot see the internal state of other agents, you must predict them. Gerald Tesauro introduced Hyper-Q Learning, which learns the value of mixed strategies by estimating opponent behavior. More recently, Zheng Tian proposed ROMMEO, which uses probabilistic inference to model an opponent's optimal future behavior, allowing an agent to adapt to changes before they fully manifest.

Fingerprinting and Importance Sampling To make experience replay viable again, researchers "tag" data. Jakob N. Foerster introduced Multi-Agent Fingerprints, which add the training iteration or exploration rate to the replay buffer. This allows the agent to disambiguate old behaviors from new ones. Alternatively, Multi-Agent Importance Sampling re-weights old transitions to reflect how likely they would be under the current policies of the other agents.

Coordination through Communication and Hierarchy

Beyond stabilization, agents must actively align their shifting strategies.

The Path to General Adaptation

The ultimate goal of this research is Ad Hoc Teamwork (AHT)—the ability to collaborate with completely unknown teammates on the fly. Frameworks like PADiff by Antao Xiang use generative diffusion models to capture the multimodal nature of cooperation, predicting teammate intentions to ensure that even as the team composition changes, the agent's policy remains aligned with the collective goal. By treating non-stationarity not as a bug to be suppressed, but as a dynamic to be modeled, these systems achieve a level of fluid coordination that mirrors human teamwork.

Papers in this version

Showing 60 of 657 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 984 papers
  2. Version 1 · 2026-10-04 · 657 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_multi_agent_non_stationarity_774e37c7,
  title = {multi-agent non-stationarity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-agent-non-stationarity/rdr_774e37c734f2469ab4fc2f9783c8c305},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_learning_in_non_stationary_multi_agent_e_774e37c7_v1,
  title = {Learning in Non-stationary Multi-Agent Environments},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-in-non-stationary-multi-agent-environments/rdr_774e37c734f2469ab4fc2f9783c8c305/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}