In standard single-agent reinforcement learning, the environment behaves like a stable, if unpredictable, physical world. The agent takes an action, the environment transitions to a new state based on fixed underlying laws, and the agent receives a reward. This stability allows the agent to steadily map out an optimal policy.
In multi-agent reinforcement learning (MARL), however, this foundational assumption breaks down completely. When multiple agents learn simultaneously, the environment becomes non-stationary from the perspective of any individual agent. An action that yielded a high reward yesterday might result in failure today simply because another agent has modified its behavior. This is known as the "moving target" problem, and it frequently causes concurrent learning algorithms to oscillate, diverge, or get trapped in poor local optima.
To solve this, a powerful paradigm has emerged: instead of updating all agents at once, we update them sequentially or alternately. By breaking the simultaneity of the updates, we can restore environmental stationarity and guarantee stable convergence.
The Core Mechanism: Restoring Stationarity by Freezing Time
The intuition behind sequential policy updates is straightforward: if simultaneous updates cause non-stationarity, we can force the environment to stand still by updating only one agent at a time.
In an alternating or sequential update schedule, the parameters of all agents except one are frozen. The active agent then optimizes its policy against what is now a temporarily static environment. Once its update step is complete, its policy is locked in place, and the next agent takes its turn, optimizing its own policy while accounting for the newly updated strategies of its peers.
Simultaneous Updates (Unstable):
Agent A Updates ──► ◄── Agent B Updates (Moving Target!)
Sequential Updates (Stable):
Agent A Updates ──► [Agent B Frozen] (Stationary Environment)
│
▼
[Agent A Frozen] ◄── Agent B Updates (Accounts for A's new policy)
This approach fundamentally changes the nature of the optimization problem. As early work by Bikramjit Banerjee et al. demonstrated in Sample Bounded Distributed Reinforcement Learning for Decentralized POMDPs, alternating updates transform a complex, moving multi-agent game into a series of isolated, single-agent best-response tasks. By ensuring that agents do not simultaneously adapt to one another's changing behaviors, this sequential handoff provides significantly better convergence stability than concurrent learning in decentralized environments like Dec-POMDPs, a benefit further reinforced by the initialization strategies explored by Landon Kraemer et al. in Informed Initial Policies for Learning in Dec-POMDPs.
Why It Works: Monotonic Improvement and Global Optimality
The reason sequential updates perform so reliably comes down to a mathematical guarantee: monotonic improvement of the joint policy.
When agents update concurrently, their combined changes can inadvertently degrade the team's overall performance. But when updates are ordered sequentially, each agent's optimization step is mathematically guaranteed to maintain or improve the joint objective, because it is optimizing directly against the real, current state of the other agents.
The theoretical boundaries of this mechanism were formalized by Yulai Zhao et al. in Local Optimization Achieves Global Optimality in Multi-Agent Reinforcement Learning. By utilizing a multi-agent performance difference lemma, the authors proved that updating agents sequentially allows a collective system to reach global optimality at a sublinear rate, even when using function approximation to handle massive state spaces.
This sequential ordering means that within a single training epoch, later agents directly inherit and account for the policy modifications made by preceding agents. This insight underpins modern heterogeneous-agent algorithms like HAPPO. For example, Youpeng Zhao et al. leverage this sequential update scheme in Multi-Agent First Order Constrained Optimization in Policy Space to successfully navigate complex safety constraints without relying on computationally expensive second-order mathematics. Similarly, Jiarong Liu et al. demonstrate in Maximum Entropy Heterogeneous-Agent Reinforcement Learning that combining sequential updates with maximum-entropy objectives allows heterogeneous teams to systematically escape suboptimal local traps and converge to a robust Quantal Response Equilibrium.
Co-Evolution in Dual-Agent Systems and LLMs
The alternating update paradigm is uniquely suited to dual-agent systems where two entities must co-evolve to solve a shared task.
An early, classic manifestation of this is the joint training of a dialog agent and a user simulator. If both networks try to learn simultaneously, the communication protocol quickly degrades into noise. To stabilize this, Bing Liu et al. in Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models and Pei-Hung Chung et al. in Joint Learning of Interactive Spoken Content Retrieval and Trainable User Simulator utilized an alternating schedule. Freezing the user simulator while optimizing the retrieval agent (and vice versa) prevents divergence, allowing a sophisticated, human-like communication policy to emerge.
This exact principle has recently become a cornerstone for training Large Language Models (LLMs) and Vision-Language Models (VLMs) on complex reasoning tasks through multi-agent framing:
- Module Communication: Diji Yang et al. developed an alternating training strategy in Tackling Vision Language Tasks through Learning Inner Monologues to optimize natural language dialogue protocols between specialized vision and language modules without encountering non-stationary instability.
- Role Decoupling: In ReMA: Learning to Meta-Think for LLMs with Multi-agent Reinforcement Learning, Ziyu Wan et al. split LLM reasoning into a high-level planner and a low-level executor. By freezing one role while updating the other via reinforcement learning, they ensure the strategic and executive capabilities co-evolve without chaotic training dynamics.
- Adversarial Co-Evolution: Claire Cardie et al. present a "dual-play" framework in Better LLM Reasoning via Dual-Play that alternates updates between a Proposer model and a Solver model. This decoupled paradigm prevents the typical volatility of online competitive updates, driving rapid improvements in mathematical reasoning without human supervision.
Managing Heterogeneity and Multi-Scale Dynamics
In complex real-world systems, agents are rarely identical. They often possess different action spaces, control distinct types of hardware, or operate across entirely different timescales. Sequential updates act as a natural coordinator across these structural divides.
In decentralized engineering applications, such as vehicular networks and digital twin infrastructures, managing resource competition requires algorithms that can handle severe agent heterogeneity. As Dongyu Chen et al. show in Mobility-Aware Multi-Task Decentralized Federated Learning for Vehicular Networks: Modeling, Analysis, and Optimization and Wenshuai Liu et al. demonstrate in Two-Timescale Synchronization and Migration for Digital Twin Networks: A Multi-Agent Deep Reinforcement Learning Approach, updating heterogeneous agents one by one ensures steady, monotonic improvement of the joint policy, bypassing the friction that occurs when diverse action spaces collide simultaneously.
This mechanism also elegantly resolves multi-timescale friction. In non-terrestrial networks where ground equipment and satellites operate on vastly different computational and temporal scales, Yang Cao et al. demonstrate in Toward Intelligent Non-Terrestrial Networks Through Symbiotic Radio: A Collaborative Deep Reinforcement Learning Scheme that sequential updates keep the environment temporarily stationary, providing rigorous convergence guarantees across distinct time horizons.
Ultimately, whether coordinating autonomous vehicles to alleviate highway bottlenecks—as explored by Lu Liu et al. via agent-by-agent policy iteration in A Multi-Agent Rollout Approach for Highway Bottleneck Decongestion in Mixed Autonomy—or training deep networks to converse, the underlying intuition remains the same: the most reliable way to optimize a collective system is to let its components adapt one at a time.
Key Foundational Work
Sample Bounded Distributed Reinforcement Learning for Decentralized POMDPs (2012) — contributed 2 concept elements to this direction.
Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models (2017) — contributed 1 concept elements to this direction.
Informed Initial Policies for Learning in Dec-POMDPs (2012) — contributed 1 concept elements to this direction.