Abstract
Offline multi-agent reinforcement learning (MARL) is often hindered by "policy mismatch," where agents learn incompatible actions from datasets containing multiple successful strategies. This paper introduces Cooperative Policy Agreement (CPA), a novel framework that uses autoregressive training to coordinate agents and an "agreement key" mechanism to ensure synchronized strategy selection during decentralized execution. CPA effectively captures diverse successful strategies while maintaining high performance across various coordination benchmarks.
The Policy Mismatch Problem
In offline MARL, agents are trained on static datasets without environment interaction. While standard offline RL focuses on avoiding out-of-distribution (OOD) actions, multi-agent settings introduce a unique hurdle: policy mismatch. If a dataset contains two distinct winning strategies (A and B), Agent 1 might learn Strategy A while Agent 2 learns Strategy B. During execution, their combined joint action fails because the individual strategies are incompatible.
As illustrated above, while online learning naturally stays on a single trajectory, offline learning from mixture datasets can lead to fragmented learning where agents fail to agree on a single mode of behavior, resulting in suboptimal joint actions.
Methods: Cooperative Policy Agreement (CPA)
To address this, CPA operates in two distinct phases that bridge the gap between centralized coordination and decentralized execution:
1. Autoregressive (AR) Teacher Training
CPA first trains an autoregressive teacher model. In this setup, agents are trained sequentially: Agent $i$ conditions its action on the actions of all preceding agents $1, \dots, i-1$. This sequential dependency naturally prevents mismatch because each agent can react to its teammates' choices, ensuring the joint action remains within the distribution of a single strategy.
2. Policy Agreement (PA) Distillation
Since real-time sequential action is impossible in decentralized execution (CTDE), CPA distills the AR teacher's knowledge into a non-autoregressive model using two clever components:
- State-Conditional Action Pool: Vector Quantization (VQ) is used to create a discrete "menu" of effective joint actions for any given state.
- The Agreement Key ($k$): A Conditional Variational Autoencoder (CVAE) learns a latent variable $k$. At execution time, agents use a shared random seed or timestamp to generate the same $k$, allowing them to simultaneously select the same strategy from the action pool without explicit communication.
Experimental Results
CPA was evaluated on five coordination benchmarks, including XOR, Permutation, Bridge, Sensor, and Aloha. The results demonstrate that CPA effectively handles mixture datasets where baselines often fail.
- Coordination Puzzles: On XOR and Permutation tasks with conflicting strategies, CPA achieved 100% success, while baselines often collapsed to 0% due to strategy clashing.
- Complex Scenarios: In the Aloha communication network task, CPA achieved a return of ~74, significantly higher than the best baseline (AlberDICE at ~49).
- Diversity Maintenance: By adjusting a diversity threshold ($\beta$), CPA can maintain multiple high-probability "modes" of behavior, making the team more robust than methods that converge to a single "average" policy.
Conclusion and Future Work
CPA demonstrates that coordination in offline MARL does not require real-time communication if agents can agree on a shared strategy "key." This conceptual shift allows for diverse, high-performing policies in complex multi-agent environments. While currently designed for fully observable environments (MMDPs), future research will explore extending the agreement mechanism to partially observable settings (Dec-POMDPs), where agents may have differing views of the world state.
Summary updated