Cooperative Policy Agreement: Learning Diverse Policy for Offline MARL

Yihe Zhou, Yuxuan Zheng, Yue Hu, Kaixuan Chen, Tongya Zheng, Jie Song, Mingli Song, Shunyu Liu

Abstract

Offline multi-agent reinforcement learning (MARL) is often hindered by "policy mismatch," where agents learn incompatible actions from datasets containing multiple successful strategies. This paper introduces Cooperative Policy Agreement (CPA), a novel framework that uses autoregressive training to coordinate agents and an "agreement key" mechanism to ensure synchronized strategy selection during decentralized execution. CPA effectively captures diverse successful strategies while maintaining high performance across various coordination benchmarks.

The Policy Mismatch Problem

In offline MARL, agents are trained on static datasets without environment interaction. While standard offline RL focuses on avoiding out-of-distribution (OOD) actions, multi-agent settings introduce a unique hurdle: policy mismatch. If a dataset contains two distinct winning strategies (A and B), Agent 1 might learn Strategy A while Agent 2 learns Strategy B. During execution, their combined joint action fails because the individual strategies are incompatible.

Comparison of online learning, OOD problems, and the policy mismatch problem in offline MARL.

As illustrated above, while online learning naturally stays on a single trajectory, offline learning from mixture datasets can lead to fragmented learning where agents fail to agree on a single mode of behavior, resulting in suboptimal joint actions.

Methods: Cooperative Policy Agreement (CPA)

To address this, CPA operates in two distinct phases that bridge the gap between centralized coordination and decentralized execution:

1. Autoregressive (AR) Teacher Training

CPA first trains an autoregressive teacher model. In this setup, agents are trained sequentially: Agent $i$ conditions its action on the actions of all preceding agents $1, \dots, i-1$. This sequential dependency naturally prevents mismatch because each agent can react to its teammates' choices, ensuring the joint action remains within the distribution of a single strategy.

2. Policy Agreement (PA) Distillation

Since real-time sequential action is impossible in decentralized execution (CTDE), CPA distills the AR teacher's knowledge into a non-autoregressive model using two clever components:

The CPA framework architecture, featuring Autoregressive Training and Policy Agreement Training phases.
  • State-Conditional Action Pool: Vector Quantization (VQ) is used to create a discrete "menu" of effective joint actions for any given state.
  • The Agreement Key ($k$): A Conditional Variational Autoencoder (CVAE) learns a latent variable $k$. At execution time, agents use a shared random seed or timestamp to generate the same $k$, allowing them to simultaneously select the same strategy from the action pool without explicit communication.

Experimental Results

CPA was evaluated on five coordination benchmarks, including XOR, Permutation, Bridge, Sensor, and Aloha. The results demonstrate that CPA effectively handles mixture datasets where baselines often fail.

  • Coordination Puzzles: On XOR and Permutation tasks with conflicting strategies, CPA achieved 100% success, while baselines often collapsed to 0% due to strategy clashing.
  • Complex Scenarios: In the Aloha communication network task, CPA achieved a return of ~74, significantly higher than the best baseline (AlberDICE at ~49).
  • Diversity Maintenance: By adjusting a diversity threshold ($\beta$), CPA can maintain multiple high-probability "modes" of behavior, making the team more robust than methods that converge to a single "average" policy.

Conclusion and Future Work

CPA demonstrates that coordination in offline MARL does not require real-time communication if agents can agree on a shared strategy "key." This conceptual shift allows for diverse, high-performing policies in complex multi-agent environments. While currently designed for fully observable environments (MMDPs), future research will explore extending the agreement mechanism to partially observable settings (Dec-POMDPs), where agents may have differing views of the world state.

Summary updated

Abstract

Offline Multi-Agent Reinforcement Learning (MARL) aims to learn optimal joint policies from pre-collected datasets without further interaction with the environment. Despite the encouraging results achieved so far, we identify the policy mismatch problem that arises from employing diverse offline MARL datasets, a highly important ingredient for cooperative generalization yet largely overlooked by existing literature. Specifically, in the case that offline datasets exhibit various optimal joint policies, policy mismatch often occurs when individual actions from different optimal joint actions are combined in a way that results in a suboptimal joint action. In this paper, we introduce a novel Cooperative Policy Agreement (CPA) method, that not only mitigates the policy mismatch problem but also learns to generate diverse joint policies. CPA firstly introduces an autoregressive decision-making mechanism among agents during offline training. This mechanism enables agents to access the actions previously taken by other agents, thereby facilitating effective joint policy matching. Moreover, diverse joint policies can be directly obtained through sequential action sampling from the autoregressive model. Then we further incorporate a policy agreement mechanism to convert these autoregressive joint policies into decentralized policies with a non-autoregressive form, while still ensuring the diversity of the generated policies. This mechanism guarantees that the proposed CPA adheres to the Centralized Training with Decentralized Execution (CTDE) constraint. Experiments conducted on various benchmarks demonstrate that CPA yields superior performance to state-of-the-art competitors.
BibTeX
@inproceedings{zhou2025cooperative,
  author = {Yihe Zhou and Yuxuan Zheng and Yue Hu and Kaixuan Chen and Tongya Zheng and Jie Song and Mingli Song and Shunyu Liu},
  title = {{Cooperative Policy Agreement: Learning Diverse Policy for Offline MARL}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i21.34465},
}
Figure