ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization

The Viet Bui, Thanh Hong Nguyen, Tien MaiICLR 2024 · OpenReview

TL;DR

This paper introduces ComaDICE, a novel algorithm for offline cooperative multi-agent reinforcement learning (MARL). ComaDICE tackles the critical challenge of distributional shift in MARL by regularizing the stationary distribution of policies, leading to superior performance across complex benchmarks compared to existing state-of-the-art methods.

What Problem Does This Paper Address?

Deep reinforcement learning (RL) has shown remarkable success, but its reliance on extensive environmental interaction is a significant limitation. Offline RL aims to learn policies solely from pre-collected datasets, but this approach faces a critical issue: distributional shift. This occurs when the learned policy deviates from the data-generating policy, leading to inaccurate value estimations and extrapolation errors. This problem is significantly amplified in multi-agent settings (MARL) due to the exponentially larger joint state-action space and the complex interdependencies between agents. Existing offline MARL methods often adapt single-agent techniques or use centralized training decentralized execution (CTDE) with local policy regularization, but they struggle to effectively address the distributional shift in the joint MARL context.

Methods

ComaDICE introduces a principled framework for offline cooperative MARL by incorporating stationary distribution regularization into the standard offline RL objective. This is achieved by adapting the DIstribution Correction Estimation (DICE) approach, which constrains the difference between the stationary distributions of the learning and behavior policies.

Key methodological contributions include:

  • Value Decomposition for Global and Local Policies: ComaDICE adopts the CTDE paradigm but uniquely decomposes the global value function and global advantage functions (rather than Q-functions) using a mixing network. This allows for theoretical analysis and ensures consistency between global and local policy optimization.
  • Stationary Distribution Correction Estimation (DICE): The core of ComaDICE lies in its use of DICE to regularize the stationary distribution, directly addressing the distributional shift problem in a principled manner.
  • Weighted Behavioral Cloning (BC) for Policy Extraction: After learning the optimal value functions, ComaDICE extracts local policies using a weighted BC approach. This method leverages the learned occupancy ratio to guide local policy optimization, ensuring consistency with the global objective.
  • Theoretical Guarantees: The paper provides theoretical analysis demonstrating that the learning objective is convex under certain conditions, ensuring stable and efficient training.

Datasets

The paper evaluates ComaDICE on three standard MARL environments:

  • SMACv1 (StarCraft Multi-Agent Challenge): A benchmark for cooperative MARL tasks based on StarCraft II.
  • SMACv2: An extension of SMACv1 that introduces increased randomness and diversity in tasks.
  • Multi-Agent MuJoCo (MaMujoco): A benchmark for continuous cooperative multi-agent robotic control tasks.

The offline datasets for these environments were generated using MAPPO-trained agents (for SMACv1 and SMACv2) and the HAPPO method (for MaMujoco).

Key Insights and Ideas

  • Stationary Distribution Regularization is Key for MARL: The paper's central insight is that directly regularizing the stationary distribution of the joint policy is more effective for offline MARL than simply regularizing local policies or Q-functions. This approach directly tackles the distributional shift in the high-dimensional joint action space.
  • Decomposing Value and Advantage Functions is Crucial: Uniquely decomposing the global value and advantage functions (not just Q-functions) within the CTDE framework is essential for enabling theoretical analysis and ensuring consistent global-local policy optimization in the context of DICE.
  • 1-Layer Mixer Networks Excel in Offline MARL: Counterintuitively to some online MARL findings, the study shows that simpler 1-layer mixer networks lead to more stable and superior performance in offline MARL settings, potentially due to reduced overfitting.
  • Alpha (α) Balances Exploration and Imitation: The regularization parameter α plays a critical role in balancing the trade-off between maximizing rewards and adhering to the behavior policy. An intermediate value of α (around 10) consistently provides the best performance, indicating an optimal balance.

Key Results

ComaDICE consistently outperforms state-of-the-art baselines across various MARL tasks:

  • SMACv1 & SMACv2: ComaDICE achieves superior winrates and returns across nearly all tested scenarios, demonstrating its effectiveness in complex cooperative settings. For instance, in SMACv2, ComaDICE achieves significantly higher average returns than baselines like OMIGA and OMAR.
  • MaMujoco: ComaDICE also demonstrates superior performance in continuous control tasks, achieving higher average returns compared to baselines.
  • Robustness: The algorithm exhibits lower variance in its performance, indicating more stable and reliable learning.

Limitations and Future Work

  • Cooperative-Competitive Settings: The current work focuses solely on cooperative MARL. Extending ComaDICE to handle cooperative-competitive scenarios is an interesting future direction.
  • Behavior Policy Dependence: The performance of ComaDICE, like other offline RL methods, is dependent on the quality of the behavior policy. Future research could explore ways to mitigate this dependence.
  • Sample Efficiency: The algorithm, like its baselines, still requires a substantial amount of data. Improving sample efficiency remains a valuable area for future research.
  • Hyperparameter Sensitivity: While the paper identifies an optimal range for α, further research into more robust hyperparameter tuning or adaptive methods could be beneficial.

Summary updated

Abstract

Offline reinforcement learning (RL) has garnered significant attention for its ability to learn effective policies from pre-collected datasets without the need for further environmental interactions. While promising results have been demonstrated in single-agent settings, offline multi-agent reinforcement learning (MARL) presents additional challenges due to the large joint state-action space and the complexity of multi-agent behaviors. A key issue in offline RL is the distributional shift, which arises when the target policy being optimized deviates from the behavior policy that generated the data. This problem is exacerbated in MARL due to the interdependence between agents' local policies and the expansive joint state-action space. Prior approaches have primarily addressed this challenge by incorporating regularization in the space of either Q-functions or policies. In this work, we introduce a regularizer in the space of stationary distributions to better handle distributional shift. Our algorithm, ComaDICE, offers a principled framework for offline cooperative MARL by incorporating stationary distribution regularization for the global learning policy, complemented by a carefully structured multi-agent value decomposition strategy to facilitate multi-agent training. Through extensive experiments on the multi-agent MuJoCo and StarCraft II benchmarks, we demonstrate that ComaDICE achieves superior performance compared to state-of-the-art offline MARL methods across nearly all tasks.
BibTeX
@inproceedings{bui2024comadice,
  author = {The Viet Bui and Thanh Hong Nguyen and Tien Mai},
  title = {{ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=K2Moh11t4s},
}
Figure