Divergence-Regularized Multi-Agent Actor-Critic

Kefan Su, Zongqing Lu

Abstract

Divergence-Regularized Multi-Agent Actor-Critic (DMAC) is a novel framework designed to bring the advantages of divergence regularization to cooperative multi-agent reinforcement learning (MARL). Unlike traditional entropy regularization, which can bias the optimal policy toward sub-optimal behavior, DMAC penalizes the divergence between the current policy and a target policy. This approach provides a theoretical guarantee of monotonic improvement in the original environment objective while enhancing exploration and stability. Evaluations on the StarCraft Multi-Agent Challenge (SMAC) and didactic stochastic games show that DMAC consistently improves the performance and sample efficiency of various state-of-the-art MARL algorithms.

Introduction

In cooperative MARL, multiple agents must learn to coordinate to maximize a shared reward. While entropy regularization is a common technique in single-agent RL to encourage exploration, its application in multi-agent settings is problematic. Entropy-based methods often change the fundamental objective of the Markov Decision Process (MDP), leading to biased policies. Furthermore, stabilizing off-policy learning in multi-agent environments is difficult due to the "moving target" problem, where agents' policies change simultaneously, causing high variance in updates.

The DMAC Framework

The authors propose DMAC to address these challenges through divergence regularization.

Key Innovations

  1. Divergence Objective: Instead of maximizing entropy $H(\pi)$, DMAC minimizes the Kullback-Leibler (KL) divergence between the current policy $\pi$ and a target policy $\rho$ (often a moving average of past policies). This acts as a "trust region," preventing the policy from changing too drastically and ensuring stable updates.
  2. Monotonic Improvement: The framework mathematically guarantees that policy updates will improve (or maintain) performance in both the regularized objective and the original, un-regularized MDP, eliminating the bias inherent in entropy-based RL.
  3. Off-Policy Compatibility: DMAC introduces a naturally off-policy update rule. By utilizing a replay buffer and a specific gradient formulation, it accounts for discrepancies between target and current policies more effectively than previous on-policy methods.
  4. Universal Compatibility: DMAC serves as a modular framework compatible with major MARL baselines, including COMA, MAAC, QMIX, DOP, and FOP.

Experimental Evaluation

The researchers evaluated DMAC in two distinct settings:

  • Didactic Stochastic Game: A controlled environment with 30 states and 3 agents, used to measure "cover rate" (exploration efficiency).
  • StarCraft Multi-Agent Challenge (SMAC): A complex benchmark featuring maps of varying difficulty, from "Easy" to "Super Hard" (e.g., 3m, MMM2, 3s5z).

Key Results and Insights

Empirical results demonstrate that DMAC consistently enhances the performance and stability of baseline algorithms.

DMAC improves episode rewards across multiple MARL baselines (COMA, QMIX, MAAC, DOP) in a stochastic game.

Performance and Stability

As shown in the stochastic game results, DMAC-enhanced versions of COMA, QMIX, MAAC, and DOP achieve higher episode rewards with lower variance compared to their vanilla counterparts. On "Super Hard" SMAC maps like MMM2, DMAC enabled algorithms to discover successful coordination strategies where standard versions often failed. Direct comparisons also showed that DMAC outperformed state-of-the-art entropy-regularized methods like FOP.

Exploration vs. Regularization

The regularization coefficient $\omega$ allows for a controllable trade-off between exploration and final performance.

The relationship between regularization weight (omega), cover rate (exploration), and episode rewards in the stochastic game.

In the stochastic game, DMAC achieved a significantly higher cover rate than baselines. The "moving target" policy acts as an anchor, providing a smoother learning signal in the chaotic multi-agent landscape. As $\omega$ varies, researchers can tune the balance: a higher $\omega$ promotes broader exploration (higher cover rate), while the theoretical bounds ensure that the gap to the true optimal policy remains proportional to the regularization weight.

Conclusion and Future Work

DMAC provides a robust theoretical and practical framework for regularizing cooperative MARL. While it shows superior performance over entropy-based methods, challenges remain in automatically scheduling the hyperparameter $\omega$ and extending the theoretical proofs to fully account for partial observability in Dec-POMDP settings. Future work may also explore alternative target policies to further enhance agent diversity or robustness.

Summary updated

BibTeX
@inproceedings{su2022divergence,
  author = {Kefan Su and Zongqing Lu},
  title = {{Divergence-Regularized Multi-Agent Actor-Critic}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=tQ2yZj4sCnk},
}
Figure