Learning Joint Behaviors with Large Variations

Tianxu Li, Kun ZhuAAAI 2025 · DOI

TL;DR

This paper introduces DJBD, a novel multi-agent reinforcement learning exploration method that encourages agents to learn diverse joint behaviors by maximizing the "traveled distance" in their behavior space. By addressing the limitations of existing mutual information-based methods, DJBD leads to more effective exploration and improved cooperative policies in complex multi-agent tasks.

What Problem Does This Paper Address?

Cooperative Multi-Agent Reinforcement Learning (MARL) aims to train multiple agents to work together to achieve a common goal. While significant progress has been made, a key challenge remains: insufficient exploration. Agents often get stuck in suboptimal strategies because they don't explore the full range of possible joint behaviors. Existing methods, particularly those using mutual information, try to encourage exploration by making agents' behaviors informative of some latent variable. However, the authors argue that these methods can be satisfied by only minor variations in joint behaviors, failing to push agents towards truly diverse and distant strategies. This limits their ability to learn complex cooperative policies needed for challenging tasks.

Methods

The core contribution of this paper is the Direction-based Joint Behavior Difference (DJBD) method. Instead of relying on mutual information, DJBD focuses on learning a representation function that maps joint behaviors into a latent space. The key ideas are:

  • Direction Term Objective: DJBD maximizes a term that aligns the direction of the difference between consecutive joint behaviors in the latent space with a latent variable. This encourages the learning of representations that capture meaningful changes in behavior.
  • 1-Lipschitz Constraint: To ensure that increased distance in the representation space translates to increased distance in the actual joint behavior space, a 1-Lipschitz constraint is imposed on the representation function. This constraint guarantees that the function doesn't stretch or shrink distances arbitrarily.
  • Per-Step Transition Reward: DJBD integrates this objective into a MARL framework (specifically QMIX) by introducing a per-step transition reward. This reward encourages agents to take actions that lead to larger differences in their joint behaviors at each step, promoting continuous exploration.

The authors implement DJBD on top of the QMIX value-decomposition architecture, a popular CTDE (Centralized Training, Decentralized Execution) framework.

Datasets

The paper evaluates DJBD on three established MARL benchmarks:

  • Level-Based Foraging (LBF): A cooperative task where agents must collect items of varying levels, requiring coordination and communication.
  • StarCraft Multi-Agent Challenge (SMAC): A suite of challenging combat scenarios requiring agents to learn complex micromanagement and tactical skills.
  • SMACv2: An extension of SMAC that introduces more randomness in unit types and initial positions, increasing stochasticity and posing a greater exploration challenge.

Key Insights and Ideas

  • Mutual Information's Blind Spot: The paper convincingly argues that mutual information, while useful, can be maximized by small, non-diverse behavioral changes, thus failing to drive sufficient exploration in complex scenarios.
  • "Traveled Distance" as an Exploration Metric: The concept of maximizing the "traveled distance" in the joint behavior space (both in the latent representation and, crucially, in the actual behavior space) is a novel and intuitive way to encourage exploration.
  • Lipschitz Constraint is Crucial: The 1-Lipschitz constraint is not just a regularization; it's essential for bridging the gap between exploration in the latent representation space and meaningful exploration in the actual joint behavior space. Without it, maximizing the direction term alone might not lead to better real-world performance.
  • Per-Step Incentives Matter: Introducing a per-step reward based on behavior differences provides a continuous signal for exploration, rather than relying solely on terminal rewards or infrequent intrinsic rewards.

Key Results

  • Superior Performance: DJBD consistently outperforms baseline methods, including QMIX and various mutual information-based approaches (MAVEN, EOI, CDS, PMIC, LIPO, FoX), across all tested benchmarks (LBF, SMAC, SMACv2).
  • Enhanced Exploration: Visualizations (e.g., visitation heatmaps in Figure 5) show that DJBD-trained agents explore the environment more thoroughly and uniformly compared to baselines.
  • Robustness in Challenging Scenarios: DJBD demonstrates significant improvements in more complex and stochastic scenarios (e.g., SMACv2), where other methods struggle.
  • Ablation Studies Confirm Contributions: Experiments show that removing any of DJBD's core components (1-Lipschitz constraint, direction term, or behavior differences) leads to a substantial drop in performance, highlighting the synergy of these elements.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, the introduction of a representation function and associated training might increase computational overhead.
  • Hyperparameter Sensitivity: The effectiveness of the transition reward coefficient ($\alpha$) and the specific implementation of the Lipschitz constraint (Spectral Normalization) might require careful tuning.
  • Future Work: The paper mentions integrating DJBD with policy gradient methods like MAPPO, suggesting potential for broader applicability beyond value-based methods. Further exploration into different representation learning techniques and Lipschitz constraint implementations could also be fruitful.

Summary updated

BibTeX
@inproceedings{li2025learning,
  author = {Tianxu Li and Kun Zhu},
  title = {{Learning Joint Behaviors with Large Variations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i22.34491},
}
Figure