AIR: Unifying Individual and Collective Exploration in Cooperative Multi-Agent Reinforcement Learning

Guangchong Zhou, Zeren Zhang, Guoliang Fan

TL;DR

Adaptive exploration via Identity Recognition (AIR) is a novel framework that unifies individual "curiosity-driven" exploration and collective "diversity-driven" exploration in multi-agent reinforcement learning. By using a simple adversarial identity classifier and a dynamically tuned "temperature" parameter, AIR allows agents to switch between exploring their own capabilities and coordinating distinct roles, leading to state-of-the-art performance on complex benchmarks like StarCraft II and Google Research Football.

What Problem Does This Paper Address?

In cooperative Multi-Agent Reinforcement Learning (MARL), agents must learn to work together to achieve a global goal. However, exploring the vast space of possible joint actions is incredibly difficult. Most popular value-based methods (like QMIX) rely on simple $\epsilon$-greedy strategies—essentially rolling the dice to pick random actions—which often fails in complex tasks.

Existing research usually tackles this in one of two ways:

  1. Individual Exploration: Encouraging agents to be "curious" about states they haven't seen much (often based on uncertainty).
  2. Collective Exploration: Encouraging agents to develop diverse behaviors so they don't all try to do the same thing (e.g., in football, you don't want every player chasing the ball at once).

The problem is that these two modes are rarely integrated. Previous attempts to combine them resulted in "heavy" models that were computationally expensive and difficult to train. AIR aims to bridge this gap with a unified, lightweight framework.

Methods

The core of AIR is a centralized Identity Classifier ($q_\zeta$). This classifier looks at an agent's trajectory (its observations and actions) and tries to guess which agent produced it.

The authors then introduce an Adversarial Action Selector. Instead of just picking the action with the highest estimated value ($Q$), the agent picks an action that maximizes: $$u = \arg \max_u [Q^k(\tau, u) - \alpha \log q_\zeta(z_k | \tau, u)]$$

The magic lies in the Adaptive Temperature ($\alpha$):

  • When $\alpha > 0$ (Individual Exploration): The agent tries to mislead the classifier. It chooses actions that make its trajectory look less like its "usual" self, effectively exploring low-probability, novel actions.
  • When $\alpha < 0$ (Collective Exploration): The agent tries to help the classifier. It adopts distinct, recognizable behaviors that separate it from its teammates, fostering behavioral diversity and role-playing.

AIR uses Dual Gradient Descent to automatically tune $\alpha$ during training, allowing the system to decide whether it needs more individual curiosity or more group coordination at any given stage.

Datasets

The authors evaluated AIR on three major benchmarks:

  • StarCraft II Multi-Agent Challenge (SMAC): A standard for MARL, focusing on micro-management tasks. They tested on 6 "Hard" and "Super Hard" scenarios.
  • SMACv2: A more difficult version of SMAC with randomized unit types and positions, requiring higher adaptability.
  • Google Research Football (GRF): A physics-based football simulator that requires sophisticated teamwork and tactical decision-making.

Key Insights and Ideas

  • The Classifier as a Multi-Tool: The same identity classifier can drive two seemingly opposite types of exploration just by flipping the sign of the temperature parameter.
  • Dynamic Switching: The authors found that agents typically benefit from individual exploration early in training (to learn basic skills) and shift toward collective exploration later (to refine coordination).
  • Efficiency Matters: Unlike previous exploration methods that add massive neural network modules, AIR is extremely lightweight. As shown in Figure 5 (below), AIR adds very little overhead compared to the baseline QMIX, while competitors like QPLEX or RODE are 2-3 times larger.
The relative model sizes of different algorithms. AIR is significantly more efficient than other exploration-heavy methods.

Key Results

AIR demonstrated significant performance gains across the board:

  • Superiority in "Super Hard" Tasks: In SMAC scenarios like 3s5z_vs_3s6z and 6h_vs_8z, where most baselines (MAVEN, RODE, LDSA) failed completely, AIR achieved win rates of 60-70%.
  • Generalizability: While some methods like RODE performed well in StarCraft but failed in Football, AIR maintained high performance across both environments.
  • Visualizing Diversity: Using t-SNE visualizations (Figure 4 in the paper), the authors showed that as training progresses and $\alpha$ turns negative, agent trajectories clearly diverge into distinct clusters, proving that the model successfully induces behavioral diversity.
Figure 2: Experiment results of AIR and baselines on SMAC.

Limitations and Future Work

While AIR is highly effective, the paper notes a few areas for further exploration:

  • Hyperparameter Sensitivity: The minimum expected entropy ($\bar{H}$) is a hyperparameter that needs to be calculated as a running mean, which might require tuning for specific new environments.
  • Decentralized Execution: While training is centralized, the identity classifier is only used during training. The authors suggest that exploring how such "identity awareness" could benefit agents during decentralized execution (inference) could be a promising future direction.

Summary updated

Abstract

Exploration in cooperative multi-agent reinforcement learning (MARL) remains challenging for value-based agents due to the absence of an explicit policy. Existing approaches include individual exploration based on uncertainty towards the system and collective exploration through behavioral diversity among agents. However, the introduction of additional structures often leads to reduced training efficiency and infeasible integration of these methods. In this paper, we propose Adaptive exploration via Identity Recognition~(AIR), which consists of two adversarial components: a classifier that recognizes agent identities from their trajectories, and an action selector that adaptively adjusts the mode and degree of exploration. We theoretically prove that AIR can facilitate both individual and collective exploration during training, and experiments also demonstrate the efficiency and effectiveness of AIR across various tasks.
BibTeX
@inproceedings{zhou2025air,
  author = {Guangchong Zhou and Zeren Zhang and Guoliang Fan},
  title = {{AIR: Unifying Individual and Collective Exploration in Cooperative Multi-Agent Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i21.34454},
}
Figure