When Is Diversity Rewarded in Cooperative Multi-Agent Learning?

Michael Amir, Matteo Bettini, Amanda Prorok

Introduction

In Multi-Agent Reinforcement Learning (MARL), a recurring design choice is whether to employ homogeneous agents (sharing a single policy) or heterogeneous agents (learning distinct roles). While heterogeneity allows for specialization, it introduces higher coordination costs and training complexity. Historically, this choice has been guided by empirical trial-and-error. This paper introduces a mathematical framework to predict when specialized teams will outperform identical ones by analyzing the "curvature" of reward functions.

Theoretical Framework: The Curvature Test

The authors model multi-agent tasks as an allocation of effort across multiple objectives. They decompose the global team reward into a two-tier aggregation system:

  1. Inner Operator ($T$): Maps individual agent efforts on a specific task to a task-level score.
  2. Outer Operator ($U$): Merges these task scores into a single global reward.
Conceptual framework of heterogeneity gain and the role of Schur-convexity in reward aggregation.

The core contribution is the "curvature test" based on Schur-convexity. The authors prove that specialization is rewarded ($\Delta R > 0$) when the inner operator is Schur-convex (favoring concentrated effort from a specialist) and the outer operator is Schur-concave (favoring a balanced distribution of success across different tasks).

Key Insights: The "Concave-Convex" Sweet Spot

A significant finding is that diversity is a predictable consequence of reward curvature. By using a Softmax aggregator, the reward structure can be tuned via a temperature parameter ($t$). At specific temperatures, the aggregator flips from favoring homogeneity to favoring heterogeneity.

Heatmap showing the Heterogeneity Gain ($\Delta R$) across different inner and outer temperature regimes, highlighting the "sweet spot" for specialization.

The "sweet spot" for diversity occurs when the reward structure incentivizes one agent to excel at a specific task (convex inner) while ensuring the team does not neglect any single objective (concave outer). Furthermore, the authors observed that "neural" heterogeneity (different weights) is most critical when agents have limited or local observations; with perfect global observability, a shared policy can often learn to produce diverse behavioral roles.

Experimental Validation

The theory was validated across several benchmarks:

  • Matrix Games: Empirical results matched theoretical predictions exactly. For instance, a min outer operator combined with a max inner operator yielded significant heterogeneity gains.
  • HetGPS (Heterogeneity Gain Parameter Search): A new algorithm that automatically discovers pro-diversity reward regimes by tuning environment parameters to maximize the performance gap between heterogeneous and homogeneous teams.
  • Complex Environments: In tasks like Multi-goal-capture, 2v2 Tag, and VMAS Football, the "curvature test" remained a robust predictor of whether specialized roles would emerge and provide a performance advantage.

Conclusion and Future Work

This research provides a principled way to design rewards that naturally incentivize agent specialization. While currently focused on "task allocation" problems and requiring differentiable simulators for the HetGPS algorithm, it lays the groundwork for more systematic MARL system design. Future work will explore extending this framework to non-differentiable environments and more complex social dilemmas.

Summary updated

BibTeX
@inproceedings{amir2026when,
  author = {Michael Amir and Matteo Bettini and Amanda Prorok},
  title = {{When Is Diversity Rewarded in Cooperative Multi-Agent Learning?}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=uJCGMBO6Qx},
}
Figure