When Is Diversity Rewarded in Cooperative Multi-Agent Learning?

Michael Amir, Matteo Bettini, Amanda Prorok

TL;DR

Behavioral diversity isn't always a win in multi-agent systems; sometimes, a "one-size-fits-all" homogeneous strategy is perfectly optimal. This paper provides a rigorous mathematical framework to predict exactly when diversity pays off based on the "curvature" of the reward function. It also introduces HetGPS, an algorithm that automatically designs environments to incentivize the emergence of specialized roles.

The Cost and Benefit of Diversity

In the world of Multi-Agent Reinforcement Learning (MARL), researchers face a constant trade-off. Homogeneous teams—where agents share the exact same neural network parameters—are easy to train and scale. However, they often struggle with complex tasks that require a "division of labor." Conversely, heterogeneous teams—where agents have unique parameters—can specialize into distinct roles but are harder to train and require more memory.

The missing piece of the puzzle has been a principled answer to a fundamental question: When is diversity actually worth the cost? Until now, reward functions in MARL were often designed by trial and error. This paper moves from "vibe-based" reward design to a formal theory of Heterogeneity Gain—the measurable performance benefit a diverse team has over a uniform one.

Heterogeneity Gain Framework

A Theory of Reward Curvature

The authors tackle the problem from two angles: theoretical analysis and algorithmic search.

1. The Curvature Theory

They model rewards as a two-step aggregation process:

  • Inner Operator ($T$): Determines how individual agent "efforts" are combined into a single task score (e.g., "Does the task need one hero or a crowd?").
  • Outer Operator ($U$): Determines how multiple task scores are combined into a global team reward (e.g., "Do we need to finish all tasks or just the best one?").

The authors prove that the Schur-convexity (a mathematical measure of how a function responds to inequality or "spread") of these operators dictates the benefit of diversity. Specifically, a concave outer operator combined with a convex inner operator mathematically guarantees a positive "Heterogeneity Gain" ($\Delta R > 0$).

To apply this theory to complex, time-extended MARL, the authors introduce a gradient-based "co-design" algorithm called HetGPS. It treats the environment's reward parameters as variables to be optimized. The "outer loop" of the algorithm adjusts the environment to maximize the performance gap between a heterogeneous team and a homogeneous baseline, while the "inner loop" trains the agents themselves using standard MARL techniques (like MAPPO).

Environments and Key Insights

The authors validated their theory across a spectrum of environments, ranging from abstract matrix games to complex continuous control physics (like VMAS Football) and classic benchmarks (like Colonel Blotto).

The "Concave-Convex" Rule in Action: Diversity is rewarded when the system wants "coverage" across tasks (concave outer) but rewards "peaked" effort on individual tasks (convex inner). For example, if a team is rewarded for the minimum score across all tasks, they are forced to diversify so that every task gets specialized attention. The theory correctly predicted that a "Min-Max" reward structure (minimizing the maximum distance to goals) would force agents to split up, while a "Max-Max" structure would cause them to collapse into a single, redundant group.

Observability vs. Diversity: A fascinating secondary finding is that the need for "neural" heterogeneity (different network weights) disappears if agents have very rich observations. If agents can see each other and their environment perfectly, they can "behave" differently even if they share the same brain. Diversity in parameters is most valuable when agents have limited information.

Key Results

  • Theoretical Match: In simple matrix games, the learned policies matched the theoretical "Heterogeneity Gain" predictions with near-perfect accuracy.
  • Emergent Specialization: In complex environments like 2v2 Tag, the "Min-Max" reward structure led to a massive heterogeneity gain, whereas "Mean-Mean" rewards resulted in zero gain. This proves that the reward shape is the primary driver of specialization.
  • HetGPS Success: When given a "blank slate" reward function (using Softmax or Power-Sum operators), HetGPS successfully "rediscovered" the optimal curvature parameters predicted by the theory to maximize diversity.
Heterogeneity Gain in MARL Environments

Limitations and Future Work

  • Differentiability: HetGPS currently relies on a differentiable simulator (like VMAS) to be efficient. While the authors suggest gradient-free alternatives for "black-box" simulators, these are significantly slower.
  • Transition Dynamics: The current theory focuses almost entirely on the reward function. However, the physical constraints of the world (transition dynamics) also play a role in forcing diversity (e.g., two robots can't stand in the same spot). Integrating these dynamics into the curvature framework is a suggested next step.
  • Symmetry: The analysis assumes agents and tasks are "interchangeable" at the start. Extending this to teams with inherently different physical capabilities (e.g., a drone and a ground rover) remains an open challenge.

Summary updated

Abstract

The success of teams in robotics, nature, and society often depends on the division of labor among diverse specialists; however, a principled explanation for when such diversity surpasses a homogeneous team is still missing. Focusing on multi-agent task allocation problems, we study this question from the perspective of reward design: what kinds of objectives are best suited for heterogeneous teams? We first consider an instantaneous, non-spatial setting where the global reward is built by two generalized aggregation operators: an inner operator that maps the N agents’ effort allocations on individual tasks to a task score, and an outer operator that merges the M task scores into the global team reward. We prove that the curvature of these operators determines whether heterogeneity can increase reward, and that for broad reward families this collapses to a simple convexity test. Next, we ask what incentivizes heterogeneity to emerge when embodied, time-extended agents must learn an effort allocation policy. To study heterogeneity in such settings, we use multi-agent reinforcement learning (MARL) as our computational paradigm, and introduce Heterogeneity Gain Parameter Search (HetGPS), a gradient-based algorithm that optimizes the parameter space of underspecified MARL environments to find scenarios where heterogeneity is advantageous. Across different environments, we show that HetGPS rediscovers the reward regimes predicted by our theory to maximize the advantage of heterogeneity, both validating HetGPS and connecting our theoretical insights to reward design in MARL. Together, these results help us understand when behavioral diversity delivers a measurable benefit.
BibTeX
@inproceedings{amir2026when,
  author = {Michael Amir and Matteo Bettini and Amanda Prorok},
  title = {{When Is Diversity Rewarded in Cooperative Multi-Agent Learning?}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=uJCGMBO6Qx},
}
Figure