Overcoming the Homogeneity Trap in Shared Multi-Agent Learning

162 concepts89 papers1 proposal

In cooperative Multi-Agent Reinforcement Learning (MARL), scaling up to teams of dozens or hundreds of agents presents a massive computational challenge. The standard solution across the field is Full Parameter Sharing: forcing all agents to share a single neural network policy. This approach drastically reduces the number of parameters to optimize and allows agents to pool their experiences, accelerating training convergence.

However, this efficiency comes at a steep cost. When agents share identical weights, they map identical or similar observations to the exact same actions. In environments that demand a specialized division of labor—such as a soccer match requiring distinct roles for defenders and strikers—parameter sharing acts as a bottleneck. Instead of diversifying, the team falls into a homogeneity trap, where agents behave as redundant clones of one another. As shown in Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning, forcing agent homogeneity can lead to joint rewards that scale exponentially worse than the optimal outcome as the team size grows.

Why Simple Fixes Fail: Gradient Interference and the ID Illusion

A common workaround is to append a unique agent identifier (an "Agent ID") to each agent's observation vector, theoretically allowing the shared network to differentiate its outputs. Yet, empirical evidence shows this often fails in complex tasks.

As explored in Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing, the primary failure mode of fully shared networks in heterogeneous environments is not a lack of raw representational capacity, but rather gradient interference. When heterogeneous agents share a single set of parameters, a gradient update intended to optimize a specialized behavior for Agent A can inadvertently overwrite or degrade the specialized knowledge required by Agent B. Simply providing an agent ID does not structurally isolate these updates; the shared layers still suffer from catastrophic cross-role interference during backpropagation.

Furthermore, Woojun Kim and Youngchul Sung note in Parameter Sharing with Network Pruning for Scalable Multi-Agent Deep Reinforcement Learning that standard parameter sharing fails to provide enough distinct representational space for agents to execute specialized, non-overlapping functions. When the global reward signal is not properly decomposed to account for individual temporal influences, the system naturally gravitates toward redundant, uniform strategies.

Architectural Decoupling: Giving Agents Unique Sub-Brains

To break policy homogenization without sacrificing the sample efficiency of shared learning, researchers have turned to architectural innovations that decouple an agent's identity from its observations.

One approach is to dynamically generate unique weights for each agent on the fly. In HyperMARL: Adaptive Hypernetworks for Multi-Agent RL, Kale-ab Tessera et al. propose using an agent-conditioned hypernetwork. By feeding the agent's unique identity into a hypernetwork that outputs the weights of the actual policy network, they effectively separate "who an agent is" from "what an agent sees." This allows agents to learn highly specialized behaviors as if they had independent networks, while maintaining the compact training footprint of a shared model.

Alternatively, instead of generating weights, we can prune them. Woojun Kim and Youngchul Sung introduced a structured network pruning method (SNP-PS) that instantiates unique "sub-brains" within a single shared network. By randomly pruning different subsets of neurons for each agent at the start of training, agents retain individual representational pathways, preventing gradient updates from completely overwriting distinct strategies.

Forcing Diversity via Contrastive and Hybrid Learning

Beyond architectural changes, another powerful mechanism involves explicitly rewarding behavioral or representational diversity.

In value-based MARL, the "mixing network" (which combines individual agent utilities into a joint action-value function) often acts as a bottleneck. If the mixing network cannot distinguish between individual contributions, the team collapses into homogeneity. To counteract this, Shunyu Liu and Shunyu Liu 0001 developed Contrastive Identity-Aware Learning (CIA), a framework that forces the mixing network to differentiate individual agent identities using a contrastive loss function.

Similarly, Chenghao Li et al. in Celebrating Diversity in Shared Multi-Agent Reinforcement Learning emphasize that encouraging unique contributions prevents agents from redundantly targeting the same objectives, significantly improving coordination in tasks requiring distinct sub-tasks.

Finally, some frameworks strike a middle ground through hybrid parameter sharing. The Coordinated Actor-Critic (CAC) algorithm, introduced by Siliang Zeng et al. in Learning to Coordinate in Multi-Agent Systems: A Coordinated Actor-Critic Algorithm and Finite-Time Guarantees, allows agents to share a core set of policy parameters for group coordination while maintaining local, "personalized" components to adapt to unique environmental constraints.

Key Foundational Work

Two Fielded Teams and Two Experts: A RoboCup Challenge Response from the Trenches (1999) — contributed 2 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 104 papers
  2. Version 1 · 2026-10-04 · 159 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_parameter_sharing_limitations_84055a07,
  title = {parameter sharing limitations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-sharing-limitations/rdr_84055a07922641ed8766557e9fdf8943},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_parameter_sharing_limitations_84055a07_v2,
  title = {parameter sharing limitations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-sharing-limitations/rdr_84055a07922641ed8766557e9fdf8943/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake