In multi-agent reinforcement learning (MARL), the naive approach to scaling is to give every agent its own neural network. However, this leads to a "parameter explosion": memory requirements grow linearly with the number of agents, and the search space becomes prohibitively large. Researchers have converged on a more efficient alternative: homogeneous parameter sharing. By forcing multiple agents to share the same underlying neural network weights, systems can scale to millions of participants while accelerating the learning process through pooled experience.
The Efficiency of Pooled Experience
The primary intuition behind parameter sharing is that in many environments, agents face similar challenges. Whether they are drones in a swarm or cars at an intersection, the fundamental "physics" of their decision-making is often identical.
Kun Shao et al. demonstrated in StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer Learning that sharing policy parameters among homogeneous units drastically reduces training time. Instead of each unit learning how to move or attack from scratch, every agent’s experience informs a single, collective model. This pooling of transitions allows the system to converge much faster than independent learners.
This efficiency is taken to the extreme in frameworks like MAgent: A Many-Agent Reinforcement Learning Platform for Artificial Collective Intelligence, which supports up to a million agents on a single GPU. By sharing weights, the memory footprint remains constant regardless of the population size, enabling the study of emergent social phenomena that would be computationally impossible with independent models. Similarly, Koen Ponse et al. utilized shared parameters in EconoJax: A Fast & Scalable Economic Simulation in Jax to achieve a 2400x speedup, showing that centralized training with shared weights is sufficient to produce complex economic behaviors like labor specialization.
The Mechanism of Individuality
If every agent shares the same "brain," how do they avoid behaving like a single, rigid block? The key is to condition the shared network on agent-specific context. Even with identical weights, agents produce different actions because they receive different inputs—specifically, their local observations and a unique agent index or embedding.
Jiechuan Jiang et al. explored this in The Emergence of Individuality, where they rewarded agents for behaving in ways that made them identifiable by a classifier. This "identifiability" objective, combined with agent indices, allows a shared-parameter model to trigger a division of labor. Similarly, Yuntao Liu et al. showed in Heterogeneous Skill Learning for Multi-agent Tasks that conditioning a shared policy on unique latent skill variables allows agents to specialize in distinct roles without the overhead of separate networks.
This concept extends to Large Language Models (LLMs) as well. Yiqun Chen et al. proposed in Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning that different functional agents in a pipeline (e.g., a rewriter and a generator) can share a single underlying LLM. Their distinct behaviors are enforced through unique system prompts rather than separate weights.
Structural Specialization through Subnetworks
While embeddings provide context, some researchers argue for more structural differentiation within the shared architecture. The goal is to allow agents to share "general knowledge" while maintaining "private" logic.
One approach is network pruning. In Parameter Sharing with Network Pruning for Scalable Multi-Agent Deep Reinforcement Learning, the authors introduced a method where agents share a weight set but operate through unique binary masks. By pruning different neurons for each agent, they create overlapping "sub-brains" within one network. This provides the representational diversity of independent networks with the memory efficiency of a shared one.
Another method involves hypernetworks. The authors of HyperMARL: Adaptive Hypernetworks for Multi-Agent RL proposed using a shared meta-network to generate unique policy weights for each agent. This effectively decouples the agent's identity from its observations, allowing for high levels of specialization while keeping the training process centralized and efficient.
Selective and Clustered Sharing
Full parameter sharing assumes all agents are similar enough to learn from one another. In highly heterogeneous teams, this can lead to "gradient interference," where the learning needs of one agent type conflict with another.
Filippos Christianos addressed this in Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing by introducing a clustering approach. Instead of global sharing, the system automatically identifies agents with similar goals and dynamics, grouping them to share parameters only within their cluster. This provides a scalable middle ground: it reduces the parameter count compared to independent models but avoids the performance bottlenecks of forcing a single policy onto a diverse population.
Hybrid architectures also offer a solution. Siliang Zeng et al. proposed the Coordinated Actor-Critic (CAC) algorithm, which splits parameters into a shared set for collective coordination and a personalized set for local adaptation. This ensures that agents can benefit from group-wide strategies while remaining flexible enough to handle unique local tasks.
Stability and Deployment
Beyond memory savings, parameter sharing acts as a regularizer. In offline settings, the authors of MOSDT found that full parameter sharing stabilizes training by mitigating the noise generated when agents learn divergent policies. This stability is crucial for deploying multi-agent systems on resource-constrained hardware. For example, Selim F. Yilmaz et al. demonstrated in Distributed Deep Joint Source-Channel Coding over a Multiple Access Channel that using identical encoder weights across multiple transmitters allows communication systems to scale to more users without increasing the complexity of edge devices.
By treating the multi-agent population as a single, distributed function, parameter sharing transforms the challenge of coordination from a problem of scale into a problem of architectural design.