In standard reinforcement learning, an agent learns to map states to actions to maximize a reward. When we move to multi-agent reinforcement learning (MARL), the complexity does not just double or triple—it explodes. If one agent has 10 possible actions, a team of 100 agents has $10^{100}$ possible joint actions. This "curse of dimensionality" makes it computationally impossible for a centralized controller to explore the space or for individual agents to account for every peer's behavior.
To solve this, researchers exploit the inherent structure of large-scale systems. By treating populations as distributions, focusing on local neighborhoods, or ignoring individual identities, they transform an exponential problem into a manageable, often linear, one.
The Mean-Field Approximation: From Individuals to Densities
When a swarm becomes sufficiently large, the identity of any single agent matters less than the collective behavior of the crowd. This is the core intuition behind Mean-Field Games (MFG) and Mean-Field Control (MFC).
Instead of an agent trying to predict the actions of $N$ separate neighbors, it interacts with a single "mean field"—an aggregate distribution of the states and actions of the entire population. Zuyue Fu, Zhuoran Yang, and Zhaoran Wang demonstrated in Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Games that this approach allows actor-critic algorithms to converge to a Nash Equilibrium at a linear rate, effectively bypassing the curse of dimensionality.
The mathematical "magic" here is that as the number of agents $N$ approaches infinity, the influence of any single agent on the global state becomes negligible ($1/N$). This allows researchers like Washim Uddin Mondal and Vaneet Aggarwal to prove that many-agent systems can be accurately approximated by simplified mean-field models, even when agents are heterogeneous.
Graph Locality: Only Neighbors Matter
In many real-world systems, such as drone swarms or power grids, an agent is primarily affected by its immediate neighbors. A drone in the back of a formation doesn't need to know the exact coordinates of a drone at the very front to avoid a collision.
Graph Policy Gradients (GPG), introduced by Arbaaz Khan, Alejandro Ribeiro, and Vijay Kumar 0001, leverages this by using Graph Convolutional Networks (GCNs) to learn local communication filters. In Graph Policy Gradients for Large Scale Robot Control, they showed that policies trained on just three robots could be transferred "zero-shot" to swarms of over a hundred. Because the GCN learns relational rules (how to move relative to neighbors) rather than absolute ones, the policy remains valid regardless of the total swarm size.
This concept of "localized policies" is further formalized by Adam Wierman and Na Li 0002 in Scalable Reinforcement Learning of Localized Policies for Multi-Agent Networked Systems. They exploit the fact that in networked systems, the effect of one agent's action decays exponentially as it travels through the network, allowing for decentralized learning with global performance guarantees.
Anonymity and Exchangeability
A major source of complexity in MARL is the assumption that every agent is a unique entity. However, in a swarm of identical robots, the agents are "exchangeable." If Agent A is in state $S_1$ and Agent B is in state $S_2$, the global value is the same as if their positions were swapped.
Philipp Robbel and Frans A. Oliehoek tackle this in Exploiting Anonymity in Approximate Linear Programming. By focusing on the count of agents in a particular state rather than their specific IDs, they reduce the state space from exponential to polynomial. This "anonymity" allows planners to scale to systems that were previously mathematically intractable.
Curriculum Learning: Growing the Swarm
Sometimes the best way to learn a complex many-agent task is not to start with the full population, but to grow into it. This is the "From Few to More" philosophy.
Tianpei Yang and Jianye HAO proposed a Large-Scale Dynamic Multiagent Curriculum Learning framework. Agents first learn basic coordination in small groups where the state-action space is small and rewards are easier to find. As they master these, the number of agents is increased. Because the agents share parameters or use relational architectures, the knowledge gained in small groups provides a powerful initialization for the massive swarm, preventing the agents from getting lost in the high-dimensional noise of a large-scale start.
Similarly, Joni Pajarinen and Zhiyuan Li use Self-Paced Multi-Agent Reinforcement Learning to treat the number of agents as a controllable variable, increasing the population only when the agents show "learning progress" (measured by TD-error), ensuring the complexity of the task never outpaces the agents' current capabilities.