Introduction
The "bystander effect" or "diffusion of responsibility" (DR) is a well-known social psychological phenomenon where individuals are less likely to offer help when others are present. This research reveals a strikingly similar failure mode in Multiagent Reinforcement Learning (MARL). When large groups of agents ($N > 30$) learn to cooperate, they frequently struggle with "minority responsibility"—tasks that only a few agents need to perform for the team to succeed. For instance, in a scenario requiring one agent to guard a gate while others attack, standard algorithms often fail because every agent "expects" another to fulfill the role.
The Root of the Problem
The authors argue that DR in MARL is not primarily a credit assignment or reward calculation issue. Instead, it is rooted in the exploration-exploitation dilemma. In standard MARL, agents explore the environment independently and simultaneously. Consequently, the "noise" from one agent's exploration is perceived by others as random environmental instability. As the team size grows, this aggregate noise becomes overwhelming. Agents, unable to distinguish between environmental randomness and teammate exploration, settle into safe "mainstream" behaviors, ignoring critical but risky minority roles.
Proposed Solution: Policy Resonance (PR)
To address this, the authors propose Policy Resonance (PR), a training-time plugin compatible with state-of-the-art algorithms like PPO or QMIX. PR forces agents to synchronize their decision-making states during Centralized Training (CT). The system alternates between two states based on a resonance probability ($\eta$):
- Resonated State: All agents act "greedily" according to their current best policy, pausing exploration to focus on exploitation.
- Non-Resonated State: Agents act autonomously and explore independently, following standard procedures.
By periodically forcing the team to act in unison without the "fidgeting" of individual exploration, the mutual influence of exploration is resolved. This clarity allows learning algorithms to identify the benefits of specialized minority behaviors. Crucially, resonance is only applied during training; during execution, agents remain fully decentralized, maintaining the Centralized Training, Decentralized Execution (CTDE) paradigm.
Experimental Evaluation
The approach was evaluated in two primary environments:
- Fast Multiagent Evaluation (FME): A diagnostic suite featuring tasks like Lv-C (Responsibility Assignment). This level simulates a "fragile bridge" that collapses if too many agents attempt to cross simultaneously, specifically testing for DR.
- Asymmetric Decentralized Collective Assault (ADCA): A high-fidelity swarm combat environment where 50 RL agents face an outnumbering force of scripted AI.
Key Results
In the FME Lv-C task, standard PPO-MA failed completely as the team size reached 7 agents. However, with Policy Resonance, teams maintained optimal performance even with 40 agents. In the ADCA swarm combat task ($N=50$), adding PR to the Conc4Hist algorithm increased the win rate from approximately 50% to over 80%.
The authors also introduced PR-Fast, a variant that freezes the "body" of the neural network during the resonance phase, achieving similar performance gains with reduced computational overhead.
Conclusion
Policy Resonance provides a simple yet effective solution to the diffusion of responsibility problem in large-scale MARL. By recognizing that independent exploration acts as noise for teammates, PR enables agents to "agree" on specialized roles through synchronized training phases. This approach paves the way for more effective coordination in complex, many-agent systems without sacrificing decentralized execution.
Summary updated